Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Authors: Yuxuan Qiao
Haodong Duan
Xinyu Fang
Junming Yang
Lin Chen
Songyang Zhang
Jiaqi Wang
Dahua Lin
Kai Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modality
VLM
evaluation
Primary area: evaluation
TL;DR: This paper presents Prism, a framework that can be used for: 1) analyzing the perception and reasoning capabilities of VLMs; 2) solving general visual questions efficiently

Abstract: Vision Language Models (VLMs) demonstrate remarkable proficiency in addressing a wide array of visual questions, which requires strong perception and reasoning faculties. Assessing these two competencies independently is crucial for model refinement, despite the inherent difficulty due to the intertwined nature of seeing and reasoning in existing VLMs. To tackle this issue, we present Prism, an innovative framework designed to disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: a perception stage that utilizes a VLM to extract and articulate visual information in textual form, and a reasoning stage that formulates responses based on the extracted visual information using a Large Language Model (LLM). This modular design enables the systematic comparison and assessment of both proprietary and open-source VLM for their perception and reasoning strengths. Our analytical framework provides several valuable insights, underscoring Prism's potential as a cost-effective solution for vision-language tasks.
By combining a streamlined VLM focused on perception with a powerful LLM tailored for reasoning, Prism achieves superior results in general vision-language tasks while substantially cutting down on training and operational expenses. Quantitative evaluations show that Prism, when configured with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on par with VLMs $10 \times$ larger on the rigorous multimodal benchmark MMStar.

Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
Yuxuan Qiao²*, Haodong Duan't, Xinyu Fang', Junming Yang", Lin Chen",
Songyang Zhang', Jiaqi Wang', Dahua Lin 1.3.7 Kai Chen
Shanghai AI Laboratory *Nanjing University 'The Chinese University of Hong Kong
Tongji University Naajjng University of Posts and Telccommunications
University of Science and Technology of China
CPII under InnoHK
yuxuanqiao@smail iaaiaaaommmalnnu........ edu. cn duanhaodong@pp lab. org. cn
Abstract
Vision Language Models (VLMs) demonstrate remarkable proficiency in address- ing wide array of visual questions, which requires strong perception and reasoning faculties. Assessing these two competencies independently is crucial for model re- finement, despite the inherent difficulty due to the intertwined nature of seeing and reasoning in existing VLMs. To tackle this issue, we present Prism, an innovative framework designed disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: perception stage that utilizes a VLM extract and articulate visual information in textual form, and reasoning stage that formulates responses based on the extracted visual infor- mation using Large Language Model (LLM). This modular design enables the systematic comparison and assessment of both proprietary and open-source VLM for their perception and reasoning strengths. Our analytical framework provides several valuable insights, underscoring Prism's 's potential cost-effective solution for vision- languaela tasks. By combining a streamlined VLM focused on perception with powerful LLM tailored for reasoning, Prism achieves superior results in general vision- languae tasks while substantially cutting down on training and operational expenses. Quantitative evaluations show that Prism, when configured with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on par with VLMs 10x larger on the rigorous multimodal benchmark MMStar.
Introduction
56 5. 19 4) Vision With the rapid development of Large Language Models (LLMs) 45 59 60
Language Models (VLMs) [46] 55 6, 36 18] have also experienced significant ad advancements. As an end-to-end approach, VLMs trained on large-scale multimodal data [36 |nI ST g exhibit superior performance on a variety of tasks. These tasks range from basic ones such as object localization [i and optical character recognition
[4 38) to more complex challenges like document or diagram [40]. For VLMs solve general visual question, two essential capabilities are required: 1) Perception: extracting necessary information from the image; 2) Reasoning: generating answers based on the extracted information and contextual understanding. Limitations in either capability can impede the overall performance of a VLM..
comprehension 26 43 42
and solving geometric problems
systematic evaluation of the perception and reasoning capabilities crucial provide valuable insights for future model optimization. However, seeing and reasoning are mostly entangled in
The work done during an internship Shanghai Al Laboratory.
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Stage 1: Perception
Instruction Options
VLM ZOO
T
Instruction
Image
@
4vT-4v
GeminirooV
Query Agnostic
Perception Module
Qwen-VL VL-Mox
Fixed Genenic
Instruction
LLaVA-Series
T
Visual Information
Quary
Reformatted Query
LLM ZoO
@
Chat6PT
SPT4
Reasoning Module
Query-specific
Llama3
Mistrol
Instruction
DeepSeek
Query Aware
Solutions
Stage 2: Reasoning
Figure 1: Prism Framework Architecture. Prism framework takes image-query pairs as input. An instruction (can be query-agnostic or query-aware) and the image are first fed into the VLM to extract visual information. Then, an LLM is used to generate the answer based on the reformatted query which combines the original question and visual information in textual form.
existing VLMs. Proprietary VLMs, such as GPT-4v are opaque systems that only provide the final answer to visual questions. Meanwhile, open-source VLMs 36 commonly utilize vision encoders to extract visual embeddings, which are often difficult to interpret, from the image and employ adapted LLMs to generate answers based on these visual and linguistic embeddings. In light of this challenge, we introduce Prism, a framework designed to disentangle the perception and reasoning processes of any given VLM. It can serve as proxy for assessing the real perception capabilities of VLMs.
Prism decomposes the process of solving general visual questions into two distinct stages: 1) perception stage that concentrates on extracting visual information from the image using VLMs and articulating this information textual form; and 2) a reasoning stage that utilizes LLMs to answer the question based on the extracted visual information. Prism facilitates the analysis of VLMs' capabilities through two approaches. To assess the true perception capabilities, one can use constant LLM in the reasoning stage while testing various VLMs in the perception stage. Conversely, by keeping the VLM fixed and varying the LLM in the reasoning stage, one can determine whether VLM's performance is limited by its reasoning capabilities. Through this analysis, we have uncovered several key insights: 1) Proprietary VLMMs such as GPT-4o and GPT-4v, take the lead in the perception capabilities competition; 2) For open-source VLMs, perception capabilities remain relatively consistent regardless of the language model's size; and 3) The overall performance of open-source VLMs, particularly those with smaller-scale language models like 7B variants, is often constrained by the limited reasoning capabilities.
Beyond its role as an evaluation framework, Prism also excels as efficient general Vision- Language Model (VLM). Building upon findings 2 and 3, we posit that integrating a small-scale VLM as visual captioner with a powerful LLM as reasoning engine offers promising and efficient strategy for general vision-language tasks. By concentrating on visual information extraction, a lightweight VLM can achieve decent performance with a powerful yet economical LLM, thanks to advancements in deployment techniques, one can achieve robust solution for visual- visual-language language information processing, requiring significantly fewer hardware resources for training and deployment. In our experiments, we trained an approximately 2B 1B-parameter vanilla LLaVA to extract visual information and observed that it exhibits perception performance comparable to LLaVA-NeXTT (35] that is equipped with a 34B powerful language model. Quantitative evaluations indicate that Prism, when instantiated with streamlined visual captioner and the freely available ChatGPT-3.5, outperforms many open-source VLMs on multiple multimodal benchmarks including the stringent multimodal understanding benchmark MMStar [10]. Notably, the advantage is particularly pronounced on reasoning-related visual questions.
par with much larger VLMs. When paired
In summary, the contributions of this work are follows:

---

1. We introduce Prism, a highly adaptable framework designed to explicitly disentangle the percep- tion and reasoning processes. Prism enables the breakdown analysis of VLM capabilities and serves as solution for vision-language tasks by integrating any given VLM and L.L.M
2. Utilizing Prism, we conduct a decoupled analysis of the perception and reasoning capabilities of existing VLMs. Several intriguing findings emerge from the ablation study.
3. Drawing inspiration from these findings, we integrate a lightweight VLM focused on perception
with a powerful LLM dedicated to reasoning within the Prism framework. Quantitative results demonstrate that this combination exhibits outstanding performance and efficiency across a range of vision- -aagual tasks.
2 Methodology
2.1 Prism Architecture
Prism is characterized by modular design that decomposes the process of solving visual questions into two stages, consisting of perception module and a reasoning module. both of which can be flexibly replaced, as depicted in Fig. [ The perception module, typically a VLM, initially follows the instruction to extract visual information from images and articulates this information in textual form. The instruction can be generic or query specific, ie., written by the reasoning module given the question (text-only) as contextual information. Meanwhile, the reasoning module, usually an L.M,, performs text-based reasoning on the textual information to generate answers to the questions. To assess the pereeption capabiliitess of various VLMs, we carefully select an appropriate benchmark corresponding to the principle we illustrate in Sec. 2.2 In Sec. 2.3 we describe how we utilize the Prism framework to assess the perception and reasoning capabilities of VLMs, respectively. To substantiate our belief in the potential of combining small-scale VLM with a powerful LLM, we train an approximately 2B- 2--parameter VLM based on the LLaVA architecture to serve as a visual captioner and integrate it into Prism as the perceptual module, detailed in Sec. 2.4
2.2 To Analyze with Suitable Benchmark
Numerous multimodal benchmarks [37 20 28
66 10 exist, each evaluating the capabilities of VLMs from various perspectives. To maximize the utility of the Prism framework, careful selection of the benchmark is imperative for the decoupling analysis. In summary, we adhere to the following principles for benchmark selection: 1) Vision Indispensability: The benchmark must require visual information for question solving, and questions that can be answered without utilizing visual information excluded; 2) Minimal Data Leakage: The visual questions should not be part of the model's training data; and 3) Complexity: The solving process of the visual question should involve both perception and reasoning components. Considering these three principals, we select MMStar [10] as the primary benchmark for the decoupling analysis and ablation. MMStar ensures vision indispensability and makes a concerted effort minimize data leakage, while many other benchmarks are plagued by the two issues.
40
2.3 Prism as an Analytical Framework
Prism functions as analytical framework for evaluating the perception and reasoning capabilities of given VLM. In this section, we elaborate on the methodology for evaluating these capabilities.
To assess the perception capabilities of various VLMs, we employ ChatGPT 1-TT-3.--turb--001 as the reasoning module and standardize instructions for description within Prism. With the reasoning module and instructions fixed, the final accuracy of the visual questions is solely determined by the quality of the visual information extracted. Under the controlled setting, we consider the VQA accuracy as proxy to measure the perception capability of VLMs.
Within the Prism framework, the instructions for visual information extraction are crucial as they are designed to elicit the fundamental perceptual capabilities of VLMs. We have adopted two types of instructions to assess the perceptual abilities of models:
1. Generic Instruction: a standardized, universal instruction aimed at extracting and describing the basic elements present in an image;
2. Query-Specific Instruction:
combination of the generic instruction and an incremental instruc- tion that directs the VLM to provide a detailed account of the visual information relevant to the question. The incremental instruction is crafted by the reasoning module given the text-only question.
3

---

Employed with the generic instruction, Prism offers a straightforward decoupling pipeline. Mean- while, using query-specific instructions is a more realistic setting and can better realize the full potential of the Prism framework. A Prism decoupled GPT-4o (GPT-4o for both perception and reasoning) achieves almost the same quantitative performance compared to the end-to-end GPT-4o. Analyzing the results, we find that with the reasoning module and instruction controlled, VLMs with language models of different sizes display a much narrowed gap in perception performance compared to the results under the end-to-end setting.
Besides evaluating the perception performance, Prism can also roughly measure the reasoning capabilities of VLMs. When used to solve general visual questions in an end-to-end manner, the VLM implicitly performs reasoning with its language encoder. Alternativey. Prism provides an explicit pipeline, in which an off-the-shelf LLM is separately used to predict the answer based on the visual information. For small-scale VI.Ms (7B-parameter, etc.), we find that Prism equipped with the VLM and ChatGPT-3.5 outperforms the end-to-end VLM in quantitative performance, especially for reasoning related VQA. The results reveal that for small-scale VLMs, the overall performance can be heavily constrained by the parameter size of the language model.
2.4 Prism as a Vision-Language Task Solver
Beyond serving an evaluation framework, Prism can also function efficient vision-language task solver. The perception module in Prism can incorporate one or multiple VLMs to extract high-quality visual information. Concurrently, the reasoning module can be instantiated with powerful LLM to harness its advanced reasoning capabilities.
In Sec. 2.3 we observed that VLMs paired with language models of varying sizes exhibit similar performance perception capability. This suggests that it is promising to employ small-scale VLMs to generate informative visual descriptions, serving as efficient perception module. To validate this concept, we conduct extensive experiments to train visual captioners, utilizing the widely adopted LLaVA architecture and only open- datasets. Below, we elaborate on the specific settings considered for this ablation study detail:
143K in ALLaVA [9] as our instruction tuning data. The former comprises 715K pairs of images and detailed visual captions produced by GPT-4v, whereas the latter contains 143K text instruction tuning data, generated by GPT-4-Turbo. Utilizing the descriptive data for instruction tuning better triggers the VLM's ability to extract and articulate visual information more effectively, compared to instruction
Instruction Tuning Data. We use ALLaVA-Caption-41 and Sroll-nnrruct-GP4--
tuning data in QA formats [3 32 52 42).
Model Architecture. To investigate the impact of vision encoder in LLaVA, we experimented with multiple encoders, including Clip ViT-L/14 14 (50), SigLip-S0400M [67] and InternViT-6B [13 For the language encoder, we tested two lightweight variants of InternI 7 InternLM2-7B and InternLM2-1 1.8B.. The inference of all combinations (excluding InternViT T-6B) can be efficiently executed on consumer-level GPUs such as RTX 4090, etc. During instruction tuning, we maintain to the language encoder.
the vision encoder fixed and apply QLoRA [17
Regarding the vision backbone, SigLip exhibits relatively superior performance. Additionally, we observed that larger language model only results in minor differences in perception performance. Our findings indicate that, within the Prism framework, 2B vision captioner can achieve strong perception performance par with LLTVA-NeXT [35 equipped with a 34B language backbone.
3 Evaluation Results
3.1 Implementation Details
Evalution Details. We use Prism to evaluate the capabilities of various VLMs, which can be
categorized into two major groups: (a) Proprietary VLMs, including GPT-4o (20240513) [46).
GPT-4v (20231 106) [66, GeminiPro-V V [55], and Qwen- VL-Max (6; (b) Open-Source VLMs, including LLaVA-V
(34], MitenL--XCoompssee
18. mPLUG-OW12 655 LeTV (35)
Intern Intern VL-Chat--V 1.5 [12.. DeepSeek-VL. (39], MiniCPM-V-2 47 When integrating these VLMS
as perception modules within Prism, we employ greedy decoding and limit the maximum number of output tokens to 512. The evaluation encompasses both generic and query-specific instructions.

---

Generic Instruction
Query SpeecilS Instruction
Model
CP
FP
IR
LR
Math
ST
Overall
CP
FP
IR
LR
Math
ST
Overall
Proprictary VLMr
GPT-4o 46
64.0
41.6
54.4
$1.6
47.6
31.2
48.4
67.2
51.6
63.2
56,4
52.0
36.8
54.5
GPT-4v 46
62.4
33.2
51.2
39.2
44.4
32.8
43.9
67.2
43.6
56.8
46.8
42.0
30.8
47.9
GeminiPro--
57.2
36.0
51.6
43.2
45.6
23.2
42.8
61.2
35.2
53.2
47.6
48.4
19.2
44.1
Qwen-VL-Max 3
62.4
28.8
49.2
45.2
37.2
21.2
40.7
62.4
36.0
54.8
46.4
38.0
24.0
43.6
Open-Sourre
VL.Ms
InternVl-Chut-11./
33.2
42.9
56.4
50.8
Internl.-XCCompasee22
18
56.0
32.0
54.8
33.6
37.2
20.8
39.1
60.8
39.2
58.4
47.6
42.4
19.6
44.7
LLaVA-NeXT (Yi-34B) 35
64.4
37.2
50.0
34.0
39.2
24.8
41.6
60.0
41.6
52.4
44.0
40.8
25.2
44.0
DeepSeck-VL-7B
31.6
49.6
41.3
34.8
42.8
LLaVA-NeXT (Mistral-71I) 3S
60.8
30.8
49.6
37.6
36.8
20.4
39.3
62.8
36.4
53.6
42.0
35.2
26.4
42.7
MiniCMM-V-2 F7
55.6
28.4
49.2
39.6
38.4
19.2
38.4
61.2
30.1
52.0
44.4
40.8
22.8
41.9
LLaVA-NeXT (Vicuna-133
35.2
41.2
19.2
LLaVA-NeXT (Vicuma-7B)
R5
63.2
28.8
46.4
40.8
34.0
22.8
39.3
56.8
37.6
47.2
42.4
36.4
21.6
40.3
mPLUG-Owl2 K
47.2
24.0
45.2
33.6
28.8
18.8
32,9
53.2
34.4
45.2
38.8
38.8
23.6
39.0
LLaVA-115-138
33.6
18.8
LLaVA-v1.5-7B 34
48.8
29.6
48.0
344
27.6
17.2
34.3
50.8
31.6
50.4
38.8
32.0
25.2
38.1
Table 1: Detailed Perception Performance on MMStar under Prism Evaluation Framework.
Reasoning module: ChatGPT. Abbreviations: Coarse Perception (CP), Fine-grained Perception (FP); Instance Reasoning (IR); Logical Reasoning (LR); Science&Teche Technology (ST). Overall best scores are marked as bold, and cntra-category best scores are marked as underline.
Unless otherwise specified, Turbbo-000 is adopted as the reasoning module. All evaluations are conducted using VLMEvalKit [14. Further details on the evaluation are provided Appx. [B.1
Post-Processiee Within the Prism framework, the LLM (particularly proprietary APIs) in the reasoning module often declines to answer questions due to insufficient clues the extracted visual information. When Prism is employed of perception for the visual question and refrain from any post-processing For fair comparisons, whether Prism is contrasted with other VLMs vision-language task solver or different LLMs are evaluated as reasoning modules with varying rejection rates, a random choice utilized as a fallback when option matching fails.
evaluation framework, we classify this as a failure
3.2 Main Results
We present the primary evaluation results of various VLMs' perception performance in Tab.[ Results categorized under two instruction types: Generic Instruction and Query-Specific Instruction
Generic Instruction Results. When using generic instructions, GPT-4o exhibits exceptional
performance across a variety of tasks. Intern MiterVI.-Chat-v1.. 1,5 achieves the highest overall perception performance among open-source VLMs, nearly on par with the proprictary GPT-4v. LLaVA-NeXT (Yi-34B) demonstrates strong coarse and fine-grained perceptual abilities but lags behind GPT-4y in recognizing abstract elements LR and ST visual questions. Smaller open-source VLMs, such as mPLUG-Ow12 and LLaVA-v1.5-7BB struggle with both coarse and fine-graie ane-grained perception and encounter difficulties recognizing essential elements in reasoning- VQA.
Query-Specific Instruction Results. GPT-4o demonstrates significantly superior performance compared to all other models in the extraction and expression of visual information across all dimensions. GPT-4y is equally adept at handling coarse perception contents. Among open-source VLMs, Intern VL--Chat-v1.5 excels in perception tasks across all dimensions excluding LRR Its performance not only surpasses that of other open-source VLMs but is also slightly ahead of GPT-4v. Occasionally, a decline in performance can be observed for specific VLM-task pairs when using query-specific instructions instead of generic ones, which can stem from difficulty in understanding query-specific instructions for specific domains.
3.3 Detailed Analysis
Are Proprietary Models Getting Ahead of the Game? When employing both generic and query- specific instructions, proprietary VLMs, particularly GPT-4o, significantly surpass other models in perceptual capabilities and can adeptly manage a wide range of tasks, as demonstrated in Tab. Certain open-source models, such as Intern V.- and VLaNN-NeXT (Yi-34B), have achieved
5

---

60
End-to-End
Perception
Performance (4)
55
50
45
40
35
InternVi-Chat- -Cht--1-.
GPT-He
Intern InterLLt-XCoppase XCosposer2
LLaVA-HeT7 (Yi-348)
GPT-dV
(wen-Vt -Rax
LLaVA-MexT (Vicuna- 138)
DeepSoek--
RiniCPH-V-2
LLavA LLVt (Mistral-mp
GominiPro-V
LLaVA-HexT (Vicuna-7D) 78)
BALUE-OW12
LLaVA-u1. LLaVA-e1.5-133 5-138
LlaVA-v1 5-78
Figure 2: Comparing End-to-End Performance and Perception Capability on MMStar. We display model accuracies in end-to-end VQA and the Prism perception test with query-specific instructions. Most small-scale (7B, 13B, etc.) VLMs achieve better performance within Prism.
notable performance, approaching the capabilities of proprietary VLMs like GPT-4v and GeminiPro-V V. Other open-source source models, due to their limited perceptual abilities, generally perform slightly worse in Math and ST assessments. Notably, MiniCPM-V-2, lightweight VLM with ~3B parameters, displays better perceptual performance compared to some 7B VLMs.
The Gap between Perception Ability and End-to-End Performance. In addition to solving visual questions in an end-to-end manner, Prism provides an alternative pipeline where the VLM is solely utilized for perception. The distinction between these two methods lies in the reasoning process: the former conducts reasoning internally within VLMs, whereas the latter performs reasoning based on VLM extracted information using external LLM (ChatGPT), The comparison between these two approaches on MMStar is depicted in Fig. 2 For state-of-the-art large-scale VLMs such as GPT-4o and Ve-enV---hatt1.... which expected possess excellent reasoning capabilities, employing an external ChatGPT for reasoning may diminish overall performance. Conversely, for most small-scale VIMs, using ChatGPT for reasoning significantly improves their performance, particularly in reasoning-related VQA, as shown in Fig. 5 This phenomenon indicates that the overall performance of small-scale VLMS can be heavily constrained by the size of the language model. To investigate whether the reasoning ability of ChatGPT constrains tofte--f--hh-ant VLMs, we implemented Prism pipeline that decouples GPT-4o by using both the perception and reasoning module. The result reveals that this Prism pipeline, with post-processing, achieves an overall accuracy of 61% %, nearly identical the end-to-end GPT-4o performance of 61.6%.
How does Language Model Size Affect Perception Ability? During evaluation, we observe that the LLaVA-v1.55 series shows no significant improvement when using larger language model (Vieuna- 13B instead of Vicuna-7B, etc.). This suggests that perception performance may be independent of the language model size when using relatively low-resolution vision backbone. However, the situation appears to differ with I.L.VA-Ne'T Quantitative results for the LLaVA-NeXT series tell that scaling up the language model slightly enhances model perception, particularly when using query-specify instructions. Through a detailed qualitative analysis, we identified the primary factors contributing the superior performance of larger LLa VA-NeXT models over smaller ones follows: (1) More Elaborate Expression: Models equipped with a larger language encoder exhibit enhanced ability to articulate visual information. More detailed and organized narratives make it easier for the reasoning module to answer the question; (2) More Adaptive to Instruction: Larger language backbones entitle the model with better understanding of instructions, yielding more suitable textual visual information for reasoning, particularly in response to query-specifts instructions. In Fig. 3 we provide some qualitative results about the two typical modes.
3.4 Error Analysis
Error Mode. We conduct a thorough analysis and categorize errors in perception and reasoning into the following major modes:
Major Error Modes in Perception: (1) Factual Errors. VLMS may describe images with inaccuracies, such as stating that prominent elements are "not visible"; (2) Incomplete Details.

---

More Elaborate Expression
Adoptive Instruction
Describe... pay spatial orrnngement sittiing at the toble
Instruction: pay
length,
mon woman
(Vicune-78) Arswer: The coach, who shirt
(Vicune-78) Answert... i. spottal arnangement of the man and woman suggests a close interaction..
pants, the of the group, clopping his hands,
(Vi-348) Answer:
short and appears to be a light brown color. The texture of his hair is straight, and it is neatly combed....
(Vi-346) Answer The woman the left side of the image...
Figure 3: The Effect of Language Model Size Perception Ability. We compare visual informa- tion extracted from different LLaVA-NeXT models. Left: LLaVA-NeXT (Yi-34B) tells the spatial arrangement more detailed way; Right: LLaVA-NeXT (Vicuna-7B) dismisses the query on the man's hair while LLa VA-NNTT (Yi-34B) tells all contents elaborately following the instruction.
VLM
Agreement Rate (VLM)
Error Rate (VLM)
Alignment
GPT-4o
68.6
GPI-35-Turbo-00125
87.1
GPT-4v
61.5
DeepSoek-v2-Chat
91.3
LLaVA-NeXT (YI-34B)
60.7
43.4
Llama-3-70B-Instruct
92.8
LLaVA-v1.5-BB
50.6
56.5
GPT-4-Turbo-0125
90.7
Table Agreement Rates and Error Rates of Different VLMs.
Table 3: Alignment Rates Different LLMss
Even in the absence of factual errors, VLMs may lack detailed content, resulting in insufficient information for reasoning; (3) Instruction Following: VLMs sometimes fail to follow instructions when providing corresponding descriptions.
2. Major Error Modes in Reasoning: Logical Errors. LLMs may produce incorrect conclusions or reasoning processes due limited reasoning abilities; (2) Lack of Knowledge. The absence of relevant domain knowledge prevents LLM from solving corresponding problems, especially in specific fields; (3) Misunderstanding the Query. In rare cases, the query-specific part generated by LLMs deviates from the original question, misleading the perception of VLMs.
Hallucination. To reveal the source of hallucinations, we employ various LLMs reason about descriptions generated by the given VLM and analyze the patterns their results. We conduct exper- iments various VLMMs with of powerful language models, including 11t.33.--urbo-01111 299-4-Turbo-0125
111mm-3-7000 3-w----------------------------- and DeepSeek-v2-Chats For each question, we gather results from four LLMS and consider cases where the same choice appears three times more as "agrremmen".. In agreement cases, we define the corresponding choice as "voted choice" and focus on the following cases:
Case 1: With descriptive texts generated by a specific VLM, the voted choice wrong.
Case 2: With descriptive texts generated by specific VLM, the voted choice is correct.
Case 3: For specific LLM, it makes the same choice the voted choice by all LLMs, while the voted choice also the correct one.
For each VLM, we analyze the rate of case 1, which indicates the potential hallucinations of VLMs. For each LLM, we correspondingly calculate the rate of case 3 to case observe the alignment of its predictions with voted choices. The higher alignment rate means more robust reasoning since voted options are considered better. The notions are as follows.
Agreement Rate (VLM) number of agreement cases
number of all cases number of case
Error Rate (VLM) number of agreement cases
number case 3
Aligmment Rate (LLM) number of case 2
As presented in Tabs. and stronger VLMs show lower error rates, indicating that more capable VLMs experience fewer ha hallucination issues. By delving deep into the specific cases, we find that VLMs are prone to hallucinations spatial awareness and fine-grained perception. All LLMs show relatively good alignment rates, demonstrating relatively robust reasoning performance. GPT-3.5- Turbo-0125 more unstable by comparison. The cases where LLM'' prediction does not align with the voted choice may stem both from reasoning ability issues and hallucinations.
7

---

LLaVA-NeXT
Instructions
GPT-4e
(Yi-34I3)
GPI-4o
GPT-4v
LLaVA-NeXT
LLaVA-11.-7B
(YI-34B)
Human 2
47.7
40.4
GPT Symthesize I
47.1
41.3
GPT-:5-Turbb-0125
54.7
48.5
44.1
38.5
GPT Synthesize 2
48.1
40.1
GPT-4-Turbo-1125
36.9
48.7
48.4
38.3
CoT
47.7
41.4
Llamm-3-70B-Instruc
Decompose
47.3
41.4
DeepSeck--2-Chat
57.8
49.7
45,8
35
Human 1
48.4
41.6
Table 5: Ablation Using Different LLMs as the Rea- soning Module.
Table 4: Ablation on Different
Generic Instructions.
Different Generic Instructions
Human I: Describe the fine-grained content of the image, including scenes, objects, relationships, instance location, and any text present.
Human Deseribe the fine-grained content image, including scones, objects, relationships, instance location, background and any text present. Please skip generating statements non-existent contents and descrihe you
sec.
GPT Synthesize 1: Given the image below, please provide detailed description of what
GPT Synthesize 2: Analyze image below and describe main elements and their relationship.
Co Describe the fine-grained content
image, including scenes, objects, relationships, instance location,
any text present. Let's think step by step.
Decompose: Decumpose the image into several parts and describe the fine-grained content of the image part by part, including scenes, objects, relationships. instance location, and any text present.
Figure 4: Different Generic Instructions we adopted in the Ablation Study.
Ablation Study
Ablation on Generic Instructions. Within Prism, the generic instructions for visual information extraction are crucial. We experimented with variety of instructions to elicit the fundamental perceptual capabilities of VLMs, including numan-written instructions, GPT-4 generated instructions. and those incorporating thain-of-thought [61] explicit decomposition, as shown in Fig. 4 We conducted an ablation study on MMStar using the state-of-the-art VL.M GPT-4o and L.L.VA-eXT (Yi-34B). As illustrated in Tab. Human 1 outperforms others in eliciting the fundamental perceptual capabilities of the models, while the differences among various instructions are not significant. Therefore, adopt Human 1 as the generic instruction for all evaluations.
Ablation on the Reasoning Module. The reasoning module is critical for accurately determining the correct answer based on the visual information. To evaluate the impact of the reasoning mod- ule on overall performance, we select four LLMs: GPT-3. 5-Turbo-0r 0125, Ll 11lam-33-2 3-BB-- Instruct, and DeepSeek- vi-e--22chat and assess these models across four VLMs with varying capacities. The comparative results are presented in Tab. 5 As freely avail- able model, GPT-3.5 demonstrates good reasoning performance under our framework, and the more advanced GPT-4 shows improved performance line with other benchmarks. Notably, Llama3-70B-Ine - as representative of open-source models, exhibits competitive capa- bilities compared to GPT-4-Turbo-012 under different perceptual conditions. This suggests that open-source models could be valuable for further exploration in visual reasoning.
-orrbo-o-o--- -0255,
Ablation on the Vision Backbone. To investigate the impact of the vision encoder on perception ability within the LLaVA architecture, we conduct an ablation study on three pre-trained visual back- We use InternLM2-1
bones, including CLIP ViT-L/14. SigLip-SO444 S0400M, and InternViT- as the fixed language encoder in LLaVA and train the VLMs for one epoch with the vision encoder fixed. The results in Tab. show that SigLip-S04009 achieves better performance compared to CLIP ViT-L/14 and InternViT-6B on MMStar.
5 Prism
Within Prism, we explore the use of small-scale Vision-Language Models (VLMs) perception module. We use SigLip as the vision encoder, InternLM2- [1.88/7B] as the language encoder to develop two visual captioners at different scales, refered as PrismCaptioner- [2B/7B].

---

Instance Reesoning
Logic Reasening Nath
15
Vision Backbone
Overall
18
SigLip-S0400M
43.5
InternVIT-6B
42.7
CLIP VIT-L/14
41.7
-5
Despleck----78 78-78
Minicow-i-2
ReeiniPreny
LLatlk-eer Micuns-t (8)
4aM4- 5-210
Llatt Chistaal (88)
Figure 5: The Performance Changes of Using External LLM Table 6: Ablation on the Vi- sion Backbone.
(ChatGPT) for Reasoning of Small Scale VLMs.
5.1 Training Details
We perform one-stage training for two epochs using 7eRO2 with XTuner [15 on 8 NVIDIA A800- 80GB GPUs, and the training lasts less than a day. The training data include of one copy of ALLaVA-Caption-
and two copies of Evol- to 16 per device for PrismCaptioner 2B and for PriemCaptioner- optimizer and 2e-4 learning rate, with the warm-up ratio set to 0.03 and (2) set to (0.9, 0.999). No weight decay was applied and a maximum norm value of is applied for gradient clipping. Full details about QLoRA are presented in Appx. B.2
T133K-133. The batch size is set We utilized the Adam
5.2 The Performance of PrismCaptioner
We conduct thorough evaluation of PrismCaptioners across multiple benchmarks, employing GPT-3. 11-3355Thum S--Turbo-0122 0015 and Llama-3- 1-mma-3700B IOB-Inst as the reasoning module. We utilize MMStar primary benchmark assess comprehensive multimodal capabilities. For domain-specific evaluations, we choose A12D to gauge diagram comprehension, MMMU for expert knowledge as- sessment, and Math to test mathematical proficiency. The results presented in Tab. 7 In line with the benchmark selection principles outlined in Sec. [.2, we apply a consistent filtering strategy across AI2D, MMMU, and Math Vista, mirroring that of MMStar, retain only vision-essential and uncontaminated questions, denoted by the suffix (F). In addition to comparisons with existing open- source and proprietary VLMs, also assess two baseline models, ILaaAA-InternMM2-1 (1.8//1_1_18B/8]) which are Pri Priisaapiioeers and ShareCaptiones an open open-source VLM designed for generating informa- tive image captions, under identical Prism framework configurations.
.8//7B],
trained on LLa 1.5 instruction tuning data. We also compare
As depicted in Tab. 7 each PrismCaptioner, with external potent L.M for reasoning, markedly surpasses its corresponding end-to-end baseline. PrismCaptioners also outperform ShareCaptioner across all multimodal benchmarks. For the 7B variant, the integration of Llama3 results in a substantial enhancement, positioning PrismCaptioner vision-language solver, particularly MMStar and MMMU. For PrismCapiione ChatGPT yields superior results, outperforming nearly all 7B VLMs in general aptitude, expert knowledge, and mathematical skills. Remarkably, it achieves performance levels on par with some ten times larger VLMs, such as LLaVA- Inte-MM 20B, Vi-VL-B- VL-34B, and Emu2-Chn Chat. This demon- strates that Prism enables the creation of robust yet efficient vision- languaee solver, exemplified by Pri Prismapptioner- 2B with ChatGPT, which delivers impressive results.
as highly competitive employing
6 Discussion
Prism's Value as an Evaluation Framework. Prism's value as an Evaluation Framework lies in its ability to disentangle and measure the perception and reasoning capabilities of VLMs across various data sources. There exist specialized multimodal benchmarks designed to assess VLMs' perception and reasoning capabilities, yet they often focus on specific domains. For instance, RealWorldQA 62) evaluates real-world perception with high-resolution images, OCRVQA 44 assesses text recognition in publications, and POPE 30 determines object existence in images. However, many interested domains (geometry, medical images, GUIs, etc.) are not covered by those perception benchmarks. Prism fills this gap by enabling the measurement and comparison of VLMs' perception capabilities

---

Model
MMStar
MMMU
MMMU
Math
Vath (F)
A12D (F)
Pmprietary VLMs
GPT-40 46
61.6
62.8
45.9
56.5
46.5
82.2
59.7
GPT-4v
49.7
53.8
Open-Source VLMS
Inter\I--Chat-v1.5
57.1
Internl.M-XComppeeet
18
56.2
41.4
21.2
59.5
49.0
81.2
57.5
LLaVA-NeXT (Yi-3BB)
51.6
48.8
26.7
40.4
51.6
LLaVA-NeXT (Vieuna-13B)
35
40.4
37.3
16.5
34.1
19.6
72.2
36.2
LLaVA-NeXT (Mistral-7B)
38.4
37.0
19.2
34.6
21.3
69.0
32.3
LLaVA-NeXT (Vicuna-7B)
35
37.6
37.6
18.0
31.5
17.1
67.0
29.3
Vi-VI-34B 1
40.5
45.1
21.2
31.5
12.0
65.9
26.4
Emu2-Chat S
40.7
35.0
27.8
30.7
14.0
49.7
22.7
LLaVA-ntermLLM2-000 15
41.9
39.4
18.0
23.3
9.9
65.4
28.4
DeepScck-VI--BB
40.5
38.3
36,9
20.7
36,9
MiniCPM-V-2 47
39.1
38.2
21.2
39.8
25.4
62.9
26.9
LLaVA-v1.5-1BB
LLaVA-11557B [4
33.1
35.7
15.7
25.6
8.5
55.5
17.8
mPLUG-Owi2
34.8
34.7
Spen-Souree VLMs (E2F. Baselines)
MhaYY--IntemLMM--1..
15
34.5
30.2
18.4
26.3
23.5
LL 15
38.3
40.1
21.9
26.0
7.4
63.6
26.6
Prism Models
ShareCappioner-ChtttOf
38.7
45.2
30.2
30.2
12.6
59.6
22.2
43.3
46.6
32.0
33.5
18.2
62.0
27.4
PrismCaationer-28--11
42.0
46.7
32.4
33.5
30.3
PrismCaptionee-7B-Caaee
43.7
47.3
29.8
35.1
24.6
65.4
30.6
PrismCaptionee-7B-222ee.
53.3
35.6
37.2
Table 7: Detail Results of Models under Prism Framework F) represents the sub-dataset filtered by our strategy order ensure vision indispensability and avoid data leakage. The suffix of Prism models (ChatGPT or Llama3) indicates the reasoning module adopted.
general VQA datasets in these domains. Additionally, existing 'reasoning' benchmarks compositional, requiring VLM to recognize key elements and before reasoning. Comparing an end- to-end VLM with Prism equipped with the same VLM and an external LLM, like ChatGPT, provides insights into the VLM's intrinsic reasoning capabilities and potential performance constraints.
40
26
are
Prism's Value as a Vision-Language Solver. By integrating small-scale VLMs with readily available external LLMs within Prism, achieve superior performance compared to the standard end- to-end capabilities of the standalone VLM. This approach also renders it practical to address vision- languae tasks using 2B parameter VLM, as reasoning is effectively outsourced to external LLMs. When implemented with an LLM API, Prism's inference process (without quantization) only consumes several gigabytes of GPU memory. Furthermore, Prism allows for the flexible incorporation of multiple VLMs to enhance perception. For instance, the straightforward concatenation of outputs from GPT-4v and GeminiPro-V has demonstrated substantial improvements across the majority of metrics the MMStar benchmark. as substantiated by the data presented in Table
VLM
CP
FP
LR
Math
Overall
GPT-4v
67.2
43.6
56.8
46.8
42.0
30.8
47.9
GeminiPro-V
61.2
35.2
53.2
47.6
48.4
19.2
44.1
Ensemble
66.4 47.21
58.81
50.81
32.07
50.37
Table 8: Performance of Perception Module with Multiple VLMs on MMStar.
For comparison, we use single image as input and limit the maximum numbers of output tokens to 512. Better performance of PrismCaptioners is presented in Appx. B.3
10

---

Acknowledgement
This project is supported by the National Key R&D Program of China (No.2022ZD01666))))
the Shanghai Postdoctoral Excellence Program (No.2023023) China Postdoctoral Science Fund NNo.2024M751559) and Shanghai Artificial intelligence Laboratory. This project is funded in part by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (TTC)'s ' InnoHK. Dahua Lin a PI of CPII under the InnoHK.
References
[1] 01-ai. Yi-vi. http:///ugginnfaaee.oo/00/>/>>>>
2023.
[2] Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8958-8957,00 2019.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-23736. 2022.
[4] Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024.
[5]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. /6609, 2023.
[6] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 129.66308..1966, 2023.
[7] Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning. Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song. Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang. Jiaqi Wang, Jiayu Wang. Rui Wang, Yudong Wang. Ziyi Wang. Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang. Haochen Ye, Huaiyuan Ying. Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang. Xinyue Zhang, Hui Zhao, Qian Zhao Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internlm2 technical report, 2024.
[8] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long- tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3555 1568-356,,,4 2021.
[9] Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for lite vision-language model, 2024.
11

---

[10] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao. Dahua Lin, al. Are we on the right way for evaluating large -aagun-an models? arXiv preprint 2Xiv:2403.203300 2024.
[11] Lin Chen, Jisong Li. Xiaoyi Dong, Pan Zhang, Conghui Hc. Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023.
[12] Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang. Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang. Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. How far are we to gpt-4v? closing the gap to commercial multimodal models with open open-source suites, 2024.
[13] Zhe Chen, Jianman Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual- Ssual--inguistic tasks. arXiv preprint arXXv:2312..42200 2023.
[14] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.
2023.
[15] XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning Ilm. https://gi thub. com/
Intern[M/xtuner; 2023.
[16] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint urXiv:2305.06500 2023.
[17] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023.
[18] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. and comprehension in vision-language large model. arXiv preprint
xcompporr:::: Mastering free-form text-image composition
16420, 2024.
[19] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 220-335, 2022.
[20] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models. ArXiv, abs/2306. //////300.1334,,.. 2023.
[21] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXIv:2404.12390 2024.
[22] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017.
[23] Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, et al. Infimm-eval: Complex open-ended reasoning evaluation for multi-modal large language models, arXiv e-prints, pages arXiv-2311, 2023.
12

---

[24] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[25] Wang Jin, Dong Shichao, Zhu Yapeng. Yao Kelu, Zhao Weidong, Li Chao, and Luo Ping. Diagnosing the compositional knowledge of vision language models from a game-theoretic view. To appear in ICMI., 2024.
[26] Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
[27] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
[28) Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge. and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307. 16125, 2023.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[30] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[31] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt 13, pages 740-755. Springer, 2014.
[32] Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023.
[33] Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image- context reasoning benchmark challenging for gpt-4v (ision), 1lava-1.5, and other multi-modality models. arXiv preprint wxXiv:330.115560 2023.
[34] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[36] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.00888 2023.
[37] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint 2xxv:2307.00688 2023.
[38] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07894 2023.
[39] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vi: towards real-world vasion-language understanding, arXiv preprint arXiv:2403.05522 2024.
13

---

[40] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-We Wei Chang. Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255: 2023.
[41] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozheh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/eof conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[42] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[43] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200-2209, 2021.
[44] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. 2019 international conference on document analysis and recognition (ICtiionnIAAAA (ICDAR), pages 947-952. IEEE, 2019.
[45] OpenAl. Chatgpt.
2023.
[46] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08874, 2023.
[47] OpenBMB. Minicpm:
Unveiling the potential of end-side large language models, 2024.
[48] Aitor Ormazabal, Che Zheng, Cyprien de Masson d' d'Attmme, Dani Yogatama, Deyu Fu, Dono- van Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. Reka core. flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404. 12387, 2024.
[49] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. /4824, 2023.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8766 8763.873..228 PMLR, 2021.
[51] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-fittred 400 million image-text pairs. arXiv preprint arXiv:2 1..02/111.00/// 2021.
[52] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 8317-8326, 2019.
[53] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint arXiv:2312.. 2.n::2282..8 13286, 2023.
[54] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprini arXiv:2303. 133300,,,39,,,. 2023.
[55] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. / 2023.
[56) InternI.M Team. Internim: multilingual language model with progressively enhanced capabilities. httpp///giihhhoom//eeee/////e/en techheert,,.t.eererportt.. 2023.
14

---

[57] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5228 5238, 2022.
[58] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, arXiv preprint arXiv:2401.06209 2024.
[59] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 230.:2332..1971,, 2023.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2tti::23077092888. 2023.
[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
[62] XAI. Grok-1.5 vision preview. 2024.
[63] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Jiao Qiao, and Ping Luo. Lylm-ehub: A comprehensive evaluation benchmark for large vision-language models. 2023.
[64] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan. Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4178, 2023.
[65] Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-moda large language model with modality collaboration, 2023.
[66] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023.
[67] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[68] Jianrui Zhang, Mu Cai, Tengyang Xie, and Yong Jae Lee. Countercurate: Enhancing physical and semantic visio-linguistic compositional reasoning via counterfactual examples. Findings of the Association for Computational Linguistics: ACL 2024, 2024.
[69] Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai Josh Susskind, and Navdeep Jaitly. How far are we from intelligent visual deductive reasoning? arXiv preprint arXiv:2403.cm 04732, 2024.
[70] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing lssion-language understanding with advanced large language models. arXiv preprint a00iv:2304. 10592, 2023.
15

---

A Related Work
A.1 Large Vision-Language Models (LVLMS)
The landscape of Large Language Models (LLMs) is continually evolving, with expanding body of research focused on integrating multimodal capabilities to enhance their perceptual abilities in real-world contexts (49 77 (9]. Early efforts this direction, such as Flamingo [3] introduced gated dense blocks of cross-attention within pre-trained language encoder layers to fuse visual features. Subsequent models like BLIP2 [29] and InstructBLIP [66 utilized Q-former to align features across different modalities, enabling tasks such as zero-shot visual question answering. More recent models, including LLaVA [36] and MiniGPT-4 [70]. have simplified modality bridging through the use of MLP-based projection layers, d offering a more straightforward approach compared to the Q-former. The architecture of LLa VA has been widely adopted in subsequent works [35] 12 39] The choice of vision encoders and the language model is considered critical for the overall performance of 27 70 employ CLIP-based Vision Transformer [50] 67
54 as the vision encoder, owing to its good pre-training alignment of visual and textual modalities. There is a prevalent belief [37 [35] that the scale of the language model significantly impacts the performance of VLMs, though detailed analyses are lacking. In addition to the open-source VLMs developed by the academic community, numerous proprietary VLMs 46 55
VLMS. Most VLMs [36] 18 35
demonstrate robust performance across various multimodal benchmarks. This paper presents breakdown capability analysis of both open-source and proprietary VLMs using the Prism framework.
6 4S
A.2 LVLMs Capability Evaluation
Large-scale VLMMs have demonstrated promising outcomes across a diverse range of multimodal tasks, as evidenced by extensive qualitative and quantitativve evaluations. Early assessments of MVL.Ms often involved open-ended Visual Question Answering (VQA) 2 24 22) and numan-bases subjective evaluations 164 63]. However, these methods face limitations in accurately reflecting the true performance of VLMs. Open-ended VQA tasks typically demand an exact match between the model's prediction and the ground truth, which can lead to a significant number of false positives. Conversely, subjective evaluations introduce biases and make the results challenging to reproduce. Subsequent research has shifted towards structuring visual questions in closed-ended formats, such as multiple-choice or Yes-or-No questions. Pioneering works like MMBench (37], MME [20], or SEED- Bench (28 have presented comprehensive evaluations of VLMs using closed-ended VQA, covering various perception and reasoning capabilities. Additionally, specialized multimodal benchmarks have emerged assess VLMs from specifie angles. For instance, MMMU [66] evaluates VLMs' ability
to handle multimodal examination questions, while POPE 30 and HallusionBench
33 scrutinize
hallucination and illusion phenomena in VLMs. RealWorldQA
(62) focuses on real-world perception
with high-resolution images. Recently, MMStar [10] has addressed the issues of vision dispensability and data contamination in existing benchmarks by compiling high-quality, vision-indissensablle questions from multiple sources, ensuring minimal data leakage and covering six core capabilities. In this study, Prism primarily utilizes MMStar for capability evaluation.
A.3 LVLMS Capability Breakdown Investigation
To offer insightful and detailed feedback for the future optimization of VLMs, some researchers have initiated efforts to dissect the abilities of VLMs, aiming to uncover strategies for enhancement. To more effectively explore the disparity between VLMs and human cognition, Zhang et al. [69] employ Raven's ss Progressive Matrices (RPM) to examine the model's deductive reasoning skills grounded in visual perception. Through error case analysis, the authors observe that VLMs often make compounding and confounding errors when articulating individual elements within RPM, which subsequently results in erroncous reasoning. Although this study provides qualitative insights, does not encompass systematic evaluation of perception and reasoning capabilities. Meanwhile, researchers have developed specialized benchmarks assess specific capabilities. For example, InfiMM-Eval 23 and Math Vista [40] have implemented rigorous, step-by-step evaluations of the model's complex reasoning abilities on natural images and mathematical VQA problems, respectively. However, these reasoning benchmarks require foundational perception capability to accurately
identify key elements. Concurrently, there are perception benchmarks [58
that exclusively focus evaluating the perception skills of VLMs across various scenarios. Moreover, some
62
30 21
16

---

specific crucial capabiliities of VLMs have received researchers' attention, such as fine-grained
understanding [68] and visio-linguistic compositional knowledge [57 68 25]. this study, Prism
introduces a general decoupling framework that facilitates a detailed analysis of perception and reasoning capabilities, applicable to any multimodal benchmark.
B
Supplementary Details of Prism
B.1 Details of Prism Evaluation Framework
B.1.1 Query-Specific Instruction Details
Query-Specific Instruction combination of generic instruction and query-specific part, as de- picted in Fig. 0 ensure that the query-specific part generated by the reasoning module closely related to the questions and options, we adopt few-shot learning to guide the LLM. For each visual question, we feed the request with multiple examples. This approach helps the reasoning module understand what "contents to observe" means in different contexts and allows it to make accurate inferences in response to specific questions, as illustrated by the prompts in Fig. 7
Instructions
Generic Instruction: Describe the fine-gr ine-grained content of the image, including scenes, objects, relationships, instance location, and any text present. Query-Specifie Instruction: Describe the fine-grained content of the image, including scenes, objects, relationships, instance location, and any text present. Especially, pay attention to <query-specific part>.
Figure 6: Generic Instruction vs. Query-Specifie Instruction.
The Few-shot Prompt Template
Your task is to give concise instruction about what basic elements are needed be described based on the given question. Ensure that your instructions do not cover the raw question, options, or thought process of answering the question.
Question: In which period the number of full-time employees the maximum? Contents to observe: the number of full-time employees
Question: What is the value the smallest bar?
Contents to observe: the heights of all bars and their values
Question: What is the main subject of the image?
Contents to observe: the central theme or object
Question: What
position of the catcher relative to the home plate?
Contents to observe: the spatial arrangement of the objects
Question: What is the expected ratio of offspring with white spots to offspring with solid coloring? Choose the most likely ratio.
Contents to observe: the genetic information Question: <question> Contents to observe:
Figure 7: The Prompt Template for the Reasoning Module to Generate the "Contents to Observe" Part.
B.1.2 Inference Prompt Template of the Reasoning Module
After the perception module generates detailed visual information about the image, the query needs to be reformatted to enable the reasoning module to answer more accurately based the information
17

---

Hyperparameter
Assignment
Image Resolution
384 * 384
Patch Size
Max Length
1296
QLoRA Quantization
4bit
LLM int8 Threshold
6.0
BnB 4bit Compute Dtype
torch.float16
BnB 4bit Double Quant
True
BnB 4bit Quant Dtype
nf4
LoRA
512
LoRA a
256
LoRA Dropout
0.05
Table 9: PrismCaptioner Training Details.
and the question. The template for reformatting is shown in Fig. 8 The reformatting process involves simple splicing, making it intuitive for the reasoning module to respond directly. An example of this can be seen in Fig. 9
Reformatting Template
You are an excellent text-based reasoning expert. You are required to answer the question based on the detailed description of the image.
Description: <description>
Question: <question>
Figure 8: The Template for Reformatting Query.
Reformatted Query Example
You are an excellent text- bast reasoning expert. You are required to answer the question based on the detailed description of the image.
Description: The image presents a delightful celebration Father's Day. Dominating the center of the image is a blue tie, adorned with white stripes, symbolizing the essence of fatherhood. The tie is slightly tilted to the right, adding a touch dynamism to the composition. On the left side of the tie, the phrase "Happy Father's Day" is elegantly inscribed in a white cursive font, extending warm wishes to all dads. The text and the tie against dark blue background, creating a striking contrast that draws attention to the main elements of the image. Adding a final touch of sophistication, thin white border frames the entire image, encapsulating the joyous message of Father's Day. The image, in its entirety, serves as heartfelt tribute to all the wonderful fathers out there. Question: Which special day associated with this poster?
Options: A. Earth Day. B. National Reading Day. C. Father's Day. D. Mother's Day Please select the correct answer from the options above.
Figure An Example of the Reformatted Query.
B.2 More Details of PrismCaptioner Training
The details about QLoRA for training PrismCaptioner is demonstrated in Tab. 9
18

---

MMMU
MMMU (F)
Vith
Vash (F)
A12D
A12D (F)
(Multiple Image Ieputs
Outpur Lengsh 2048))
43,0
46.1
31.4
34,8
16.7
62.1
27.9
Prism Captiooee-2244llamms.
42.6
49.0
34.1
35.1
19.8
59.8
30.3
65.4
30.8
PrismCappiioee-7--LJaana
46.2
56.5
42.0
39.8
26.2
67.9
37.7
Table 10: More Detailed Performance Results of PrismCaptioners
B.3 More Performance Details of PrismCaptioner
In addition to the results of Tab. 7 we use multiple images as iputs if there are and set maximum output length 2048. The performace is presented in Tab. TO
C Detailed Examples of Prism
C.1 End-to-end v.s. Prism Predictions
Some end-to-end models with small language models, often predict incorrect answers due their limited reasoning capabilities. Utilizing an external LLM,, Prism endows VLMs with the reasoning ability solve the vision-language tasks, as presented in Tab. and Tab, [2
C.2 PrismCaptioner Performances
PrismCaptioner can professionally extract and express detailed visual information to solve coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and math tasks, as presented in Tabs.
18
C.3 Performances Comparison between PrismCaptioner and GPT-4o
Although PrismCaptioner generate detailed descriptions of images, there is still room for im- provement compared to GPT-4o.
(1) GPT-4o generates descriptions that are more relevant to the questions. Due to limited training data, PrismCaptioner's 's captions are less adaptive to query-speciffi parts compared to those of GPT-4o, as shown in Tab. [9]
(2) GPT-4o's expression is more detailed and specific, and the responses about spatiotemporal information are also more accurate, as shown in Tab. 20
D Limitations and Broader Impacts
Limitations. In this study, we introduce the Prism framework and showcase its effectiveness both an analytical tool and versatile vision-language task solver. Prism relies on language descriptions while natural languages may struggle express some obscure visual elements, especially some aesthetic contents like abstract artworks and surrealist paintings. It is difficult to obtain high-quality descriptions in these cases. Given budget constraints, our evaluation focuses on a select group of representative open-source and proprietary VLMs, which may not encompass all the most recent bighhpeeforming models. Our experiments with training visual captioners aim to illustrate Prism's 's capability to deliver strong performance on languae tasks while minimizing costs. To fully realize the potential of Prism, additional experiments are recommended, particularly specific visual instruction tuning data, such as tables and diagrams, screens, and graphical user interfaces (GUIs), medical images, etc., to thoroughly assess Prism's 's efficacy in these specialized contexts.
domain-
Broader Impacts. As an analytical framework, Prism offers detailed insights into the perception and reasoning capabilities of languaae models (VLMss,, providing valuable guidance for future model optimization. Training large-scale VLMs necessitates extensive multi-mod data and
19

---

Question:
What is the primary scene depicted in the image?
Image:
gettyme
Options:
A. A group of people taking pictures of statue in plaza
B. A large crowd taking pictures of each other in an outdoor event
C. A group of people taking pictures of Christmas tree in plaza
D. group of people taking pictures of a famous building in New York City
Category: Coarse Perception
Model: LLaVA-NeXT (Vicuna-7B)
End-to-end Prediction: D. A group of people taking pictures of a famous building in New York City ( (x)
The image captures a bustling scene in city square, where large crowd of people has gathered. The crowd is diverse, with individuals of various ages and attire, all seemingly engaged in the event taking place. The central focus of the image is a large, illuminated Christmas tree, which stands prominently in the square. The tree is adorned with multitude of colorful lights and ornaments, creating festive atmosphere.
Perception Description:
In the background, the cityscape is visible with tall buildings, some of which adorned with American flags, suggesting location in the United States. The flags are flying at half-mast, adding solemn tone the scene.
The sky above is overcast, casting a soft light over the scene. The overall mood of the image one of celebration and unity, the crowd seems be enjoying the holiday festivities. The image does not contain any text. The relative positions of the objects suggest well-organized event, with the Christmas tree as the focal point and the crowd surrounding it. The flags in the background provide sense of location and national pride. The overcast sky adds a touch of solemnity to the otherwise joyful scene. Perception Prediction: C. (
Table 11: An Example End-to-End v.s. Prism LexVA-NeXT (Vicuna-7B) Prediction.
computational resources. Prism mitigates this challenge by training small-scale VLMs that specialize in visual captioning tasks and perform reasoning with large language models (LLLMs), which are now cost-effective When employed as vision-language task solver, Prism can be trained and deployed at significantly reduced cost, making it promising approach for highly customized applications or tasks with limited training data. However, Prism also carries potential societal impacts, as could lower the barrier to building multimodal applications, some of which may be harmful. There is risk that Prism could be used to develop harmful multimodal AI systems. Additionally, data-driven methods often inherit biases, which persist in downstream tasks. We urge users to thoughtfully consider the implications of these biases when implementing our model.
2Thanks to the development in deploying technologies, the inference of LLM is now cheap, provided by various corporations price low millions of tokens per US dollar.
20

---

Question: Which number missing?
Image:
4
5
7
10
14
19
?
25
32
Options: A. 40 B. 10 C.
3.
Category: Logical Reasoning
Model: MiniCPM-V-
End-to-end Prediction: B. 10
Perception Description: The image presents grid of nine circles, each containing a number from 4 to 25, The numbers are arranged in 3x3 grid, with the numbers 4. 5, 10, 14, 19, 25, 32, and question mark (?) at the bottom right corner. The sequence of numbers in the grid is 4, 7, 10, 14, 19, 25, 32, with the question mark indicating the missing number.
Perception Prediction: The pattern in the sequence as follows: 4 7 5 = 2, 10 7 = 3, 14 10 = 4, 19 14 = 5, 25 19 6, 32 25 7. Therefore, the missing number should be 32 8 40.
So, the correct answer is: A. 40 ( )
Table 12: An Example of End-to-End v.s. Prism MiniCPM-V-2 Prediction.
21

---

Question:
Which image is more brightful?
Image:
Options: A. The first image Coarse Perception
B. The second image
Category:
Description:
The image is split composition with two distinct halves.
On the left side, there photograph of a bright red sports car parked on the side of road. The car appears to be modern coupe with sleek design, featuring a long hood and a short rear deck, which are characteristic high-performance vehicles. has five-spoke alloy wheels with silver finish and low-profile tires. The car's windows are tinted, and it has a visible antenna on the rear part of the roof. The background shows a blurred natural landscape with greenery, suggesting rural or semi-rural setting.
On the right side of the image, there is digital artwork depicting swirling, fractal-like pattern in shades of green and purple, set against dark background. The pattern has hypnotic, almost hypnotic quality, with a sense motion and energy emanating from The artwork has hypnotic, almost hypnotizing effect, drawing the viewer's attention to the center of the swirl where the colors seem to converge. Prediction: A. The first image
Table 13: PrismCaptioner Generates Description to Solve Coarse Perception Task.
22

---

Question:
What color is the shirt of the man playing the guitar?
Image:
Options: A. Blue
B. Brown Category: Fine-grained Perception
C. Red D. Black
Description: The image depicts live music performance by band on stage. There are three members visible: vocalist, a guitarist, and a keyboard player. The vocalist, positioned in the center, is woman with long hair, wearing red top and black skirt. She is holding a microphone stand with her right hand and appears to be singing into To her left, man is playing electric guitar, dressed in black shirt and jeans, focused on his instrument. To the right of the vocalist, another man is playing keyboard, wearing black shirt and dark pants. The stage is equipped with various musical instruments and equipment, including drum set behind the vocalist, keyboart in front the keyboard player, and microphone stand in front of the vocalíst. The lighting dim, with spotlights highlighting the performers, and the background is a draped curtain in shades of blue and red. The atmosphere suggests intimate concert setting. Prediction: D. Black
Table 14: PrismCaptioner Generates Description Solve Fine-Grained Perception Task.
23

---

Question: Which solution has higher concentration of pink particles? Image:
Solvent volume: 40 mL
Solvent volume: 40 mL
Solution A
Solution B
Options: A. neither, their concentrations are the same Category: Instance Reasoning
B. Solution A
C. Solution B
Description: The image displays two beaker- containers, labeled Solution A and Solution B. each containing clear liquid. Solution A has volume of 40 mL, while Solution B has volume of 40 ml.. Inside Solution A, there are multiple small, round, purple particles dispersed uniformly throughout the liquid. In contrast, Solution B contains similar number of purple particles, but they are clustered more densely, suggesting higher concentration. The background is plain white, focusing attention on containers and their contents. There are additional background elements context provided. Prediction: C. Solution
Table 15: PrismCaptioner Generates Description to Solve Instance Reasoning Task.
24

---

Question: In nature, what's the relationship between these two creatures?
Image:
Options:
A. Predatory relationships
B. Competitive relationships
C. Parasitic relationships D. Symbiotic relationship
Category: Logical Reasoning
Description: The image depicts two antelopes engaged in what appears to physical confrontation on grassy savanna. The antelope the left is rearing up its hind legs, with its front legs extended towards the other antelope, which is also on its hind legs, facing the first antelope. Both animals have prominent, curved horns, and their fur is rich, reddish- brown color. The background is blurred natural landscape, suggesting a wild, open environment typical of savanna ecosystem. There artificial structures visible, reinforcing the natural setting. The colors are vibrant, with the green of the grass contrasting with the reddish-brown of the antelopes. Prediction: B. Competitive relationships
human elements
Table 16: PrismCaptioner Generates Description to Solve Logical Reasoning Task.
25

---

Question: What comes after the full moon?
Image:
Sunlight
waxing Gibbous
1t
Quarter
waxing Cresent
Moon
New
Full
Earth
waning Gibbous
Quarter
waning
Cresent
Options: A. waning cresent B. 3rd quarter
C. waning Gibbous
D. new moon
Category: Science & Technology
Description: The image depicts a simplified diagram illustrating the phases of the Moon in relation to the Earth and the Sun. It is circular diagram with the Sun the top right and the Earth at the bottom left. The Moon is shown as a crescent in the top left, waxing crescent, first quarter, waxing gibbous, full, waning gibbous, third quarter, and waning crescent. Arrows indicate the direction of the waxing and waning of the Moon's 's phases. The background is solid black, emphasizing the diagram's 's white and gray elements. There are no background elements or colors other than the white and gray of the Moon and Earth, and the black of the background.
Prediction: C. waning Gibbous.
Table 17: PrismCaptioner Generates Description Solve Science & Technology Task.
26

---

For case A accompanying table, answer the questions that follow. Calculate the future value of the annuity, assuming that it is an ordinary annuity. Image:
Question:
Case
Amount of annuity
Interest rate
Deposit period (years)
$ 2,500
8%
10
500
12
30,000
20
11,500
6,000
14
30
Options: A. $126,827
B. $223,248
C. $36,216
D. $4,058
Category: Math
Description: The image displays table with five and four columns, excluding the header row. The header row is labeled from left to right as "Case," "Amount of annuity," "Interest rate," and "Deposit period (years)." Below the header, there five cases labeled A, B, C, D, and E. The Amount of annuity" column lists monetary values: $2,500 for case $500 for case B, $30,000 for case C, $1,500 for case D, and $6,000 for The "Interest rate" column lists percentage values: 8% for case A, 12% for case B, 20% for case C, 9% for case D, and 14% for case E. The "Deposit period (years)" column lists numerical values: 10 years for case A, 6 years for case B, 5 years for case C, 8 years for case D. and 30 years for case The table is set against plain background with no additional elements or decorations.
Prediction: To caleulate the future value of an ordinary annuity, you can use the formula:
FV = Pmt * [(1 + r) 1]/r,
where FV Future Value, Pmt Amount of annuity ($2,500), Interest rate (8% or 0.08), = Deposit period (10 years).
Plugging in the values:
FV $2, 500 [[1 + 0.08) 10 1]/0.08
FV = $2 500 * (1.08) 10 11/0.08
FV $2 500 22.159924 1]/0.08
FV 500 * 1.158924/0.08
FV $36, 216.55
Therefore, the correct answer is: C. $36,2166 (V)
Table 18: PrismCaptioner Generates Description to Solve Math Task.
27

---

Question
Where the rug in the living room located?
Options
A. Next the door
B. Under the table D. Under the couch
C. In front of the window
Models
GPT-4o
PrismCaptioner
Descriptions
The image depicts a cozy living room cor- ner with a modern and inviting aesthetic. Here is a detailed description the scene:
The image shows a neatly arranged liv- ing room corner with a focus on side table and a portion of a couch. The side table, which appears to be made of wood with a natural finish, has two drawers with metal handles. On top of the table, there is a large blue lamp with white lampshade, small white bowl, and a green potted plant with broad leaves. To the left of the side table, there is a couch with grey up- holstery, adorned with teal throw pil- low and pink throw blanket casually draped over it. The floor is covered with a grey and white patterned area rug, and the walls are painted in a light grey tone. A window with white trim is partially visible, dressed with pat- terned curtains in shades of grey and white On the wall, there is framed artwork featuring bouquet of flowers in a vase. The door the background is dark grey with a black doorknob, and there is a light switch plate visible on the wall to the right of the door. The overall aesthetic of the room is mod- with touch of warmth provided by the wooden furniture and the cozy textiles.
1. Furniture and Objects:
Sofa: A gray upholstered sofa is positioned the left side of the image. It has few decorative pillows in various patterns and colors, including white, green, and teal. Side Table: Next to the sofa, there is a wooden side table with two drawers. On top of the table, there is blue ceramic lamp with a white lampshade and potted plant with large green leaves.
Rug: A gray rug with white geometric pattern is placed on the floor. The rug is po- sitioned in front of the sofa and extends par- tially under it, covering the wooden floor. Curtains: Light gray curtains with subtle pattern hang from a black curtain rod above window on the left side of the image. Wall Art: Above the side table, there is framed piece of art featuring a colorful flo- ral arrangement.
2. Placement and Relationships:
The rug is centrally placed in front of the sofa, extending slightly under it, and cov- ers portion of the wooden floor. The side table is positioned between the sofa and the door, with the lamp and plant on top providing both functional and deco- rative elements.
The framed art is centered above the side table, adding a pop of color the wall.
Overall, the arrangement of furniture and decor items creates a balanced and aestheti- cally pleasing living room corner. The rug's 's placement in relation to the sofa and side table helps define the seating area and adds warmth to the space.
Predictions
D. Under the couch
C. In front of the window ( (
Table 19: Performances between GPTT40 and PrismCaptioner.
28

---

Question
What is the relative position of the man and the woman sitting at the table?
Options
A. The man is the left and the woman is the right B. The man is on the right and the woman is on the left
C. The man and the woman are facing each other
D. The man and the woman are back-to-back
Models
GPT-4o
Descriptions
The image depicts man and woman sitting closely together at a table in what appears to be a social setting, possibly bar or restaurant. The woman, on the left, has blonde hair styled in loose curls and is wearing white dress with plunging neckline. She leaning in towards the man, appearing whisper something in his ear. Her left hand is resting the table, and her right hand is not visible.
The image depicts man and woman seated at table in what appears to be or restaurant setting. The man is dressed in a dark suit with white shirt and a dark tie, and he holding cigarette between his fingers. The woman is wearing a white dress with a plunging neckline, and her blonde hair is styled curls.
Both individuals are looking at each other, and the woman seems to be whispering into the man's On the table, there are two glasses, one of which appears to contain clear liq- uid, possibly water, and the other is not fully visible. There is also small, dark object that could be an ashtray or coaster.
The man, the right, is wearing a dark suit with a white shirt and a dark tie. He is holding cigarette in his right hand, which is raised near his face. His left hand is resting on the table. He look- ing directly at the camera with a slight smile.
On the table in front of them, there is a champagne glass near the woman and glass with a dark liquid, possibly a cock- tail, near the man. There are also other objects on the table, including what ap- pears be a hand of another person par- tially visible the left side of the image. In the background, there is colorful ar- rangement of red and white flowers the left side. The backdrop includes a painted mural artwork featuring ab- stract shapes and possibly a cityscape with buildings and sailboat.
In the background, there is bouquet of flowers with red and white blooms, and the wall behind them features large, abstract painting with blue and yellow tones. The lighting in the scene is warm, and the overall ambiance sug- gests casual yet intimate setting.
The spatial arrangement shows the man and woman sitting very close each other, with the woman leaning in to- wards the man, indicating sense of inti- macy or private conversation. The man is positioned slightly turned towards the camera, while the woman is more fo- cused him.
Predictions
A. The first image
B. The second image
Table 20: Performances between GPT-40 and PrismCaptioner.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As denoted by the abstract and introduction, in this work, we present the Prism framework that can serve as an analytical framework and a general vision-language task solver. Qualitative and quantitative results in this paper can support the contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes)
Justification:
The limitations of Prism are discussed in Appx. D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
we describe the hyper-parameters and settings of Prism inference and the training settings of VLMs trained this work in detail. So would be easy to reproduce the results reported.
Justification: In Secs. 3.1 and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Data, codes, and other resources required to reproduce the main results will be released later. We need some time to prepare more tidy version of those resources.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Those settings are described in In Secs. 3.1 and 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: During the inference process in Prism, perform greedy decoding for both VLMs and LLMs, so the quantitative results we reported will be stable. Besides, reporting error bars will be expensive and is not common practice for research in large vision-language models.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
These information is included in Sec. 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential impacts of this work are discussed Appx. D
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
33

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work will release the evaluation results and some weights of small-scale VLMs trained on publicly available dataset. We believe this paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in this work are properly credited. Our evaluation codes are modified based on Vii,Eal which goes with Apache 2.0 License. Our self-trained vision captioner is built with SigLip and InternLM2, and trained on ALLaVA and Evol- Instruct-GPT---Turbb all of those assets go with Apache 2.0 License. Other evaluation datasets adopt in this paper go with CC-BY 4.0 License.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "m1KGeNjXme", "reviewer": "Reviewer_KmBE", "summary": "In this paper, the authors propose prism, a framework to decouple the VLMs' capabilities in two stages: perception stages and reasoning stages. This framework allows the breakdown analysis of VLM capabilities and can also serve as a framework to integrate any given VLM and LLM. Based on their explorations and decoupled analysis, they discover that the integration of a lightweight VLM combined with a powerful LLM can be useful and exhibit outstanding performance and efficiency. The author provide good amount of experimentations to support their claims.", "questions": "See weakness section.\n\n1. How about failure modes in different part of the VLMs (reasoning and perception)? Is there any analysis or ideas on analyze the hallucinations based on your framework?", "limitations": "limitations and broader impacts are properly addressed in the appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The author present good analysis, findings and insights based on their framework. The insights are valuable.\n2. The author provide decent amount of experimentation results on many VLMs. The author demonstrate the soundness and effectiveness of their framework with experimentations.\n3. Prism can be useful in both evaluation and task solver.", "weaknesses": "1. There is not much unique and novel contributions in terms of algorithms and model designs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere gratitude to the reviewer for the constructive feedback. We are glad that the reviewer appreciates the \"good analysis, findings, and insights\" presented in this work and recognizes the potential of the Prism framework in both evaluation and as a task solver. We address the concerns of the reviewer below:\n\nQ1. **Lacks Novel Contributions in terms of algorithms and model design:**\n\nIn this work, we aim at designing **a straightforward framework to disentangle and analyze the perception and reasoning capability** of VLMs. In pipeline design, we come up with the **query-agnostic and query-specific settings** to present more comprehensive evaluation results. When developing the PrismCaptioner, we also aim at **providing the simplest possible baseline** to validate the hypothesis that Prism equipped with a small-scale VLM and a large-scale LLM can potentially be **a powerful multi-modal task solver**. Thus, we select the most widely adopted LLaVA architecture and simply adopt ALLaVA as the instruction tuning corpora. Developing more sophisticated tuning algorithms or more advanced model architecture would improve the Prism framework's overall performance, but it's not our focus, and we leave it to future works.\n\nQ2. **Error Mode and Hallucination**\n\nWe conduct a thorough analysis and categorize errors in perception and reasoning into the following major modes (the content will be added to the refined version of this manuscript):\n\n1. **Major Error Modes (Perception):**\n   - **Factual Errors:** VLMs may describe images with inaccuracies, such as stating that prominent elements are \"not visible\".\n   - **Incomplete Details:** Even in the absence of factual errors, VLMs may lack detailed content, resulting in insufficient information for reasoning.\n   - **Instruction Following:** VLMs sometimes fail to follow instructions when providing corresponding descriptions.\n2. **Major Error Modes (Reasoning):**\n   - **Logical Errors:** LLMs may produce incorrect conclusions or reasoning processes due to limited reasoning abilities.\n   - **Lack of Knowledge:** The absence of relevant domain knowledge prevents LLM from solving corresponding problems, especially in specific fields.\n   - **Misunderstanding of the Query**: In rare cases, the query-specific part generated by LLMs deviates from the original question, misleading the perception of VLMs.\n\nRegarding the issue of hallucinations, identifying their source is crucial in our cascading framework. We can leverage various powerful LLMs to reason about the descriptions generated by a VLM and analyze the patterns in their reasoning results. Here are three potential scenarios.\n\n1. If most LLMs yield the correct answer while one LLM yields an incorrect one, the latter may struggle with reasoning or experience a hallucination.  \n2. If most LLMs indicate a wrong answer, it likely means that hallucinations generated by the VLM cause some misdirection. \n3. If the answers of the LLMs are chaotic or if there are refusals to answer, it may indicate that the VLM did not provide sufficient detailed information.\n\nWe conduct experiments with various VLMs, by selecting a set of powerful language models, including GPT-3.5-Turbo-0125, GPT-4-Turbo-0125, Llama-3-70B-Instruct, and DeepSeek-v2-Chat. For each question, we gather results from four LLMs and consider cases where the same choice appears three times or more as \"agreement\". In agreement cases, we define the corresponding choice as \"voted choice\" and focus on the following cases:\n\n- Case 1: With descriptive texts generated by a specific VLM, the voted choice is wrong.\n- Case 2: With descriptive texts generated by a specific VLM, the voted choice is correct.\n- Case 3: For a specific LLM, it makes the same choice as the voted choice by all LLMs, while the voted choice is also the correct one. \n\nFor each VLM, we analyze the rate of case 1, which indicates the potential hallucinations of VLMs. For each LLM, we correspondingly calculate the rate of case 3 to case 2 to observe the alignment of its predictions with voted choices. The higher alignment rate means more robust reasoning since voted options are considered better. The notions and results are as follows.\n\n$$\n\\text{Agreement Rate (VLM)} = \\frac{\\text{number of agreement cases}}{\\text{number of all cases}}\n$$\n\n$$\n\\text{Error Rate (VLM)}=\\frac{\\text{number of case 1}}{\\text{number of agreement cases}}\n$$\n\n$$\n\\text{Alignment Rate (LLM)}=\\frac{\\text{number of case 3}}{\\text{number of case 2}}\n$$\n\n| VLM | Agreement Rate (VLM) | Error Rate (VLM) |\n| -- | -- | -- |\n| GPT-4o | 68.6 | 29.9 |\n| GPT-4v | 61.5 | 38.0 |\n| LLaVA-NeXT (Yi-34B) | 60.7 | 43.4 |\n| LLaVA-v1.5-7B | 50.6 | 56.5 |\n\n| LLM                  | Alignment Rate (LLM) |\n| -------------------- | ---- |\n| GPT-3.5-Turbo-0125   | 87.1 |\n| DeepSeek-v2-Chat     | 91.3 |\n| Llama-3-70B-Instruct | 92.8 |\n| GPT-4-Turbo-0125     | 90.7 |\n\nThe results show that stronger VLMs show lower error rates, indicating that more capable VLMs experience fewer hallucination issues. By delving deep into the specific cases, we find that VLMs are prone to hallucinations in spatial awareness and fine-grained perception.  \n\nAll LLMs show relatively good alignment rates, demonstrating relatively robust reasoning performance. GPT-3.5-Turbo-0125 is more unstable by comparison. The cases where LLM's prediction does not align with the voted choice may stem both from reasoning ability issues and hallucinations. It requires further manual checking and labeling to clarify whether the failure in alignment is caused by LLM's hallucination.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3eGC7UJZcT", "reviewer": "Reviewer_rB4V", "summary": "The paper introduces Prism, a framework designed to decouple and independently assess the perception and reasoning capabilities of VLMs. Prism operates in two stages: a perception stage that extracts visual information and converts it into text using a VLM, and a reasoning stage that generates answers based on this textual information using a LLM. This modular approach allows for systematic evaluation of each component. Several insightful results are presented through extensive experiments.", "questions": "* Another line of work focuses on assessing the compositional or fine-grained understanding abilities of VLMs, such as [1][2][3][4]. Could you discuss the relevance or differences of these works to Prism?\n* Can you discuss the advantages and disadvantages of the 'end-to-end' approach versus the 'decoupling' approach?\n\n\nMy overall judgment of this article is positive. I am open to raising my score if the author can address my concerns listed above.\n\n****\n[1] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality, CVPR 2022 \n\n[2] Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding, CVPR 2024\n\n[3] Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View, ICML 2024\n\n[4] CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples, ACL 2024", "limitations": "Maybe the authors need to discuss potential limitations and future directions for improvement.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* the paper is well-organized and easy to follow\n* the experiments are extensive and the analysis provides some valuable insights\n* the Prism is effective and demonstrates competitive performance on several benchmarks\n* decoupling the end-to-end inference into perception and reasoning stages bring a new approach to solve the tasks that require complex reasoning process", "weaknesses": "* The prompt design for the perception and the reasoning stage is an important aspect. The impact of different prompt designs should be analyzed.\n* Decoupling the inference into two stages might introduce additional computational overhead and latency, and cost-effectiveness needs to be discussed.\n* Prism relies on language descriptions; however, in some scenarios, language struggles to describe the content of images because certain visual or logical concepts lack corresponding linguistic expressions, e.g. medical pathology images and graphs of mathematical functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging comments and address the main concerns below. Limitations and broader impacts have been discussed in Appendix. D. We are inspired that the reviewer believes our work \"provides some valuable insights\" and \"brings a new approach to solve the tasks that require complex reasoning process.\" These are indeed what we want to emphasize.\n\nQ1. **Prompt Design**\n\nWe carefully designed the prompts used in the Prism framework:\n\n1. **Generic Instruction (Perception):** The generic instruction provides common perspectives when describing images. We refer to MathVista and consider scenes, objects, relationships, and text. We further add instance location into the instruction and obtain better results. We experiment with variants of generic instructions and choose the relatively better one, as shown in Table 2.\n2. **Query-Specific Instruction (Perception):** We incrementally construct the query-specific instruction by merging the generic instruction with the query-specific part. Generating the query-specific part is a reasoning task assigned to LLM. Meanwhile, it is essential to ensure that only the `contents to observe` appear in the instruction since redundant content (questions, etc.) can stimulate the reasoning capability of VLMs. We design precise tasks and add in-context examples for the LLM, as shown in Fig. 7.\n3. **Instruction for Reasoning:** Visual descriptions and questions can be lengthy and complex, which easily leads to confusion for LLMs. We prompt the LLM to act like a text-based reasoning expert to derive the answer based on the provided information, which shows the best results during our iterations. The prompt template is shown in Fig. 8.\n\nQ2. **Computational Cost and Latency** \n\n1. **Training Cost**: For a large end-to-end VLM (like VLM based on Llama3-70B), the training cost is extremely high regarding hardware resources and time. Prism offers an alternative which only requires training a lightweight VLM as the perception module, significantly reducing the VRAM usage (GPUs with 40/80GB VRAM are still scarce and expensive) and the training computation. Prism doesn't tune the LLM. The total training cost of Prism is much less than that of an end-to-end VLM equipped with large language encoders (like Llama3-70B).\n2. **Deployment Cost**: The user spends much less cost to deploy a lightweight visual captioner compared to a large-scale VLM. On the LLM side, thanks to the advanced deployment techniques of LLMs, the LLM inference APIs are available at an extremely low price (also see Appendix D), so the user can take advantage of vast numbers of LLM APIs with low financial cost. \n3. **Latency**: Prism requires a VLM to generate intermediate descriptive text, bringing additional computation overhead. When solving a single visual question (VQ), a higher latency is expected compared to the end-to-end VLM with a language encoder of the same size. However, when one asks for multiple VQs corresponding to one image, Prism will display a smaller latency since the descriptive text can be reused. Moreover, the advanced LLM deployment techniques (compared to VLMs) further help reduce Prism's latency.\n\nQ3. **Lack of Linguistic Expressions for Some Images**\n\nIn Appendix. D, we mentioned that the current PrismCaptioner (learned on the general multi-modal corpora ALLaVA) may struggle to describe images in unseen domains (GUIs, medical images, etc.).   However, we do believe that almost all images can be described in detail by linguistic expressions. For example, a well-trained doctor can describe a medical image in detail with natural language (only noteworthy things, not all pixels, need to be described) and a graduate student can convert a screenshot of math equations into natural language descriptions or latex. With sufficient high-quality visual instruction tuning data, PrismCaptioner also has the potential to master these professional tasks.\n\nQ4. **Relevance and Differences of Referred Works to Prism**\n\nWe would incorporate those literature in related works and include a detailed discussion. The relevance and differences between Prism and these works lie in the following aspects:\n\n1. **Motivation**: Prism and referred works have different motivations: referred works mainly focus on evaluating visio-linguistic compositional knowledge or fine-grained understanding (mostly perception tasks), while Prism focuses on analyzing the perception and reasoning abilities of VLMs in a decoupled manner.\n2. **Methodology:** Prism and referred works all establish an evaluation system to analyze the capabilities of interest. [1, 2, 4] construct benchmarks to evaluate certain capabilities, [3] establish a new paradigm to study the compositional knowledge of VLMs, while Prism creates a decoupling framework to study the perception and reasoning capabilities, respectively.\n3. **Application:** From the application perspective, Prism can not only serve as an evaluation framework but also effectively address visual language tasks.\n\nQ5. **Decoupling Versus End-to-End (e2e)**\n\n1. **Cost and Latency**: As discussed in Q2, to reach similar performance, the decoupling paradigm features a smaller training and deployment cost (VRAM usage, hardware resources, financial costs, etc.) compared to the e2e approach. For a single visual question, Prism shows a higher latency compared to the e2e VLM. Meanwhile, the average latency of Prism can be lower when Prism deals with a batch of VQs corresponding to the same image.\n2. **Performance**:  With abundant computation resources and training data to train a large-scale VLM (equipped with large language encoders such as Llama3-70B), adopting the decoupling paradigm may not be a good choice for the final performance. However, under resource-limited (computation, training data, financial budget, etc.) scenarios, Prism can outperform e2e VLMs regarding performance and flexibility (please refer to Tables. 5, 6).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqjDylSMLj", "reviewer": "Reviewer_onkG", "summary": "This paper introduces Prism, a framework to disentangle and evaluate the perception and reasoning capability of vision-language models (VLMs). They evaluate state-of-the-art VLMs, including proprietary ones and open-source ones, with varying model sizes. The evaluation results demonstrate that VLMs' perception ability is consistent regardless of the language encoder size while the reasoning ability is constrained by the model size. Moreover, they develop a vision-language model based on the discovered principles, which is lightweight yet effective.", "questions": "1. To evaluate a VLM's perception capability, Prism uses the VLM as an image captioner and feeds the generated caption to an external LLM to answer the question. The final accuracy can indicate how accurately the VLM can describe the image. However, evaluating the VLM on image captioning benchmarks can also reflect the perception ability and is more straightforward than the proposed framework. What are the advantages of Prism over image captioning benchmark evaluation?", "limitations": "Limitations and broader impacts are discussed in Appendix D.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel framework to evaluate disentangled perception and reasoning abilities of VLMs.\n2. The paper conducts thorough experiments to validate the framework and evaluate state-of-the-art VLMs. Various proprietary and open-source VLMs with different model sizes are covered.\n3. Based on experimental conclusions, a lightweight yet effective VLM is developed, showing the soundness of the conclusions.", "weaknesses": "1. In the perception capability evaluation, besides accuracy, how detailed the image captions given by the VLMs are can also affect the final accuracy. As different VLMs are pre-trained on different datasets for different down-stream purposes, some of them might be trained to give concise response, leading to low final accuracy in Prism. However, giving concise response does not necessarily mean the models cannot see the other details. Therefore, Prism might be inaccurate to evaluate their perception capability.\n2. Different VLMs are pre-trained on different datasets, which could cause their differences in perception and reasoning capability. It would provide more insights if the paper investigates the relationship between the models' pre-training datasets and their perception and reasoning capabilities.\n3. The results of query-specific instruction in perception ability evaluation is not only affected by the models' capability in perception, but also their capability in instruction-following, e.g., the second example in Figure 3. It would be more accurate to reflect the perception capability by first checking whether the model response follows the instruction, e.g., by using an external LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback. We are uncertain if there is some misunderstanding and would like to clarify that **VLMs** discussed in our paper specifically refer to **large visual language models (LLaVA, GPT-4v, etc.)** designed for **solving general visual language tasks** (as stated in Line 23 and Line 26). We are encouraged that the reviewer believes our work \"proposes a novel framework to evaluate disentangled perception and reasoning abilities of VLMs\". We address the reviewer's main concerns below.\n\nQ1. **The Evaluation Might Be Inaccurate**\n\nThe reviewer suggests that Prism evaluation might be unfair for VLMs that are pre-trained for specific purposes and generate concise responses. However, we would like to clarify that such a premise doesn't align with the assumption of this work. We adopt the Prism framework to evaluate **large visual language models (LLaVA, GPT-4v, etc.)** designed for **solving general visual language tasks**. As stated in Line 27, we define the perception capability as \"extracting necessary information from the image\", which is beyond \"seeing\" (we are not sure if a VLM can \"see\" if it cannot clearly express what it sees). By design, VLMs for general tasks should be able to follow the instructions and express detailed information, regardless of response patterns or training strategies. Thus, we believe that perception evaluation under the Prism framework is a reasonable approximation of true perception capacity.\n\nQ2. **Effects of Pre-training Dataset on VLM's Abilities**\n\nWe adopt the LLaVA architecture with InternLM2-7B to investigate the relationship between VLM's pre-training datasets and their perception abilities under the Prism evaluation framework. We construct four different datasets based on ALLaVA dataset to study the impact of descriptive and text-only data: \n\n- **ALLaVA-Caption:** a set of all descriptive data in ALLaVA.\n\n- **ALLaVA-QA:** a set of all the QA data used for instruction fine-tuning in ALLaVA with the same amount and images as ALLaVA-Caption.\n\n- **ALLaVA-Caption-1xText:** a mix of one copy of text-only data in ALLaVA with ALLaVA-Caption, with a ratio of 1:5 (text-only:image-text)\n\n- **ALLaVA-Caption-2xText:** a mix of two copies of text-only data in ALLaVA with ALLaVA-Caption, with a ratio of 2:5 (text-only:image-text)\n\nThe performance of their perception abilities on MMStar is shown below.\n\n| Dataset | Perception Performance (Generic) | Perception Performance (Query-Specific) |\n| -- | -- | -- |\n| ALLaVA-Caption        | 40.9  | 43.6 |\n| ALLaVA-QA             | 38.3  | 37.0 |\n| ALLaVA-Caption-1xText | 41.7  | 44.3 |\n| ALLaVA-Caption-2xText | 40.9  | 44.5 |\n\nThe results of ALLaVA-Caption and ALLaVA-QA reveal that utilizing the descriptive data better triggers the VLM’s ability to extract and articulate visual information compared with QA data. Focusing on text-only data, one can observe that training with a small amount of text data can improve the VLM's ability to extract and express information. When the ratio of text data increases, VLM shows a degradation in the perception performance of generic instructions, indicating that an appropriate data recipe is crucial for VLM's perception capabilities. \n\nAs for the reasoning aspect of VLM, it is difficult for Prism to give a precise metric to quantify it. Prism evaluation framework is more about assessing whether the overall performance of the VLM is limited by its reasoning capabilities. We will further investigate the particular impact of training data on reasoning in subsequent work.\n\nQ3. **Instruction Following Failures**\n\nExcluding all cases of instruction following failures can potentially be a way to assess the true perception capabilities more accurately. However, such practice is rarely adopted. First of all, if a VLM fails to extract the relevant information according to given instructions, it's difficult to know whether the issue lies with the instruction following or the VLM's perception capabilities. Meanwhile, most existing multi-modal benchmarks (MMMU, MathVista, etc.) do not make specific adjustments to mitigate the effect of instruction following.\n\nQ4. **Image Caption Benchmarks**\n\nDirectly evaluating the quality of image description can always be the best way to assess the perception capability. However, accurately evaluating image description quality is a significant challenge, and using traditional caption metrics is not a viable solution. Traditional image caption metrics are too sensitive to the caption styles and are not comprehensive. To validate this, we conduct thorough experiments on COCO Caption (val). The partial results are as follows.\n\n| Model | BLEU-1 | BLEU-4 | ROUGE-L | CIDEr |\n| -- | -- | -- | -- | -- |\n| Qwen-VL-Chat | 75.8 | 34 | 54.9 | 98.9 |\n| InstructBLIP-7B | 56.8 | 20.9 | 39.9 | 58.1 |\n| GPT-4o | 21.2 | 3.9 | 20.3 | 0 |\n| GPT-4v | 18 | 3.3 | 18.1 | 0 |\n| InternVL-Chat-v1.5 | 15.9 | 3 | 15.8 | 0 |\n| LLaVA-Next-Yi-34B  | 12.8 | 2.4 | 13.1 | 0 |\n\nExcellent VLMs, such as GPT-4v, GPT-4o, and InternVL-Chat-v1.5, struggle to score well, with all metrics significantly lower than the low-profile Qwen-VL-Chat (the reason is that those advanced VLMs generate much longer responses compared to the ground-truth caption). This contradicts the consensus and further indicates that the caption benchmarks are unsuitable for assessing VLMs' perception abilities. In Prism, we use an external LLM to answer the question based on the image description generated by VLMs to evaluate the VLM's perception capability. Compared to the caption evaluation, the score obtained with Prism is more comprehensive and less sensitive since whether an LLM can successfully answer the question is basically determined by the quality of the descriptive text, not the style. Moreover, existing image caption benchmarks are often limited to specific domains. In Contrast, Prism can be applied to any general multi-modal benchmark to study the VLM's capability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Systematic analyzing VLMs is still a challenging and open research topic, the authors presented an aspect of doing it through decoupling framework. All reviewers and AC hold positive view of this work, and suggest accept.
Decision: Accept (poster)