Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs

Authors: yuxuan  qiao
haodong  duan
xinyu  fang
junming  yang
lin  chen
songyang  zhang
jiaqi  wang
dahua  lin
kai  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 07:25:36

Abstract: Vision Language Models (VLMs) demonstrate remarkable proficiency in addressing a wide array of visual questions, which requires strong perception and reasoning faculties. Assessing these two competencies independently is crucial for model refinement, despite the inherent difficulty due to the intertwined nature of seeing and reasoning in existing VLMs. To tackle this issue, we present Prism, an innovative framework designed to disentangle the perception and reasoning processes involved in visual question solving. Prism comprises two distinct stages: a perception stage that utilizes a VLM to extract and articulate visual information in textual form, and a reasoning stage that formulates responses based on the extracted visual information using a Large Language Model (LLM). This modular design enables the systematic comparison and assessment of both proprietary and open-source VLM for their perception and reasoning strengths. Our analytical framework provides several valuable insights, underscoring Prism's potential as a cost-effective solution for vision-language tasks.
By combining a streamlined VLM focused on perception with a powerful LLM tailored for reasoning, Prism achieves superior results in general vision-language tasks while substantially cutting down on training and operational expenses. Quantitative evaluations show that Prism, when configured with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on par with VLMs $10 \times$ larger on the rigorous multimodal benchmark MMStar.

Main paper: 
 introduction: 
 With the rapid development of Large Language Models (LLMs) [45, 59, 60, 56, 5, 19, 4] , Vision Language Models (VLMs) [46, 55, 6, 36, 16, 18] have also experienced significant advancements. As an end-to-end approach, VLMs trained on large-scale multimodal data [36, 11, 51, 8] exhibit superior performance on a variety of tasks. These tasks range from basic ones such as object localization [31] and optical character recognition [44, 38] to more complex challenges like document or diagram comprehension [26, 43, 42] and solving geometric problems [40] . For VLMs to solve a general visual question, two essential capabilities are required: 1) Perception: extracting necessary information from the image; 2) Reasoning: generating answers based on the extracted information and contextual understanding. Limitations in either capability can impede the overall performance of a VLM.A systematic evaluation of the perception and reasoning capabilities is crucial to provide valuable insights for future model optimization. However, seeing and reasoning are mostly entangled in existing VLMs. Proprietary VLMs, such as GPT-4v are opaque systems that only provide the final answer to visual questions. Meanwhile, open-source VLMs [36] commonly utilize vision encoders to extract visual embeddings, which are often difficult to interpret, from the image and employ adapted LLMs to generate answers based on these visual and linguistic embeddings. In light of this challenge, we introduce Prism, a framework designed to disentangle the perception and reasoning processes of any given VLM. It can serve as a proxy for assessing the real perception capabilities of VLMs.Prism decomposes the process of solving general visual questions into two distinct stages: 1) a perception stage that concentrates on extracting visual information from the image using VLMs and articulating this information in textual form; and 2) a reasoning stage that utilizes LLMs to answer the question based on the extracted visual information. Prism facilitates the analysis of VLMs' capabilities through two approaches. To assess the true perception capabilities, one can use a constant LLM in the reasoning stage while testing various VLMs in the perception stage. Conversely, by keeping the VLM fixed and varying the LLM in the reasoning stage, one can determine whether a VLM's performance is limited by its reasoning capabilities. Through this analysis, we have uncovered several key insights: 1) Proprietary VLMs, such as GPT-4o and GPT-4v, take the lead in the perception capabilities competition; 2) For open-source VLMs, perception capabilities remain relatively consistent regardless of the language model's size; and 3) The overall performance of open-source VLMs, particularly those with smaller-scale language models like 7B variants, is often constrained by the limited reasoning capabilities.Beyond its role as an evaluation framework, Prism also excels as an efficient general Vision-Language Model (VLM). Building upon findings 2 and 3, we posit that integrating a small-scale VLM as a visual captioner with a powerful LLM as a reasoning engine offers a promising and efficient strategy for general vision-language tasks. By concentrating on visual information extraction, a lightweight VLM can achieve decent performance on par with much larger VLMs. When paired with a powerful yet economical LLM, thanks to advancements in deployment techniques, one can achieve a robust solution for visual-language information processing, requiring significantly fewer hardware resources for training and deployment. In our experiments, we trained an approximately 2B-parameter vanilla LLaVA to extract visual information and observed that it exhibits perception performance comparable to LLaVA-NeXT [35] that is equipped with a 34B powerful language model. Quantitative evaluations indicate that Prism, when instantiated with a streamlined visual captioner and the freely available ChatGPT-3.5, outperforms many open-source VLMs on multiple multimodal benchmarks including the stringent multimodal understanding benchmark MMStar [10] . Notably, the advantage is particularly pronounced on reasoning-related visual questions.In summary, the contributions of this work are as follows:1. We introduce Prism, a highly adaptable framework designed to explicitly disentangle the perception and reasoning processes. Prism enables the breakdown analysis of VLM capabilities and serves as a solution for vision-language tasks by integrating any given VLM and LLM. 2. Utilizing Prism, we conduct a decoupled analysis of the perception and reasoning capabilities of existing VLMs. Several intriguing findings emerge from the ablation study.3. Drawing inspiration from these findings, we integrate a lightweight VLM focused on perception with a powerful LLM dedicated to reasoning within the Prism framework. Quantitative results demonstrate that this combination exhibits outstanding performance and efficiency across a range of vision-language tasks.2 Methodology 
prism architecture: 
 Prism is characterized by a modular design that decomposes the process of solving visual questions into two stages, consisting of a perception module and a reasoning module, both of which can be flexibly replaced, as depicted in Fig. 1 . The perception module, typically a VLM, initially follows the instruction to extract visual information from images and articulates this information in textual form. The instruction can be generic or query specific, ie., written by the reasoning module given the question (text-only) as contextual information. Meanwhile, the reasoning module, usually an LLM, performs text-based reasoning on the textual information to generate answers to the questions. To assess the perception capabilities of various VLMs, we carefully select an appropriate benchmark corresponding to the principle we illustrate in Sec. 2.2. In Sec. 2.3, we describe how we utilize the Prism framework to assess the perception and reasoning capabilities of VLMs, respectively. To substantiate our belief in the potential of combining a small-scale VLM with a powerful LLM, we train an approximately 2B-parameter VLM based on the LLaVA architecture to serve as a visual captioner and integrate it into Prism as the perceptual module, as detailed in Sec. 2.4.Numerous multimodal benchmarks [37, 20, 28, 40, 66, 10] exist, each evaluating the capabilities of VLMs from various perspectives. To maximize the utility of the Prism framework, careful selection of the benchmark is imperative for the decoupling analysis. In summary, we adhere to the following principles for benchmark selection: 1) Vision Indispensability: The benchmark must require visual information for question solving, and questions that can be answered without utilizing visual information are excluded; 2) Minimal Data Leakage: The visual questions should not be part of the model's training data; and 3) Complexity: The solving process of the visual question should involve both perception and reasoning components. Considering these three principals, we select MMStar [10] as the primary benchmark for the decoupling analysis and ablation. MMStar ensures vision indispensability and makes a concerted effort to minimize data leakage, while many other benchmarks are plagued by the two issues.Prism functions as an analytical framework for evaluating the perception and reasoning capabilities of a given VLM. In this section, we elaborate on the methodology for evaluating these capabilities.To assess the perception capabilities of various VLMs, we employ ChatGPT (GPT-3.5-turbo-0125) as the reasoning module and standardize instructions for description within Prism. With the reasoning module and instructions fixed, the final accuracy of the visual questions is solely determined by the quality of the visual information extracted. Under the controlled setting, we consider the VQA accuracy as a proxy to measure the perception capability of VLMs.Within the Prism framework, the instructions for visual information extraction are crucial as they are designed to elicit the fundamental perceptual capabilities of VLMs. We have adopted two types of instructions to assess the perceptual abilities of models:1. Generic Instruction: a standardized, universal instruction aimed at extracting and describing the basic elements present in an image; 2. Query-Specific Instruction: a combination of the generic instruction and an incremental instruction that directs the VLM to provide a detailed account of the visual information relevant to the question. The incremental instruction is crafted by the reasoning module given the text-only question.Employed with the generic instruction, Prism offers a straightforward decoupling pipeline. Meanwhile, using query-specific instructions is a more realistic setting and can better realize the full potential of the Prism framework. A Prism decoupled GPT-4o (GPT-4o for both perception and reasoning) achieves almost the same quantitative performance compared to the end-to-end GPT-4o.Analyzing the results, we find that with the reasoning module and instruction controlled, VLMs with language models of different sizes display a much narrowed gap in perception performance compared to the results under the end-to-end setting.Besides evaluating the perception performance, Prism can also roughly measure the reasoning capabilities of VLMs. When used to solve general visual questions in an end-to-end manner, the VLM implicitly performs reasoning with its language encoder. Alternatively, Prism provides an explicit pipeline, in which an off-the-shelf LLM is separately used to predict the answer based on the visual information. For small-scale VLMs (7B-parameter, etc.), we find that Prism equipped with the VLM and ChatGPT-3.5 outperforms the end-to-end VLM in quantitative performance, especially for reasoning related VQA. The results reveal that for small-scale VLMs, the overall performance can be heavily constrained by the parameter size of the language model.Beyond serving as an evaluation framework, Prism can also function as an efficient vision-language task solver. The perception module in Prism can incorporate one or multiple VLMs to extract high-quality visual information. Concurrently, the reasoning module can be instantiated with a powerful LLM to harness its advanced reasoning capabilities.In Sec. 2.3, we observed that VLMs paired with language models of varying sizes exhibit similar performance in perception capability. This suggests that it is promising to employ small-scale VLMs to generate informative visual descriptions, serving as an efficient perception module. To validate this concept, we conduct extensive experiments to train visual captioners, utilizing the widely adopted LLaVA architecture and only open-source datasets. Below, we elaborate on the specific settings considered for this ablation study in detail:Instruction Tuning Data. We use ALLaVA-Caption-4V and Evol-Intruct-GPT4-Turbo-143K in ALLaVA [9] as our instruction tuning data. The former comprises 715K pairs of images and detailed visual captions produced by GPT-4v, whereas the latter contains 143K text instruction tuning data, generated by GPT-4-Turbo. Utilizing the descriptive data for instruction tuning better triggers the VLM's ability to extract and articulate visual information more effectively, compared to instruction tuning data in QA formats [36, 32, 52, 42] .Model Architecture. To investigate the impact of vision encoder in LLaVA, we experimented with multiple encoders, including Clip ViT-L/14 [50] , SigLip-SO400M [67] , and InternViT-6B [13] .For the language encoder, we tested two lightweight variants of InternLM2 [7] : InternLM2-7B and InternLM2-1.8B. The inference of all combinations (excluding InternViT-6B) can be efficiently executed on consumer-level GPUs such as RTX 4090, etc. During instruction tuning, we maintain the vision encoder fixed and apply QLoRA [17] to the language encoder.Regarding the vision backbone, SigLip exhibits relatively superior performance. Additionally, we observed that a larger language model only results in minor differences in perception performance.Our findings indicate that, within the Prism framework, a 2B vision captioner can achieve strong perception performance on par with LLaVA-NeXT [35] equipped with a 34B language backbone.3 Evaluation ResultsThe rug is centrally placed in front of the sofa, extending slightly under it, and covers a portion of the wooden floor. The side table is positioned between the sofa and the door, with the lamp and plant on top providing both functional and decorative elements. The framed art is centered above the side table, adding a pop of color to the wall. ... Overall, the arrangement of furniture and decor items creates a balanced and aesthetically pleasing living room corner. The rug's placement in relation to the sofa and side table helps define the seating area and adds warmth to the space.The image shows a neatly arranged living room corner with a focus on a side table and a portion of a couch. The side table, which appears to be made of wood with a natural finish, has two drawers with metal handles. On top of the table, there is a large blue lamp with a white lampshade, a small white bowl, and a green potted plant with broad leaves. To the left of the side table, there is a couch with a grey upholstery, adorned with a teal throw pillow and a pink throw blanket casually draped over it. The floor is covered with a grey and white patterned area rug, and the walls are painted in a light grey tone. A window with white trim is partially visible, dressed with patterned curtains in shades of grey and white. On the wall, there is a framed artwork featuring a bouquet of flowers in a vase. The door in the background is a dark grey with a black doorknob, and there is a light switch plate visible on the wall to the right of the door. The overall aesthetic of the room is modern with a touch of warmth provided by the wooden furniture and the cozy textiles.Predictions D. Under the couch ( ) C. In front of the window ( ) Justification: As denoted by the abstract and introduction, in this work, we present the Prism framework that can serve as an analytical framework and a general vision-language task solver. Qualitative and quantitative results in this paper can support the contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations of Prism are discussed in Appx. D.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
implementation details: 
 Evalution Details. We use Prism to evaluate the capabilities of various VLMs, which can be categorized into two major groups: (a) Proprietary VLMs, including GPT-4o (20240513) [46] , GPT-4v (20231106) [46] , GeminiPro-V [55] , and Qwen-VL-Max [6] ; (b) Open-Source VLMs, including LLaVA-v1.5 [34] , InternLM-XComposer2 [18] , mPLUG-Owl2 [65] , LLaVA-NeXT [35] , InternVL-Chat-v1.5 [12] , DeepSeek-VL [39] , MiniCPM-V-2 [47] . When integrating these VLMs as perception modules within Prism, we employ greedy decoding and limit the maximum number of output tokens to 512. The evaluation encompasses both generic and query-specific instructions. Unless otherwise specified, GPT-3.5-Turbo-0125 is adopted as the reasoning module. All evaluations are conducted using VLMEvalKit [14] . Further details on the evaluation are provided in Appx. B.1.Post-Processing. Within the Prism framework, the LLM (particularly proprietary APIs) in the reasoning module often declines to answer questions due to insufficient clues in the extracted visual information. When Prism is employed as an evaluation framework, we classify this as a failure of perception for the visual question and refrain from any post-processing. For fair comparisons, whether Prism is contrasted with other VLMs as a vision-language task solver or different LLMs are evaluated as reasoning modules with varying rejection rates, a random choice is utilized as a fallback when option matching fails.We present the primary evaluation results of various VLMs' perception performance in Tab. 1. Results are categorized under two instruction types: Generic Instruction and Query-Specific Instruction .Generic Instruction Results. When using generic instructions, GPT-4o exhibits exceptional performance across a variety of tasks. InternVL-Chat-v1.5 achieves the highest overall perception performance among open-source VLMs, nearly on par with the proprietary GPT-4v. LLaVA-NeXT (Yi-34B) demonstrates strong coarse and fine-grained perceptual abilities but lags behind GPT-4v in recognizing abstract elements in LR and ST visual questions. Smaller open-source VLMs, such as mPLUG-Owl2 and LLaVA-v1.5-7B, struggle with both coarse and fine-grained perception and encounter difficulties in recognizing essential elements in reasoning-related VQA.Query-Specific Instruction Results. GPT-4o demonstrates significantly superior performance compared to all other models in the extraction and expression of visual information across all dimensions. GPT-4v is equally adept at handling coarse perception contents. Among open-source VLMs, InternVL-Chat-v1.5 excels in perception tasks across all dimensions excluding LR. Its performance not only surpasses that of other open-source VLMs but is also slightly ahead of GPT-4v. Occasionally, a decline in performance can be observed for specific VLM-task pairs when using query-specific instructions instead of generic ones, which can stem from difficulty in understanding query-specific instructions for specific domains.Are Proprietary Models Getting Ahead of the Game? When employing both generic and queryspecific instructions, proprietary VLMs, particularly GPT-4o, significantly surpass other models in perceptual capabilities and can adeptly manage a wide range of tasks, as demonstrated in Tab. 1. Certain open-source models, such as InternVL-Chat-v1.5 and LLaVA-NeXT (Yi-34B), have achieved Other open-source models, due to their limited perceptual abilities, generally perform slightly worse in Math and ST assessments. Notably, MiniCPM-V-2, a lightweight VLM with ∼3B parameters, displays better perceptual performance compared to some 7B VLMs.G P T -4 o I n t e r n V L -C h a t -v 1 . 5 I n t e r n L M -X C o m p o s e r 2 L L a V A -N e X T ( Y i -3 4 B ) G P T -4 v Q w e n -V L -M a x D e e p S e e k -V L -7 B L L a V A -N e X T ( V i c u n a -1 3 B ) M i n i C P M -V -2 G e m i n i P r o -V L L a V A -N e X T ( M i s t r a l -7 B ) L L a V A -N e X T ( V i c u n a -7 B ) m P L U G -O w l 2 L L a V A -v 1 . 5 -1 3 B L L a V A -v 1 . 5 -7 BThe Gap between Perception Ability and End-to-End Performance. In addition to solving visual questions in an end-to-end manner, Prism provides an alternative pipeline where the VLM is solely utilized for perception. The distinction between these two methods lies in the reasoning process: the former conducts reasoning internally within VLMs, whereas the latter performs reasoning based on VLM extracted information using an external LLM (ChatGPT). The comparison between these two approaches on MMStar is depicted in Fig. 2 . For state-of-the-art large-scale VLMs such as GPT-4o and InternVL-Chat-v1.5, which are expected to possess excellent reasoning capabilities, employing an external ChatGPT for reasoning may diminish overall performance. Conversely, for most small-scale VLMs, using ChatGPT for reasoning significantly improves their performance, particularly in reasoning-related VQA, as shown in Fig. 5 . This phenomenon indicates that the overall performance of small-scale VLMs can be heavily constrained by the size of the language model. To investigate whether the reasoning ability of ChatGPT constrains state-of-the-art VLMs, we implemented a Prism pipeline that decouples GPT-4o by using it as both the perception and reasoning module. The result reveals that this Prism pipeline, with post-processing, achieves an overall accuracy of 61%, nearly identical to the end-to-end GPT-4o performance of 61.6%.How does Language Model Size Affect Perception Ability? During evaluation, we observe that the LLaVA-v1.5 series shows no significant improvement when using a larger language model (Vicuna-13B instead of Vicuna-7B, etc.). This suggests that perception performance may be independent of the language model size when using a relatively low-resolution vision backbone. However, the situation appears to differ with LLaVA-NeXT. Quantitative results for the LLaVA-NeXT series tell that scaling up the language model slightly enhances model perception, particularly when using query-specify instructions. Through a detailed qualitative analysis, we identified the primary factors contributing to the superior performance of larger LLaVA-NeXT models over smaller ones as follows:(1) More Elaborate Expression: Models equipped with a larger language encoder exhibit enhanced ability to articulate visual information. More detailed and organized narratives make it easier for the reasoning module to answer the question; (2) More Adaptive to Instruction: Larger language backbones entitle the model with a better understanding of instructions, yielding more suitable textual visual information for reasoning, particularly in response to query-specific instructions. In Fig. 3 , we provide some qualitative results about the two typical modes.Error Mode. We conduct a thorough analysis and categorize errors in perception and reasoning into the following major modes:1. Major Error Modes in Perception: (1) Factual Errors. VLMs may describe images with inaccuracies, such as stating that prominent elements are "not visible"; (2) Incomplete Details. Hallucination. To reveal the source of hallucinations, we employ various LLMs to reason about descriptions generated by the given VLM and analyze the patterns in their results. We conduct experiments on various VLMs with a set of powerful language models, including GPT-3.5-Turbo-0125, GPT-4-Turbo-0125, Llama-3-70B-Instruct, and DeepSeek-v2-Chat. For each question, we gather results from four LLMs and consider cases where the same choice appears three times or more as "agreement". In agreement cases, we define the corresponding choice as "voted choice" and focus on the following cases:Case 1: With descriptive texts generated by a specific VLM, the voted choice is wrong. Case 2: With descriptive texts generated by a specific VLM, the voted choice is correct. Case 3: For a specific LLM, it makes the same choice as the voted choice by all LLMs, while the voted choice is also the correct one.For each VLM, we analyze the rate of case 1, which indicates the potential hallucinations of VLMs. For each LLM, we correspondingly calculate the rate of case 3 to case 2 to observe the alignment of its predictions with voted choices. The higher alignment rate means more robust reasoning since voted options are considered better. The notions are as follows.Error Rate (VLM) = number of case 1 number of agreement cases Alignment Rate (LLM) = number of case 3 number of case 2 As presented in Tabs. 2 and 3, stronger VLMs show lower error rates, indicating that more capable VLMs experience fewer hallucination issues. By delving deep into the specific cases, we find that VLMs are prone to hallucinations in spatial awareness and fine-grained perception. All LLMs show relatively good alignment rates, demonstrating relatively robust reasoning performance. GPT-3.5-Turbo-0125 is more unstable by comparison. The cases where LLM's prediction does not align with the voted choice may stem both from reasoning ability issues and hallucinations. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Data, codes, and other resources required to reproduce the main results will be released later. We need some time to prepare a more tidy version of those resources.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
ablation study: 
 Ablation on Generic Instructions. Within Prism, the generic instructions for visual information extraction are crucial. We experimented with a variety of instructions to elicit the fundamental perceptual capabilities of VLMs, including human-written instructions, GPT-4 generated instructions, and those incorporating chain-of-thought [61] or explicit decomposition, as shown in Fig. 4 . We conducted an ablation study on MMStar using the state-of-the-art VLM GPT-4o and LLaVA-NeXT (Yi-34B). As illustrated in Tab. 4, Human 1 outperforms others in eliciting the fundamental perceptual capabilities of the models, while the differences among various instructions are not significant. Therefore, we adopt Human 1 as the generic instruction for all evaluations.Ablation on the Reasoning Module. The reasoning module is critical for accurately determining the correct answer based on the visual information. To evaluate the impact of the reasoning module on overall performance, we select four LLMs: GPT-3.5-Turbo-0125, GPT-4-Turbo-0125, Llama-3-70B-Instruct, and DeepSeek-v2-Chat, and assess these models across four VLMs with varying capacities. The comparative results are presented in Tab. 5. As a freely available model, GPT-3.5 demonstrates good reasoning performance under our framework, and the more advanced GPT-4 shows improved performance in line with other benchmarks. Notably, Llama3-70B-Instruct, as a representative of open-source models, exhibits competitive capabilities compared to GPT-4-Turbo-0125 under different perceptual conditions. This suggests that open-source models could be valuable for further exploration in visual reasoning.Ablation on the Vision Backbone. To investigate the impact of the vision encoder on perception ability within the LLaVA architecture, we conduct an ablation study on three pre-trained visual backbones, including CLIP ViT-L/14, SigLip-SO400M, and InternViT-6B. We use InternLM2-7B as the fixed language encoder in LLaVA and train the VLMs for one epoch with the vision encoder fixed. The results in Tab. 6 show that SigLip-SO400M achieves better performance compared to CLIP ViT-L/14 and InternViT-6B on MMStar. 
prismcaptioner: 
 Within Prism, we explore the use of small-scale Vision-Language Models (VLMs) as a perception module. We use SigLip as the vision encoder, InternLM2-[1.8B/7B] as the language encoder to develop two visual captioners at different scales, refered as PrismCaptioner-[2B/7B]. Table 6 : Ablation on the Vision Backbone.We perform one-stage training for two epochs using ZeRO2 with XTuner [15] on 8 NVIDIA A800-80GB GPUs, and the training lasts less than a day. The training data include of one copy of ALLaVA-Caption-4V and two copies of Evol-Intruct-GPT4-Turbo-143K. The batch size is set to 16 per device for PrismCaptioner-2B and 8 for PrismCaptioner-7B. We utilized the AdamW optimizer and 2e-4 learning rate, with the warm-up ratio set to 0.03 and (β 1 , β 2 ) set to (0.9, 0.999).No weight decay was applied and a maximum norm value of 1 is applied for gradient clipping. Full details about QLoRA are presented in Appx. B.2.We conduct a thorough evaluation of PrismCaptioners across multiple benchmarks, employing GPT-3.5-Turbo-0125 and Llama-3-70B-Instruct as the reasoning module. We utilize MMStar as our primary benchmark to assess comprehensive multimodal capabilities. For domain-specific evaluations, we choose AI2D to gauge diagram comprehension, MMMU for expert knowledge assessment, and MathVista to test mathematical proficiency. The results are presented in Tab. 7. In line with the benchmark selection principles outlined in Sec. 2.2, we apply a consistent filtering strategy across AI2D, MMMU, and MathVista, mirroring that of MMStar, to retain only vision-essential and uncontaminated questions, denoted by the suffix (F). In addition to comparisons with existing opensource and proprietary VLMs, we also assess two baseline models, LLaVA-InternLM2-[1.8B/7B], which are PrismCaptioners trained on LLaVA-v1.5 instruction tuning data. We also compare PrismCaptioners and ShareCaptioner, an open-source VLM designed for generating informative image captions, under identical Prism framework configurations.As depicted in Tab. 7, each PrismCaptioner, with an external potent LLM for reasoning, markedly surpasses its corresponding end-to-end baseline. PrismCaptioners also outperform ShareCaptioner across all multimodal benchmarks. For the 7B variant, the integration of Llama3 results in a substantial enhancement, positioning PrismCaptioner-7B as a highly competitive vision-language solver, particularly on MMStar and MMMU. For PrismCaptioner-2B, employing ChatGPT yields superior results, outperforming nearly all 7B VLMs in general aptitude, expert knowledge, and mathematical skills. Remarkably, it achieves performance levels on par with some ten times larger VLMs, such as LLaVA-InternLM2-20B, Yi-VL-34B, and Emu2-Chat. This demonstrates that Prism enables the creation of a robust yet efficient vision-language solver, exemplified by PrismCaptioner-2B with ChatGPT, which delivers impressive results. 
discussion: 
 Prism's Value as an Evaluation Framework. Prism's value as an Evaluation Framework lies in its ability to disentangle and measure the perception and reasoning capabilities of VLMs across various data sources. There exist specialized multimodal benchmarks designed to assess VLMs' perception and reasoning capabilities, yet they often focus on specific domains. For instance, RealWorldQA [62] evaluates real-world perception with high-resolution images, OCRVQA [44] assesses text recognition in publications, and POPE [30] determines object existence in images. However, many interested domains (geometry, medical images, GUIs, etc.) are not covered by those perception benchmarks. Prism fills this gap by enabling the measurement and comparison of VLMs' perception capabilities on By integrating small-scale VLMs with readily available external LLMs within Prism, we achieve superior performance compared to the standard endto-end capabilities of the standalone VLM. This approach also renders it practical to address vision-language tasks using a 2B parameter VLM, as reasoning is effectively outsourced to external LLMs. When implemented with an LLM API, Prism's inference process (without quantization) only consumes several gigabytes of GPU memory. Furthermore, Prism allows for the flexible incorporation of multiple VLMs to enhance perception. For instance, the straightforward concatenation of outputs from GPT-4v and GeminiPro-V has demonstrated substantial improvements across the majority of metrics on the MMStar benchmark, as substantiated by the data presented in Table 8 .The landscape of Large Language Models (LLMs) is continually evolving, with an expanding body of research focused on integrating multimodal capabilities to enhance their perceptual abilities in real-world contexts [49, 70, 29] . Early efforts in this direction, such as Flamingo [3] , introduced gated dense blocks of cross-attention within pre-trained language encoder layers to fuse visual features. Subsequent models like BLIP2 [29] and InstructBLIP [16] utilized a Q-former to align features across different modalities, enabling tasks such as zero-shot visual question answering. More recent models, including LLaVA [36] and MiniGPT-4 [70] , have simplified modality bridging through the use of MLP-based projection layers, d offering a more straightforward approach compared to the Q-former. The architecture of LLaVA has been widely adopted in subsequent works [35, 1, 12, 39] . The choice of vision encoders and the language model is considered critical for the overall performance of VLMS. Most VLMs [36, 18, 35, 27, 70] employ CLIP-based Vision Transformer [50, 67, 54] as the vision encoder, owing to its good pre-training alignment of visual and textual modalities. There is a prevalent belief [37, 35] that the scale of the language model significantly impacts the performance of VLMs, though detailed analyses are lacking. In addition to the open-source VLMs developed by the academic community, numerous proprietary VLMs [46, 55, 6, 48] demonstrate robust performance across various multimodal benchmarks. This paper presents a breakdown capability analysis of both open-source and proprietary VLMs using the Prism framework.Large-scale VLMs have demonstrated promising outcomes across a diverse range of multimodal tasks, as evidenced by extensive qualitative and quantitative evaluations. Early assessments of LVLMs often involved open-ended Visual Question Answering (VQA) [2, 24, 41, 22] and human-based subjective evaluations [64, 63] . However, these methods face limitations in accurately reflecting the true performance of VLMs. Open-ended VQA tasks typically demand an exact match between the model's prediction and the ground truth, which can lead to a significant number of false positives. Conversely, subjective evaluations introduce biases and make the results challenging to reproduce. Subsequent research has shifted towards structuring visual questions in closed-ended formats, such as multiple-choice or Yes-or-No questions. Pioneering works like MMBench [37] , MME [20] , or SEED-Bench [28] have presented comprehensive evaluations of VLMs using closed-ended VQA, covering various perception and reasoning capabilities. Additionally, specialized multimodal benchmarks have emerged to assess VLMs from specific angles. For instance, MMMU [66] evaluates VLMs' ability to handle multimodal examination questions, while POPE [30] and HallusionBench [33] scrutinize hallucination and illusion phenomena in VLMs. RealWorldQA [62] focuses on real-world perception with high-resolution images. Recently, MMStar [10] has addressed the issues of vision dispensability and data contamination in existing benchmarks by compiling high-quality, vision-indispensable questions from multiple sources, ensuring minimal data leakage and covering six core capabilities. In this study, Prism primarily utilizes MMStar for capability evaluation.To offer insightful and detailed feedback for the future optimization of VLMs, some researchers have initiated efforts to dissect the abilities of VLMs, aiming to uncover strategies for enhancement. To more effectively explore the disparity between VLMs and human cognition, Zhang et al. [69] employ Raven's Progressive Matrices (RPM) to examine the model's deductive reasoning skills grounded in visual perception. Through error case analysis, the authors observe that VLMs often make compounding and confounding errors when articulating individual elements within RPM, which subsequently results in erroneous reasoning. Although this study provides qualitative insights, it does not encompass a systematic evaluation of perception and reasoning capabilities. Meanwhile, researchers have developed specialized benchmarks to assess specific capabilities. For example, InfiMM-Eval [23] and MathVista [40] have implemented rigorous, step-by-step evaluations of the model's complex reasoning abilities on natural images and mathematical VQA problems, respectively. However, these reasoning benchmarks require a foundational perception capability to accurately identify key elements. Concurrently, there are perception benchmarks [58, 62, 30, 21] that exclusively focus on evaluating the perception skills of VLMs across various scenarios. Moreover, some specific crucial capabilities of VLMs have received researchers' attention, such as fine-grained understanding [68] and visio-linguistic compositional knowledge [57, 68, 25] . In this study, Prism introduces a general decoupling framework that facilitates a detailed analysis of perception and reasoning capabilities, applicable to any multimodal benchmark. and the question. The template for reformatting is shown in Fig. 8 . The reformatting process involves simple splicing, making it intuitive for the reasoning module to respond directly. An example of this can be seen in Fig. 9 .You are an excellent text-based reasoning expert. You are required to answer the question based on the detailed description of the image. Description: <description> Question: <question> You are an excellent text-based reasoning expert. You are required to answer the question based on the detailed description of the image. Description: The image presents a delightful celebration of Father's Day. Dominating the center of the image is a blue tie, adorned with white stripes, symbolizing the essence of fatherhood. The tie is slightly tilted to the right, adding a touch of dynamism to the composition. On the left side of the tie, the phrase "Happy Father's Day" is elegantly inscribed in a white cursive font, extending warm wishes to all dads. The text and the tie are set against a dark blue background, creating a striking contrast that draws attention to the main elements of the image. Adding a final touch of sophistication, a thin white border frames the entire image, encapsulating the joyous message of Father's Day. The image, in its entirety, serves as a heartfelt tribute to all the wonderful fathers out there. Some end-to-end models with small language models, often predict incorrect answers due to their limited reasoning capabilities. Utilizing an external LLM, Prism endows VLMs with the reasoning ability to solve the vision-language tasks, as presented in Tab. 11 and Tab. 12.PrismCaptioner can professionally extract and express detailed visual information to solve coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and math tasks, as presented in Tabs. 13 to 18.Although PrismCaptioner can generate detailed descriptions of images, there is still room for improvement compared to GPT-4o.(1) GPT-4o generates descriptions that are more relevant to the questions. Due to limited training data, PrismCaptioner's captions are less adaptive to query-specific parts compared to those of GPT-4o, as shown in Tab. 19.(2) GPT-4o's expression is more detailed and specific, and the responses about spatiotemporal information are also more accurate, as shown in Tab. 20.Limitations. In this study, we introduce the Prism framework and showcase its effectiveness as both an analytical tool and a versatile vision-language task solver. Prism relies on language descriptions while natural languages may struggle to express some obscure visual elements, especially some aesthetic contents like abstract artworks and surrealist paintings. It is difficult to obtain high-quality descriptions in these cases. Given budget constraints, our evaluation focuses on a select group of representative open-source and proprietary VLMs, which may not encompass all the most recent high-performing models. Our experiments with training visual captioners aim to illustrate Prism's capability to deliver strong performance on vision-language tasks while minimizing costs. To fully realize the potential of Prism, additional experiments are recommended, particularly on domainspecific visual instruction tuning data, such as tables and diagrams, screens, and graphical user interfaces (GUIs), medical images, etc., to thoroughly assess Prism's efficacy in these specialized contexts.Broader Impacts. As an analytical framework, Prism offers detailed insights into the perception and reasoning capabilities of vision-language models (VLMs), providing valuable guidance for future model optimization. Training large-scale VLMs necessitates extensive multi-modal data and The tree is adorned with a multitude of colorful lights and ornaments, creating a festive atmosphere.In the background, the cityscape is visible with tall buildings, some of which are adorned with American flags, suggesting a location in the United States. The flags are flying at half-mast, adding a solemn tone to the scene. The sky above is overcast, casting a soft light over the scene. The overall mood of the image is one of celebration and unity, as the crowd seems to be enjoying the holiday festivities. The image does not contain any text. The relative positions of the objects suggest a well-organized event, with the Christmas tree as the focal point and the crowd surrounding it. The flags in the background provide a sense of location and national pride.The overcast sky adds a touch of solemnity to the otherwise joyful scene. Perception Prediction: C. ( ) Table 11 : An Example of End-to-End v.s. Prism LLaVA-NeXT (Vicuna-7B) Prediction.computational resources. Prism mitigates this challenge by training small-scale VLMs that specialize in visual captioning tasks and perform reasoning with large language models (LLMs), which are now cost-effective. 2 When employed as a vision-language task solver, Prism can be trained and deployed at a significantly reduced cost, making it a promising approach for highly customized applications or tasks with limited training data. However, Prism also carries potential societal impacts, as it could lower the barrier to building multimodal applications, some of which may be harmful. There is a risk that Prism could be used to develop harmful multimodal AI systems. Additionally, data-driven methods often inherit biases, which can persist in downstream tasks. We urge users to thoughtfully consider the implications of these biases when implementing our model. GPT-4o PrismCaptioner Descriptions The image depicts a cozy living room corner with a modern and inviting aesthetic.Here is a detailed description of the scene: 1. Furniture and Objects: Sofa: A gray upholstered sofa is positioned on the left side of the image. It has a few decorative pillows in various patterns and colors, including white, green, and teal. Side Table : Next to the sofa, there is a wooden side table with two drawers. On top of the table, there is a blue ceramic lamp with a white lampshade and a potted plant with large green leaves. Rug: A gray rug with a white geometric pattern is placed on the floor. The rug is positioned in front of the sofa and extends partially under it, covering the wooden floor. Curtains: Light gray curtains with a subtle pattern hang from a black curtain rod above a window on the left side of the image. Wall Art: Above the side table, there is a framed piece of art featuring a colorful floral arrangement.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Those settings are described in In Secs. 3.1 and 5.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: During the inference process in Prism, we perform greedy decoding for both VLMs and LLMs, so the quantitative results we reported will be stable. Besides, reporting error bars will be expensive and is not a common practice for research in large vision-language models.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work will release the evaluation results and some weights of small-scale VLMs trained on publicly available dataset. We believe this paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets used in this work are properly credited. Our evaluation codes are modified based on VLMEvalKit, which goes with Apache 2.0 License. Our self-trained vision captioner is built with SigLip and InternLM2, and trained on ALLaVA and Evol-Instruct-GPT-4-Turbo, all of those assets go with Apache 2.0 License. Other evaluation datasets adopt in this paper go with CC-BY 4.0 License.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.For comparison, we use single image as input and limit the maximum numbers of output tokens to 512. Better performance of PrismCaptioners is presented in Appx. B.3Thanks to the development in deploying technologies, the inference of LLM is now cheap, provided by various corporations at a price as low as millions of tokens per US dollar. 
 
 Appendix:  


Decision: Systematic analyzing VLMs is still a challenging and open research topic, the authors presented an aspect of doing it through decoupling framework. All reviewers and AC hold positive view of this work, and suggest accept.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose prism, a framework to decouple the VLMs' capabilities in two stages: perception stages and reasoning stages. This framework allows the breakdown analysis of VLM capabilities and can also serve as a framework to integrate any given VLM and LLM. Based on their explorations and decoupled analysis, they discover that the integration of a lightweight VLM combined with a powerful LLM can be useful and exhibit outstanding performance and efficiency. The author provide good amount of experimentations to support their claims.
strength_weakness: strength_weakness: 1. There is not much unique and novel contributions in terms of algorithms and model designs.
questions: questions: See weakness section.

1. How about failure modes in different part of the VLMs (reasoning and perception)? Is there any analysis or ideas on analyze the hallucinations based on your framework?
limitations: limitations: limitations and broader impacts are properly addressed in the appendix.
paper_summary: paper_summary: The paper introduces Prism, a framework designed to decouple and independently assess the perception and reasoning capabilities of VLMs. Prism operates in two stages: a perception stage that extracts visual information and converts it into text using a VLM, and a reasoning stage that generates answers based on this textual information using a LLM. This modular approach allows for systematic evaluation of each component. Several insightful results are presented through extensive experiments.
strength_weakness: strength_weakness: * The prompt design for the perception and the reasoning stage is an important aspect. The impact of different prompt designs should be analyzed.
* Decoupling the inference into two stages might introduce additional computational overhead and latency, and cost-effectiveness needs to be discussed.
* Prism relies on language descriptions; however, in some scenarios, language struggles to describe the content of images because certain visual or logical concepts lack corresponding linguistic expressions, e.g. medical pathology images and graphs of mathematical functions.
questions: questions: * Another line of work focuses on assessing the compositional or fine-grained understanding abilities of VLMs, such as [1][2][3][4]. Could you discuss the relevance or differences of these works to Prism?
* Can you discuss the advantages and disadvantages of the 'end-to-end' approach versus the 'decoupling' approach?


My overall judgment of this article is positive. I am open to raising my score if the author can address my concerns listed above.

****
[1] Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality, CVPR 2022 

[2] Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding, CVPR 2024

[3] Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View, ICML 2024

[4] CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples, ACL 2024
limitations: limitations: Maybe the authors need to discuss potential limitations and future directions for improvement.
paper_summary: paper_summary: This paper introduces Prism, a framework to disentangle and evaluate the perception and reasoning capability of vision-language models (VLMs). They evaluate state-of-the-art VLMs, including proprietary ones and open-source ones, with varying model sizes. The evaluation results demonstrate that VLMs' perception ability is consistent regardless of the language encoder size while the reasoning ability is constrained by the model size. Moreover, they develop a vision-language model based on the discovered principles, which is lightweight yet effective.
strength_weakness: strength_weakness: 1. In the perception capability evaluation, besides accuracy, how detailed the image captions given by the VLMs are can also affect the final accuracy. As different VLMs are pre-trained on different datasets for different down-stream purposes, some of them might be trained to give concise response, leading to low final accuracy in Prism. However, giving concise response does not necessarily mean the models cannot see the other details. Therefore, Prism might be inaccurate to evaluate their perception capability.
2. Different VLMs are pre-trained on different datasets, which could cause their differences in perception and reasoning capability. It would provide more insights if the paper investigates the relationship between the models' pre-training datasets and their perception and reasoning capabilities.
3. The results of query-specific instruction in perception ability evaluation is not only affected by the models' capability in perception, but also their capability in instruction-following, e.g., the second example in Figure 3. It would be more accurate to reflect the perception capability by first checking whether the model response follows the instruction, e.g., by using an external LLM.
questions: questions: 1. To evaluate a VLM's perception capability, Prism uses the VLM as an image captioner and feeds the generated caption to an external LLM to answer the question. The final accuracy can indicate how accurately the VLM can describe the image. However, evaluating the VLM on image captioning benchmarks can also reflect the perception ability and is more straightforward than the proposed framework. What are the advantages of Prism over image captioning benchmark evaluation?
limitations: limitations: Limitations and broader impacts are discussed in Appendix D.

Comments: comment: We express our gratitude to the reviewer for the constructive feedback. We are encouraged to learn that our response could address some of the reviewer's concerns.



In terms of Q3: **Lack of Linguistic Expressions for Some Images**, we believe most images can be described in detail by linguistic expressions, even if they are medical pathology images or math contents. However, we acknowledge that natural languages may struggle to express some obscure visual elements, especially some aesthetic contents, e.g., abstract artworks and surrealist paintings. In these cases, it is difficult to obtain high-quality descriptions. Thus, we will further focus on potential representation in a more abstract way instead of linguistic expressions.



Once again, we thank the reviewer for the time and effort. If our response could address the reviewer's concerns, we hope the reviewer could raise the score. (now it is 6.) We will ensure that the additional discussions in the rebuttal are properly incorporated into our revised paper.
We thank the reviewer for the encouraging comments. We are pleased to note that our response could address some of the reviewer's concerns.

Regarding the proportion of the different types of failures, we sample 100 cases from all VLMs and conduct quantitative analysis based on the categorization in our response to the reviewer KmBE. There are 72 perception failures and 34 reasoning failures in all 100 cases. In some cases, both perception and reasoning errors exist. The detailed proportion result of errors (in perception/reasoning) is as follows:

- **Perception:**  **Factual Errors (43.1%)**, **Incomplete Details (40.2 %)**, **Instruction Following (16.7%)**
- **Reasoning:**  **Logical Errors (67.6%)**, **Lack of Knowledge (26.5%)**, **Misunderstanding of the Query (5.9%)**



For the cases where perception and reasoning errors both exist, here is an example from MMStar:

Index: 1222

Image: A bar chart titled "Accuracy of different algorithms". The vertical axis is labeled "Accuracy" and ranges from 0 to 10. The horizontal axis has two categories: "ivory" and "calf." The height of "ivory" is 9, and that of "calf" is 4.

Question:  What is the sum of the accuracies of the algorithms calf and ivory?

LLaVA-NeXT (Vicuna-13B) succeeds in obtaining the heights of two categories but gives the wrong horizontal interval, 0-9, which is a perception error. ChatGPT gives the wrong sum, 11, even if the heights 4 and 9 are correctly expressed by the VLM, which is a reasoning error.



Once again, we thank the reviewer for the time and insights. We hope this comment could address the reviewer's concerns and will make sure to properly incorporate the additional discussions in the rebuttal into our revised paper.
I appreciate the response from the authors. Most of my concerns have been resolved. However, regarding Q3, I still believe that some information is difficult to accurately describe at the linguistic level and instead needs to be represented in a more abstract space. I hope the author can provide a more in-depth discussion on this. 
Overall, I am inclined to accept this paper, and I will raise my score to 7.
I read the other reviewers' review and the authors' responses to them. I revisited the model prompts and understood that they are instructed to give very detailed image descriptions. And I agree with the authors that failures in instruction following is also a type of perception failure. I would advise the authors to discuss the proportion of the different types of failures mentioned in their response to reviewer KmBE. Moreover, I highly appreciate the authors' discussion about Prism's advantages over image captioning evaluation. Therefore, I raise my rating to 6 (Weak Accept).
Dear Reviewer KmBE,

Thank you for the time and patience you have dedicated to reviewing our submission. We hope we have addressed the concerns raised in your initial reviews and eagerly await your thoughts and further guidance to refine our work.

As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you require additional information or clarification. We are eager and ready to engage in further discussions to enhance and elevate our work.

Best regards and thanks,

Paper 8561 Authors
Dear Reviewer onkG,

We greatly appreciate the time and effort you've taken to review our submission. We hope that our response has addressed the concerns raised in your initial reviews, and we look forward to your feedback. 

As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you need any further information or clarification. We are fully open to engaging in further discussions to improve our work.

Best regards and thanks,

Paper 8561 Authors
Dear Reviewer rB4V,

Thank you for the time and effort you have dedicated to reviewing our submission. We hope our response has effectively addressed the concerns raised in your initial reviews, and we eagerly await your thoughts and further guidance to refine our work.

As the author-reviewer discussion period for NeurIPS 2024 is half past, please let us know if you require any additional information or clarification. We are more than willing to engage in further discussions to enhance our work.

Best regards and thanks,

Paper 8561 Authors