Title: PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models?
Venue: NeurIPS 2025
Accepted: no
Keywords: pixel-level understanding; foundation models
Abstract: Multiple works have emerged to push the boundaries on multi-modal large language models (MLLMs) towards pixel-level understanding. The current trend in pixel-level MLLMs is to train with pixel-level grounding supervision on large-scale labelled data with specialized decoders for the segmentation task. However, we show that such MLLMs when evaluated on recent challenging vision-centric benchmarks, exhibit a weak ability in visual question answering (VQA). Surprisingly, some of these methods even downgrade the grounding ability of MLLMs that were never trained with such pixel-level supervision. In this work, we propose two novel challenging benchmarks with paired evaluation for both VQA and grounding. We show that MLLMs without pixel-level grounding supervision can outperform the state of the art in such tasks.  Our paired benchmarks and evaluation enable additional analysis on the reasons for failure with respect to VQA and/or grounding. Furthermore, we propose simple baselines to extract the grounding information that can be plugged into any MLLM, which we call PixFoundation. More importantly, we study the research question of ``When does grounding emerge in MLLMs that are not trained with pixel-level grounding supervision?'' We show that grounding can coincide with object parts, its location, appearance, context or state, where we show 27-45% of the examples in both benchmarks exhibit this phenomenon. Our code and datasets will be made publicly available and some are part of the supplemental.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the current capabilities of MLLMs on VQA tasks, with a particular focus on their ability to provide visually grounded answers. To facilitate this analysis, the authors construct two new benchmarks by extending existing datasets, MMVP and CV-Bench, originally introduced by Tong et al. (Eyes Wide Shut, CVPR 2024; Cambrian-1, NeurIPS 2024). The new benchmarks include additional segmentation masks that indicate the pixel regions necessary to answer each question, enabling evaluation of grounding behavior.\n\n\nBased on the benchmarking results, the paper challenges the existing pixel-level MLLMs and makes several key claims:\n- Pixel-level supervision in MLLMs degrade VQA performance.\n- MLLMs trained without pixel-level grounding supervision can outperform state-of-the-art methods on VQA and grounding.\n- Vanilla MLLMs - without segmentation decoders or pixel-level grounding supervision - can achieve superior performance when evaluated using a harmonic mean of VQA accuracy and grounding quality.\n- The authors introduce a lightweight method for extracting segmentation masks from vanilla MLLMs. Unlike recent work such as Cao et al. (Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision, ICLR 2025), their approach relies on identifying token attention corresponding to descriptive cues (e.g., appearance or location) rather than object nouns.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This work extends two existing benchmarks by providing additional segmentation masks to indicate the pixel regions necessary to answer questions. While GPT was used to accelerate the annotation process, human verification was still involved to ensure data quality. I saw a large supplementary zip file containing the annotated data. Although I haven’t walked through all examples in detail, the additional annotations appear to be a valuable resource for the community to evaluate and better understand MLLMs’ grounding behavior.\n\n**Weaknesses**\n\nOverall, I do not think this work is ready for readers, primarily due to the rooms to increase research depth of methodology, provide rigorous evaluation, and especially provide actionable insight (if the claim is that existing pixel-level foundation models are not heading the right direction, what are your suggestions, how should we do the MLLM design and training with pixel-level supervision? The paper is too ambiguous on actionable items).\n\n- The paper raises the question, “Are we heading in the right direction?”, but does not offer a concrete alternative or provide actionable next steps on what direction should be taken. If the claim is that current methods are not heading correct direction, the paper should articulate the gaps more clearly and propose steps to address them. The proposed solution is to extract attention maps from vanilla MLLMs and using SAM (which itself is large-scale trained with pixel-level supervision) and GPT-4o for post-hoc segmentation and mask selection. The proposed solution does not present a principled technical alternative. As such, it falls short of addressing the directional question posed by the authors.\n\n- The benchmarking methodology does not look rigorous. Models are broadly grouped into two categories: (1) MLLMs with pixel-level supervision, and (2) vanilla MLLMs without it. However, models vary significantly in terms of training data, scale, intended use cases, and favorite prompts. For example, LISA is designed for segmenting specific objects based on user input (e.g., “please segment…”), while GLaMMa focuses on captioning with interleaved segmentation (e.g., “Describe the image in detail…”). These models were not instruction fine-tuned to handle multiple-choice (a)(b)(c) or yes/no VQA formats. Such discrepancies can heavily influence accuracy, making it problematic to draw strong conclusions about the value of pixel-level supervision. Hence, the claims in this paper make me feel unjustified given the number of confounding factors.\n\n- Building on the above, the evaluation protocol and metric in Eq. (1) lack rigor. GPT-4o is used as part of the evaluation process, yet no effort is made to isolate or control for its influence. Factors like model versioning, conversation history, and response randomness can all affect the outcome. A standalone, well-defined evaluation metric is essential for reproducibility and for enabling future work to build on these results.\n\n- The claim that “MLLMs not trained with pixel-level grounding or segmentation decoders can outperform others” (from Line 42) is not clearly supported. From my reading, the strong performance reported relies on assembling multiple foundation models. The final grounding masks are generated by SAM (which is trained with pixel-level supervision), and GPT-4o is then used to select the most relevant one. In essence, this suggests that vanilla MLLMs struggle with precise grounding. They may attend to multiple plausible regions but cannot reliably identify the correct one and output it. Therefore, another foundation model GPT-4o comes in for the mask selection.\n\n- Overall, the paper looks to me is incomplete. While it introduces useful benchmarks and makes some interesting observations, the analysis remains at a surface level, and no solid solution is proposed to address the gaps identified for pixel-level MLLMs model design and training strategies. This feels more like a preliminary investigation than a fully developed research contribution.\n\nQUESTIONS:\nSeveral clarification questions to ensure my understandings.\n\n- Clarify “some of these methods” (Line 7). In multiple instances, the paper uses vague phrases such as “some of these” without specifying which models or approaches are being referenced. This ambiguity makes me have to guess what you are referring to. Please explicitly list or define the methods to help the readers follow your arguments and comparisons more clearly.\n\n- Elaborate on “reasons for failures” (Line 12). Could you provide concrete examples or categories of failure modes? Also, are there any insights or hypotheses on how future models might overcome these failures?\n\n- Line 18: \"We show that grounding can coincide with object parts, its location, appearance, context or state, where we show 27-45% of the examples in both benchmarks exhibit this phenomenon.\" What characterizes the remaining >50% of examples? Do they mostly correspond to object nouns, or are there other patterns?\n\n- Line 66 - Line 68: \"Our mechanism uses the observation that grounding can emerge corresponding to different output tokens describing the object’s appearance or location, not necessarily the exact text of the object of interest, which we call PixFoundation.\" Is this grounding influenced by SAM’s inherent preferences for certain types of point prompts? Could it be that the descriptive tokens happen to produce better SAM prompts by coincidence rather than model grounding ability?\n\n- Interpretability of token-location connection (Lines 257–263): Figure 6(a) analyzes the distribution of grounding across the output token positions in free-form MLLM responses. To me, it looks like a relatively uniform distribution, what actionable insight is being drawn from this analysis? Do you intend to use this distribution to inform the mask selection strategy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper benchmarks pixel-level MLLMs' VQA and grounding ability. Instead of evaluating on standard grounding benchmarks or VQA benchmarks, this paper proposed PixMMVP and PixCV-Bench for comprehensive VQA and grounding evaluation. Evaluating pixel-level supervised and vanilla MLLMs on these benchmarks, they find that pixel-supervised MLLMs may not perform as well as vanilla MLLMs in grounding. They further propose a plugin module, PixFoundation, which extracts segmentation maps from MLLMs. Experiments and analyses are conducted.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper points out an interesting task that combines grounding and VQA.\n- They curate VQA and Grounding paired benchmarks, PixMMVP and PixCV-Bench, which provide referring segmentation given a question.\n- They use the proposed benchmarks to evaluate pixel-level and vanilla MLLMs, revealing some unintuitive findings, such as the pixel-level supervised MLLMs may not outperform the vanilla variants in grounding tasks.\n- They propose PixFoundation, a way to extract segmentation from MLLMs.\n\nWeaknesses\n- The writing could be improved. Some parts are misleading and confusing, and some necessary definitions are not well-explained. For example, the title \"Are we heading in the Right Direction with **Pixel-level Vision Foundation Models**\" sounds like they work on Vision Foundation models like ResNet or ViT, but they only study MLLMs. Moreover, the pixel-level grounding supervision is not a common term in the grounding community. The task looks relevant to referring expression segmentation. Preliminaries and definitions are essential to help the readers understand the background.\n- The paper aims to investigate whether \"the pixel-level MLLMs trained with full grounding supervision are heading in the right direction to improve both grounding and VQA\" (ln 38-39). However, this motivation may suffer from logical flaws.\n\na) The importance of pixel-level MLLMs is questionable. The state-of-the-art or dominant models do not seem to be trained with pixel-level supervision. Those general MLLMs have superior performance on various tasks, including VQA and grounding. \n\nb) The second part is the correlation between the \"right direction\" and performing \"grounding and VQA\" at the same time. Not to mention, the grounding ability studied in this paper refers to expression segmentation only. Combining grounding and VQA into a new task can be interesting, but it does not indicate that evaluating grounding and VQA ability through standard benchmarks separately is convincing.  \n\n- A task-specific module helps well-established, powerful MLLMs outperform pixel-level MLLMs, but this is not completely fair. The paper proposes a new task that has not been seen during the training of MLLMs. Therefore, fine-tuning or a task-specific module is intuitively helpful in this case. Vanilla MLLMs are well-trained and are well-known for superior performance on various tasks. It is not surprising that they outperform pixel-level MLLMs after integrating a task-specific module. The unfair setting makes the findings doubtful.\n\nQUESTIONS:\n- What is the pixel-level grounding supervision? How are the MLLMs trained?\n- Why is the pixel-level grounding so important? Why does the paper also study referring expression comprehension? Why is answering questions with segmentation the right indicator of grounding and VQA abilities, compared to separate standard benchmarks?\n- Can a method similar to PixFoundation help the pixel-level MLLMs in the proposed new task? Or is there a way to fine-tune or off-the-shelf test both pixel-level and vanilla MLLMs?\n- Could you provide the VQA and grounding (both REC and RES) performance of the selected models on standard benchmarks?\n\nLIMITATIONS:\nThe paper tries to propose an interesting new task. However, they overstate the scope and fail to explain concepts clearly. The motivation possibly has fundamental flaws, and the comparisons seem unfair.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper addresses an important and timely question about the effectiveness and direction of pixel-level grounding in multi-modal large language models (MLLMs). The authors present two novel benchmarks, PixMMVP and PixCV-Bench, specifically designed to evaluate the paired visual question answering (VQA) and pixel-level grounding capabilities of MLLMs. Additionally, they introduce a baseline method, PixFoundation, aiming to analyze the emergent grounding capabilities of MLLMs that have not been explicitly trained with pixel-level supervision.

STRENGTHS AND WEAKNESSES:
Strenghts:
Two new paired benchmarks (PixMMVP and PixCV-Bench) are introduced, providing valuable resources for future research.

Weaknesses:
It is unclear what the authors mean by MLLMs not heading in the right direction. Is it because pixel-level MLLMs have shown degradation in their capabilities and chat performance, or because MLLMs without pixel-level grounding supervision can outperform the state of the art in grounding tasks? If the authors mean the former, degradation in VQA performance might be acceptable due to multi-task joint training conflicts or limited model capacity rather than an indication of moving in the wrong direction. If the authors mean the latter, the new proposed method should be rigorously compared against existing widely-used grounding datasets to validate its superiority.

QUESTIONS:
Please address the weaknesses above.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores whether the current pixel-level MLLM is really moving in the right direction. To this end, the authors proposed two new paired benchmarks (PixMMVP and PixCV-Bench) for jointly evaluating VQA and pixel-level grounding. The authors argue that pixel-level MLLM often hurts the model's ability to answer visual questions. They also proposed a simple baseline method (PixFoundation) for extracting good localization segmentation from regular MLLMs that are not trained on this task, and tried to show that MLLMs without pixel-level localization supervision have the potential to outperform pixel-level MLLMs in these tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper is clearly written, and the figures are very helpful in understanding the main points.\n2. This paper proposes two new benchmarks (PixMMVP and PixCV-Bench) that can more comprehensively evaluate VQA and segmentation simultaneously, rather than separately, which is very reasonable and helpful.\n3. The experiments compare multiple pixel-level MLLMs with vanilla MLLMs, including quantitative and qualitative analysis, ablation and comparison.\n4. The proposed baseline method (PixFoundation) is simple but effective, demonstrating that grounding can be mined from the attention map of a vanilla MLLM without additional supervision.\n\nWeaknesses:\n\nThe main weaknesses of this paper lies in its presentation, which may mislead readers who are not familiar with this particular field. The paper only makes a superficial comparison, highlighting the performance differences without digging into the reasons behind them (or at least without providing potential explanations).\nThis has the potential to create a false opposition between \"pixel-level MLLM\" and \"vanilla MLLM\", which may lead readers to prematurely conclude that the former is a flawed or less valuable research direction.\nThe goal of this paper should be to stimulate discussion on how to improve all types of models, rather than positioning the two methods as adversarial. Specifically:\n\n1. The PixFoundation baseline relies on a strong multi-stage workflow: it uses SAM to generate a large number of high-quality mask candidates and then uses GPT-4o for the final mask selection. Its strong performance is likely due in large part to SAM's excellent multi-granularity mask generation capabilities and GPT-4o's strong visual perception capabilities. This makes the comparison with smaller, independent open source models in Table 1 seem unfair.\nThis is not a comparison with similar methods because it compares a single end-to-end model to a complex pipeline that decomposes the task into three subtasks: coarse attention localization (via native MLLM), fine mask candidate generation (via SAM), and optimal mask selection (via GPT-4o).\n\n2. In contrast, the results of this paper can also be interpreted more favorably for pixel-level MLLMs. The baseline performance (the M metric) of models such as OMG-LLaVA, GLAMM, and LISA is comparable to or even exceeds PixFoundation Oracle (7B), which fully demonstrates their potential. This shows that a single unified model is able to achieve comparable performance to complex multi-model workflows including SAM and GPT-4o. This suggests that these models are not failures.\n\n3. The performance degradation of some pixel-level MLLMs on VQA and referred visual grounding instruction following may not be an inherent flaw of the pixel-level methods themselves. More likely, they employed a specialized training regimen in which the models were fine-tuned to perform well only on specific task formats and textual cues. This is most likely a problem of insufficient distribution and diversity of the training data rather than a fundamental flaw in the methodology. The current narrative in this paper may be misleading because it does not adequately consider this critical context.\n\nQUESTIONS:\n1. PixFoundation uses SAM for mask candidate generation and GPT-4o for the critical mask selection. Can you elaborate on the importance of the mask candidate model and mask selection model, and explain whether it is fair to compare this workflow with unified open-source models?\n\n2. Can you provide a more detailed analysis of the computational overhead and inference time of the PixFoundation pipeline compared to the pixel-level MLLMs ? This will help readers understand the trade-offs of the methods in practical applications.\n\n3. For the \"PixFoundation\" baseline, have you tried using it for pixel-level MLLMs?\n\n4. For future work, do you think it is possible to combine the advantages of supervised pixel-level MLLMs and regular MLLMs, or are they fundamentally in conflict?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}