How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks
Authors: Rahul Ramachandran
Ali Garjani
Roman Bachmann
Andrei Atanov
Oğuzhan Fatih Kar
Amir Zamir
Venue: ICLR 2026 Poster (2026)
Keywords: vision benchmark
multimodal foundation models
vision language models
standard computer vision tasks
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Multimodal foundation models, such as GPT-4o, have recently made remarkable progress, but it is not clear where exactly these models stand in terms of understanding vision. In this paper, we benchmark the performance of popular multimodal foundation models (GPT-4o, o4-mini, Gemini 1.5 Pro and Gemini 2.0 Flash, Claude 3.5 Sonnet, Qwen2-VL, Llama 3.2) on standard computer vision tasks (semantic segmentation, object detection, image classification, depth and surface normal prediction) and using established datasets (e.g., COCO, ImageNet and its variants, etc).

The main challenges to performing this are: 1) most models are trained to output text and cannot natively express versatile domains, such as segments or 3D geometry, and 2) many leading models are proprietary and accessible only at an API level, i.e., there is no weight access to adapt them. We address these challenges by translating standard vision tasks into equivalent text-promptable and API-compatible tasks via prompt chaining to create a standardized benchmarking framework.

We observe that 1) the models are not close to the state-of-the-art specialist models at any tasks, and 2) they perform semantic tasks notably better than geometric ones. However, 3) they are respectable generalists; this is remarkable as they are presumably trained on primarily image-text-based tasks. 4) While the prompt-chaining techniques affect performance, better models exhibit less sensitivity to prompt variations. 5) GPT-4o performs the best among non-reasoning models, securing the top position in 4 out of 6 tasks and 6) reasoning models, e.g. o3, show improvements in geometric tasks.

Published as a conference paper at ICLR 2026
HoW WELL DOES GPT-40 UNDERSTAND VISION? EVALUATING MULTIMODAL FOUNDATION MODELS ON STANDARD COMPUTER VISION TASKS
Rahul Ramachandran'
Ali Garjani
Roman Bachmann'
Andrei Atanov*
Oğuzhan Fatih Kar*
Amir Zamir'
Swiss Federal Institute of Technology (EPFL)
ftttss:/ffmvvision-eeeeee wwwpaalss.p......mmmmm ch/
MFM Prodictions
MFM Performance
Ganee 20 Pauht
Chank Sweee
Canalicatto
Stakeol-dbe-it
Shat--f--teed Quan
Classification
Object Dataction
Sagmentation
Sedee
Okued (POSS1)
Input Image
(IT)
Depth
Surface Normals
Pign
Segradaccs
Figure 1: We benchmark multimodal foundation models (MFMs) on established computer vision tasks and datasets. Left: GPT-40's predictions for each task. Right: The performance of the MFMs on several classical computer vision tasks. We compare MFMs with specialist models both directly and by calibrating for the chosen structure and constraints of the used prompt chain (+chain; see Sec. 4). The axes normalized using task-specific lower and upper bounds, defined by blind guessing and state-of-the-ee specialist performance, respectively.
ABSTRACT
Multimodal foundation models (MFMs), such as GPT-4o, have recently made remarkable progress. However, their detailed visual understanding beyond question answering remains unclear. In this paper, we benchmark popular MFMs (GPT- 4o, o4-mini, Gemini 1.5 Pro and Gemini 2.0 Flash, Claude 3.5 Sonnet, Qwen2- VI., Ilama 3.2) on standard computer vision tasks (semantic segmentation, object detection, image classification, depth and surface normal prediction) using established datasets (e.g., COCO, ImageNet and its variants, etc).
The main challenges in performing this analysis are: 1) most models are trained to output text and cannot natively express versatile domains, such as segments 3D geometry, and many leading models are proprietary and accessible only an API level, i.c., there is no weight access to adapt them. We address these by translating vision tasks into text-promptable. API-compatible formats via prompt chaining, creating standardized benchmarking framework.
We observe that: 1) The MFMs are not close to the state-of-the-art specialist models at any tasks. 2) They are respectable generalists; this is remarkable as they are presumably trained on primarily image-text-base tasks. 3) They perform semantic tasks notably better than geometric ones. 4) GPT-4o performs the best among non-reasoning models, securing the top position in 4 out of 6 tasks. 5) Reasoning
Equual advisorship.
Work done while at EPFL, now at Apple. Work done while EPFL., now UTUC.

---

Published as a conference paper at ICLR 2026
models, e.g. o3, show improvements in geometric tasks. 6) While prompt chaining techniques affect performance, better models are less sensitive to prompt variations. 7) An analysis of models with native image generation, such as the latest GPT-4o, shows they exhibit failure modes, such as hallucinated objects or misalignment between imput and output.
INTRODUCTION
Multimodal foundation models (MFMs), such as GPT-4o, Gemini 1.5 Pro and 2.0 Flash, and Claude 3.5 Sonnet (Anthropic [024] |
OpenAl 2024b| Reid al. 2024). have advanced rapidly, demonstrating impressive capabilities in their public releases (OpenAl 2024b). However, while the community has extensively investigated their language proficiency (Chen al. 2021] | et al. 2024 Hendrycks ict al. 2020; Rein et al. (023), the extent of their vision capabilities remains underexplored. We still lack well-calibrated quantitative understanding of their performance on established vision tasks and datasets, particularly across diverse axes of vision, e.g. semantics, 3D, grouping, etc.
Most existing vision benchmarks for MFMs primarily target text (e.g., VQA) or tasks closely tied
to text, such as classification (Fu et al. 2024 Rahmanzadehgervi et al. 2024, TTng et al. 2024a by
Wu and Xie 2024, Yue et al.) 2024). While they provide useful insights, several key limitations
persist. First, it is unclear how much solving these benchmarks truly depends on the visual input, and some were shown to mainly measure the language capabilities of MFMs while overlooking the vision component (Tong et al. 20244). Second, they all require the model to output text, making it hard to compare the vision capabilities of MFMs on vision-only tasks against vision specialist models. Third, they do not shed light on other aspects of visual understanding, such as 3D geometry, grouping, or segmentation, that are less text-oriented.
We address these limitations by evaluating MFMs on well-established vision tasks and datasets developed by the community. Specifically, we test GPT-4o, 04-mini, Claude 3.5 Sonnet, Gemini 2.0 Flash, Gemini 1.5 Pro, Qwen2-VI, and Llama 3.2 on classification, object detection, semantic segmentation, grouping. depth prediction, and surface normal prediction using COCO (Lin et al.) 2014). Hypersim (Roberts et al. (021). as well as ImageNet (Russakovsky et al. 2014) and its variants (Hendrycks and Dietterich 2019 Hendrycks et al. 2021 Kar et al.) 2022b) Recht et al. 2019 Wang et al. 2019). Most of these tasks, however, require dense pixel-wise predictions not readily compatible with the default text output of most MFMs. Furthermore, direct prompting usually leads to a varying and often weak performance across tasks, hence it may not represent the actual visual understanding capabilities of MFMS (see Sec. 4.2 and App.
To address these challenges, we split each task into multiple sub-tasks, each of which can be solved in textual form via prompting (see Sec. (3). This results in prompt chaining framework that can be applied to any MFM with a text interface (e.g., chatbot APIs) to solve standard vision tasks. Specifically, our proposed approach allows MFMs to 1) detect bounding boxes, 2) generate complete segmentation masks for complex scenes, 3) extract semantic entities from images similar to SAM (Kirillov al. 2023b), 4) estimate dense depth and surface normal maps (see Fig. I
We emphasize that this prompt chaining framework is not proposed as an alternative methodology for solving vision tasks, nor do we suggest that MFMs should adopt such approuches in practice. Rather, our framework serves specifically as a standardized method to measure and benchmark any MFM that can input images and output text. Crucially, this enables quantifiable and holistic understanding of MFMs' vision capabilities on various established vision lasks and benchmarks, as well as a direct comparison with vision-only models.
We find that the current generation of MFMs achieves a good performance in most cases and are respectable as generalists, with GPT-4o scoring the best in 4 out of 6 tasks. However, they still lag behind task-specific state-of-thee -or---r-----r vision models in all tasks. In particular, we find that the MFMs perform geometric tasks significantly worse than semantic ones. Furthermore, prompt sensitivity analysis for each task and find that, while performance varies for different prompts, better models exhibit less sensitivity. To enable further research in this direction and enable the community to benchmark future MFMs on vision tasks, we open-source our evaluation code and prompt chains, along with interactive visualizations available our project website.
perform detailed
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Advances in MFMs. There has been remarkable progress in MFMs (Achiam et al. 2023 Alayrac
ce al.
2022
Anthropic 2024
Bai et al.
2023 Beyer et al. 2024; Dai et 2033] Li el al. 2023a Liu
al.
2024
OpenAl 2024a b 2025b;
Reid et al.
2024
Team 2024 Team et al. 2023] Wang et al.
2022
2024)
(see Yin et al. 2023] Zhang et al.
2024) for surveys), leading to strong performance
on multimodal tasks like VQA and instruction following. Despite the progress, is unclear how
well these models perform tasks that require dense visual understanding, which is our main focus.
Benchmarking vision capabilities of MFMs. Many works investigate the vision capabilities of MFMs via -QA-style benchmarks that combine visual and textual inputs to generate textual outputs (Al-Tahan et al.) 2024, Fu al., 2222]] Jiang et al., 2024a Li et al. 2023bb Liu al. 2024 Rahmanzadehgervi et al., [024]] Tong et al.] 22024 bj Yue et al. 2024). While these approaches
offer valuable insights, they are incompatible with vision-only models, making direct comparisons
difficult. In contrast, we evaluate MFMs on standard vision lasks, enabling direct comparison with strong vision specialists to track MFMs' progress. Cambrian-1 (Tong et al. 2024a) evaluates MFMs on vision datasets (Brazil et al. 2023 Lin et al.] 2014] (Zho et al.) 2017) by repurposing dataset annotations into text format. We differ by translating MFM outputs into the annotation format instead, e.g., segmentation maps. To the best of our knowledge, this is the first approach that enables an apples-to-apples comparison with vision specialist models using standard task-specific metries and qualitative analyses in the tasks' native oulput space.
Prompting MFMs. Various prompting techniques have been developed for MFMs (Khot al. 2022 Wei et al. 2022; Yao et al. 2024; Zhou et al. 2022). We follow a similar strategy and decompose complex vision tasks into simpler sub-tasks that MFMs can handle. Several works
developed prompting techniques to unlock the vision capabilities of MFMs (Hu et al. | [2024] Wu and
Xie, 2024 Wu et al., 2024 Yang al. 2023a). The related DetToolChain (Wu et al.| (024) for object detection is not fully reproducible at the time of writing. We differ by 1) focusing on a wider
range of tasks including semantic and geometric ones 2) for several MFMs including closed- and
open-weight ones 3) with a simpler yet effective and uniform prompt chaining mechanism.
3 PROMPT CHAINING FOR SOLVING VISION TASKS
In this section, describe the developed prompt chaining techniques that enable MFMs solve standard computer vision tasks. The core idea is to break each task into simpler, vexlle sub-tasks, e.g., identifying whether an object is present in a patch of an image. We then solve each sub-task by prompting the MFM. To guide the choice of how to split each task into sub-tasks, we rely on our early key observation that most MFMs are relatively strong at image classification (see, e.g., Tab. I and, therefore, split each task into multiple classification sub-tasks. We provide the pseudo-code for each technique in App. D
Object detection. The goal is to predict bounding box coordinates that tightly localize the objects in the image. Similar to Yang et al. (2023b), our initial attempts showed that many MFMs fail at predicting the coordinates directly. Thus, we develop a prompt chaining method and divide the original task into two stages. The first stage is to identify all objects in the image, similar to classification. In the second stage, for each object, we regress its coordinates via recursive zooming (see Fig. (2). Specifically, we divide the image into grid cells and ask the model to identify whether (a part of) the object is present in each cell. We then discard cells without objects, reducing the search space. We apply this process recursively, progressively eliminating irrelevant regions of the image until only the object of interest remains present in the image. We use two grid resolutions: a coarse grid for quick downsampling and finer grid for precise edge refinement to reduce the number of steps.
Semantic segmentation. The goal is to assign a single semantic class to each pixel in an image. Instead of per-pixel querying that is prohibitively expensive, we split the image into pixel groups using an unsupervised superpixel clustering algorithm (Achanta et al. 2012) and assign single label per group to decrease the number of API calls (or forward passes). This approach is used primarily for cost efficiency. as the superpixel algorithm is comantically-agnnssse
it only provides candidate regions based on low-level features like color and texture (Stutz al. 2018), while the MFM remains solely responsible for the semantic labeling. We confirm in App. [.8 E.8 that our findings hold even when increasing the superpixel granularity. In all our comparisons, we account for potential biases
3

---

Published as a conference paper at ICLR 2026
Dupth prediction
Bemantic segmentation
with dar
Object detection
Grouping
Figure 2: Prompt chain algorithms overview. a) Depth prediction randomly samples pairs of superpixels and performs pairwise depth comparisons, which are globalized by minimizing an objective function, resulting in relative depth map. Surface normal estimation is performed in a similar manner (see App. D.4. b) Semantic segmentation constructs multi-scale "pyramids" of superpixel crops (superpixel, local context, full image) and classifies these sequentially with the MFM. c) Object detection iteratively queries 3 x 3 grid of image crops for the target class (e.g.. Sheep), discarding empty cells and refining the search until the object is localized. d) Grouping grows cluster around query point by merging adjacent superpixels predicted to belong to the same object. "Summary of the actual prompt; see full prompt chain descriptions in the supplementary material.
introduced by superpixelation (and other approximations in prompting) by introducing calibration control baselines (see Sec. 4
After dividing the image into superpixels, we classify them in batches to decrease the overall cost, as in the classification task. Similar to the object detection algorithm, this approach utilizes the strength of MFMs as strong image classifiers. To maintain consistency, we include prior predictions in the prompt history, which improves performance.
In our early experiments, we found that simply outlining individual superpixels on an input image leads to poor performance. This aligns with findings that MFMs have "blurry vision" and struggle with fine-grained localization (Fu et al., 20244 Wu and Xiel 2024). To address this, we provide the MFM with the crops of each superpixel
multiple scales, which we found to improve the performance significantly. Please see Fig. 2 for an overview.
Grouping. Given an image and a query (or anchor) point on it, the goal is identify other pixels that belong to the same object or background. Unlike semantic segmentation, this task has no predefined classes, making it more challenging. To tackle this task, before, we superpixels and the MFMs' capability to determine visual similarity (Fu et al.] 2024). We create a graph of neighboring superpixels, identify the one with the query point, and explore its neighbors. The model decides whether each adjacent superpixel belongs to the same object as the initial superpixel. The selected superpixels then merged with the initial one form the next input cluster. This process continues until no more superpixels are added. Please see Fig. 2 for an overview.
Depth prediction. We perform relative depth prediction by querying the model rank different parts of the image according to their distance from the camera. As per-pixel querying is infeasible, adopt region-wise comparison strategy inspired from Zoran et al. (2015). To identify suitable regions for comparison, we first segment the image into superpixels. We then randomly sample pairs of superpixels and query the MFM to rank these pairs based relative depth (see Fig. [3 in the appendix). These pairwise rankings are then globalized by minimizing the objective function from (Zoran et al. 2015). which assigns larger values to superpixels ranked deeper in pairwise comparisons (see App. D.3). We then use the values found by the optimization method to rank all superpixels. For simplicity, we assume that all pixels within a superpixel share the same depth rank, allowing us to extend the superpixel-level depth predictions to pixel-wise ranking across the entire image (control baselines are included in evaluations).
4

---

Published as a conference paper at ICLR 2026
Figure 3: Qualitative results. Visual comparisons showing the performance of MFMs across each task. We find that all models perform relatively better on semantic tasks compared to the geometric ones. For surface normal visualizations, we combine the per-axis normalized predictions and project them onto the unit sphere. Please see the appendix for additional qualitative results.
Surface normal prediction. We follow ranking approach similar to that used for depth. We use standard basis vectors relative to the camera (right, up, and forward) as reference directions, and for each randomly sampled pair of superpixels, with each basis vector (see Fig. 14 in the appendix). After we obtain the pairwise comparisons for each direction, we globalize them using the same algorithm used for depth (Zoran et al., 2013). This results in three distinct surface normal maps, one for each basis direction. Similar to depth, we assume uniformity within superpixels and assign the same rank all pixels within each superpixel group (control baselines are included in the evaluations).
query the MFM to determine their relative alignment
Image classification. The MFM classifies the image from a list of predefined classes. For efficiency, we follow (Jiang et al. 2024a) and classify images in batches, which does not significantly harm accuracy.
3.1 ACCOUNTING FOR ALGORITIMIC CONSTRAINTS
Using superpixels and the zooming algorithm can impose constraints on MFMs' performance. We address this two ways. First, we introduce largeted control buselines in all our experiments (see Sec. 4 to ensure fair and calibrated comparisons with other vision models. Second, in App. E.8 we
demonstrate that employing more fine-grained prompt chains (e-g., using more superpixels) yields consistent conclusions, confirming that our findings are robust to the chosen granularity.
4 EXPERIMENTS
We present the experimental results for different tasks and MFMs. First, we describe our setting, including the choice of the datasets and models. Then, we discuss our main results. We provide qualitative examples for all tasks in Fig. 3 Finally, we provide further analysis and ablations in Sec. Please see the appendix for additional results.
We evaluate closed- weight MFMs, namely GPT-4o (OpenAl 2024b), Gemini 2.0 Flash (Google Deep 2024), Gemini 1.5 Pro (Reid et al. 2024). and Claude 3.5 Sonnet An Thropic 2024 by querying them via their APIs. We also include
Evaluated MFMs.
VLL72BB (Wang al. 2024
and Llama 3.2 90B (Ai 2024 Meta) as recent open-weight models that yield competitive results with GPT-4o and Claude 3.5 on several benchmarks. In addition, we evaluate recent multimodal reasoning models such as ol, o3, and o4-mini (a lightweight distilled variant) (OpenAl 2024a; 2025b). We evaluate o4-mini across the full benchmark, and evaluate ol and o3 on smaller representative subsets due to cost constraints. For each model and task, we select the best prompt on a validation set for final testing.
Datasets. We use the following common vision datasets for evaluations:
Image classification. We standard benchmarks including ImageNet (Russakovsky et al., 2014) and ImageNet-v2 (Recht et al. 2019). To test robustness, we use ImageNet-R (Hendrycks et al. 202Tb, ImageNet-S (Wang et al. 2019 and two corruption benchmarks from RobustBench, namely,
5

---

Published as a conference paper at ICLR 2026
Table Image classification. We compare the top-1 acc. (1) of the
Table 2: Grouping. Com- parison of MFMs with SAM and SAM-2 (Kirillov et al. 2023b; Ravi et al.
MFMs with vision specialists, Model Soups (Wortsman et al. 2022)
and OpenCLIP (Cherti al. 2023). Although their performance falls short of the top specialist models, MFMs, particularly GPT-4o, demonstrate competitive results across a broad range of benchmarks.
2024 on the grouping task.
Model
ImageNet
Ies-eNNs-V2
Corruptions
Domain Shift
Model
mlo
2DCC
3DCC
InngeNet-R
ImageNet Skotch
SAME
30,57
SAM-2 + Chain
72.35
Soups
SAM
SI.12
OpenCLIP
84.37
78.33
66.96
65.95
93.76
73.24
SAM Chain
72.32
GPT-4o
77,20
71.57
62.46
61.13
$4.38
67.30
GPTT-4o
59.06
Gemini Flash
04-mini
74.78
$5.90
46.99
75.79
55.67
17.22
56.23
56.92
36.68
56.05
$2.05
45.18
69.43
04-mins
Gamini 2.0 Fladh
Gemini 1.5 Pro
Clanle 3.5 Sonnet
41.68
46.00
55.25
46.13
Clande 3.5 Sonnot
62.85
54.45
40.76
41.41
70.36
5742
Qwen2:VV VT.
21.64
Qwen2-VI.
53.54
49: 39
18.92
36.45
66.31
51.18
Llama 3.2
25,69
Llama 3.2
49.15
08.21
34.15
31,37
65.05
47.11
Orgcle Chain
$1.77
ImageNet-C (Croce et al. 2020 Hendrycks and Dietterich, 2019) and ImageNet-3DCC (Kar et al.
2022b).
Object detection. We use the COCO (Lin et al. 2014) validation set and choose images containing a single instance of each present class, resulting in 1.7K examples.
Semantic segmentation & grouping. We use random subset of 500 COCO (Lin et al. 2014) validation
images for semantic segmentation for cost-efficiency. For grouping. we filter 100 images from the COCO validation set by measuring the consistency of SAM (Kirillov et al.) 2023a) predictions
between different query points within every instance. More details are provided in the appendix.
Depth & surface normal prediction. We use Hypersim (Roberts al. 202T) and randomly subsample 100 validation images from it for cost-efficienyy.
Baselines. We include the following control baselines contextualize the performance of MFMs and account for the impact of superpixelation and other design choices:
Vision Specialist. We report the performance of leading vision models for each task. We specify each model used in the corresponding task sections, In addition to the state-of-the-art models, we benchmark 4M-21 (Bachmann al., 2024 Mizrahi et al.) (2023) as a zero-shot vision baseline for an unbiased comparison. Although 4M-21 is an MFM, here we treat it as vision specialist because it is specifically trained for solving these tasks. Overall, these baselines indicate the current state of the (specialized) computer vision models.
Vision Specialist + Chain. This control baseline applies the same algorithmic constraints to the vision specialist as those experienced by MFMs, such as superpixels and recursive zooming. This control baseline provides a fair and calibrated comparison between vision specialists and MFMs, ensuring that our benchmark remains accurate in a relative and ordinal sense.
Oracle Chain. This baseline shows the performance of the prompt chain if the MFM gave the ground-truth answer for each sub-task. It isolates MFM performance from limitations imposed by our chain's 's granularity. Note that this is not a hard upper bound, and can alleviate these limitations by using more fine-grained chains (see App. E.8)
Blind Guess. We prompt the model with blank image, revealing potential biases and assessing whether the model genuinely utilizes the image content for its predictions.
4.1 EVALUATION RESULTS
Object detection. The results are summarized in Tab. We use DETR (Carion et al. 2020 and Co-DETR (Zong et al. 2023). ,, a state-of-th -het--arrt COCO model, as the vision specialists, and 4M-21 as zero-shot baseline. We observe that all MFMs lag behind the vision models, with GPT-4o achieving the highest performance, significantly outperforming other MFMs. For the "Specialist + Chain" baselines, we apply the same recursive algorithm, but each stage we only keep the grid cells that intersect with the original bounding boxes predicted by the specialist. As mentioned earlier, this calibration confirms that the gap between MFMs and specialists remains significant.

---

Published as a conference paper at ICLR 2026
Table 3: Object Detection: Average precision of MFMs vs, vision specialists (DETR, Co-DETR) and 4M-21, GPT-40 leads among MFMs.
Table 4: Semantic Segmentation: mIoU and pixel accuracy of MFMs vs. OneFormer and 4M- 21. GPT-4o again performs best among MFMs.
Baselines
Model
A"
AP C*)
AP
Baselines
Model
mloli
Pixel (1)
Co-RETR
91.
96. 17
90:35
OneFormer OneFormer + Chain
65.52 52
83,26
Vision Specialists
DETR
DETR + Chain
CINRTT + Chiin
90.06
72.33
73:31
52.78
03.51
38.36
59.07
39.36
$1.54
Vision Specialists
4M-21
54.31
60.64
79.66
$1.69
iM-21
59.54
51.57
42:71
4M-21 + Clain
52.72
78.59
4M-21 + Chain
55,46
30,48
30.74
GPT-4o
44.89
68.60
00:62
31,97
o4-mini
39.19
64.26
MPMs
Flash
10.75
MFMs
Oemini 1.5 Pro
Gemini
40.46
64.88
Classk:
Claude 3.5 Sonnet
32.05
58.41
Owend
35.62
Qwen2-VL
Liama
31,57
Llama 3.2
36.63
59.95
Canteel
Ocaele + Clais (fall)
Ocarle Cleii (prod. clone)
Ulind guess
+0.01
92.18
75.44
+0000
41.31
40.33
<0.01
41.56
50:14
Control
Oraece + Chain
Blind guess
83.41
94.68
Finally, we assess the performance of the "Oracle + Chain" baseline. We evaluated two variants: one using GPT-4o's class predictions, and another using the ground-truth class labels. The first baseline examines the outcome if GPT-4o correctly selects the grid cells at each step of the chain, while the second assumes both correct class predictions and accurate grid cell selection. These provide the upper bounds for both the grid search component and the overall pipeline when using specific grid resolution to calibrate the performance.
Semantic segmentation. Table and F show that MFMs achieve rather non-trivial performance, yet they still lag behind the vision specialist, i.e. OneFormer (Jain et al. 2022). Similar to object detection, we include the baseline of constraining the performance of the vision specialist using the chain algorithm: we assign the majority class prediction to each superpixel and flood-fil the entire superpixel with that class.
Grouping.
Table shows that MFMs have varying performance on this task, and GPT-4o performs the best, achieving overall good performance as can also be seen in Fig. 3 All MFMs still lag behind the vision specialist SAM (Kirillov al.) 2023a).
Depth prediction. The results are summarized in Tab. 5 Alongside standard metrics, we also report 1) the Spearman correlation coefficient (p). which serves as a relative metric by measuring the correlation between the ground-trun -ruth depth ranking of the pixels and the predicted ranking and 2) accuracy, which reflects the percentage of correct pairwise depth comparisons. MFMs demonstrate nontrivial performance and outperform the blind guess baseline. Notably, o4-mini achieves the strongest performance among MFMs, despite trailing behind some models in the semantic tasks shown
before. Still, there remains significant gap compared to vision specialists like Omnidata (Eftekhar
al. 202T: Kar et al.) (202a)). Lotus (He al. 2025), and MoGe-2 (Wang et al.) 2025). which is more pronounced compared to the semantic tasks.
Similar to previous tasks, we analyze the performance using the "Oracle Chain" baseline, which assumes 100% accuracy in all pairwise comparisons. and the "Omnidata + Chain" baseline, which uses depth predictions from Omnidata to perform these pairwise comparisons. Due to the coarse granularity of the evaluation, the two baselines closely match each other. Importantly, we find that MFMs still lag behind these baselines, suggesting that our conclusions hold despite the chosen granularity level (see App. E.S)
Surface normal prediction. To assess performance, we employ Spearman's rank correlation coefficient, Ps. measuring the correlation between ground truth and predicted pixel alignments along each basis direction i. Aligmment for a pixel is measured as the dot product of the surface normal with the direction i.
Tab. T demonstrates that most MFMs struggle with this task, with some showing negative correlation for certain directions, revealing a systematic bias in their understanding of these directions. Notably, 04-mini outperforms all other MFMs, indicating stronger geometric understanding. This extends to other recent reasoning models, with ol and o3 also showing strong performance on the evaluated subset.
Furthermore, we show in the appendix that the MFMs not directly trained for reasoning improve their performance on the up-down ambiguity resolution with CoT prompting (Wei et al. 2022). Similar to
7

---

Published as a conference paper at ICLR 2026
Table 5:
Depth prediction. MFMs can coarsely estimate depth, but their gap to vision specialists is larger than in semantic tasks. All MFMs perform similarly, with GPT-4o and 04-mini slightly ahead.
Table 6: Surface normal prediction. For surface normal prediction, MFMs again show large gap specialists, particularly on the r-axis. 04-mini is the strongest MFM, while Gemini of- ten fails, performing near chance.
Higher is better
Baselines
Model
5,
4g
6g
Omnidata
0.768
0.867
0
0.95
Baselines
Model
Pb
Omnidata Chain
0.568
0.772
0.864
0.81
Onnidaata
0.80
0.83
0.78
Vision
Lotus Chain
Lotus
0.776
0.578
0.861
0.779
0.913
0,866
0.93
0.82
Vision
Lotus Chain
Lots
Omnidata
Clain
0.64
0.64
0.77
0.70
0.70
0.80
0.75
0.54
Moge
0.795
0.859
0,903
0.90
Mage
0.80
0.80
Moge+ Chain
0.576
0.774
0.861
0.82
Moge Chain
0.64
0.69
0.56
4M-21
0.636
0,814
0,888
0.89
DSINE:
0.76
0.77
0.75
4M-21 + Chuin
0.565
0.774
0.865
0.81
DSINE Chain
0.63
0.69
0.55
0.74
GPT--40
0.461
0.716
0.840
0.54
4M-21 Chain
0.65
0.70
0.56
04-mini
0.467
0.718
0,841
0.58
GPT-4a
-0.14
0.57
0.40
Gemini 2.0 Flash
0.461
0.715
0.839
0.59
o4-minii
MFMs
Gemini 1.5 Pro
0.458
0.709
0.835
0.51
Gemini 2.0 Hash
40.39
-0.04
0.02
Claude 3.5 Sonnet
0.428
0.693
0,830
0.49
MFMs
Gemini 1.5 Pro
-0.17
-0.57
0.01
Qwen2-VL
0.432
0.698
0.831
0.44
Claude 3.5 Sunnet
-0.19
0.61
0.40
Qwen2-VL
0.09
-0.07
Llama 3.2
0.458
0.711
0.835
0.53
Llama 3.2
0.41
-0.42
0.22
Control
Oracle Chain
Blind Guess
0.375
0.571
0.774
0.628
0.863
0.773
0.83
0.25
Control
Oracle Chaan
Blind guess
-0.48
0.64
-0.1
0.70
0.60
depth, these results suggest that while MFMs have limited 3D visual understanding, newer reasoning models like ol, o3, and 04-mini exhibit promising progress.
Image classification. The classification results across all datasets are summarized in Tab. [ We use Model Soups ViT-G (Wortsman al. 2022) the vision specialist, and we also include OpenCLIP H (Cherti et al. 2023) to assess zero-shot capabilities. Although MFMs do not reach the performance levels of vision specialists, they demonstrate strong results and resilience to corruptions and distribution shifts. Notably, GPT-4o and Gemini 2.0 Flash stand out with a particularly strong performance, followed by Gemini 1.5 Pro. Claude 3.5 Sonnet, 04-mini, Qwen2-VL. and Llama 3.2. We also observe that o4-mini is especially sensitive to the batch size used during inference, with performance improving at smaller batch sizes (see ablation in the appendix).
Reasoning models. As discussed earlier, we evaluate ol and o3 on a representative subset of images (see App. F for details). For comparison, we also include GPT-4o as baseline, along with evaluations of 04-mini at varying levels of reasoning effort (low, medium, and high). The re- sults are presented in Fig. 4 ol and o3 slightly outperform GPT-4o on most semantic tasks, and significantly outper- form it geometric tasks. As before, GPT-40 excels in semantic tasks on this subset, while 04-mini performs better on geometric ones. We find no clear trend with the reasoning effort; medium and high reasoning improve results over low, but not always. We refer the reader to the appendix for further details and experiments.
Classiication (ecereayy)
Otpei Deteedoe
Morrels
Oepth
Segmortatio (nBUT)
Groading
yetoU"]
Figure 4: Evaluation of reasoning mod- els. The performance of ol, o3, and 04-mini (at varying levels of reasoning effort) compared against GPT-4o on a representative subset of images. The reasoning models exhibit a particularly strong performance in geometric tasks.
GPT-40 with image generation capability. Recent MFMs, such as the updated GPT-4o (OpenAT 2025a), can generate dense image outputs rather than being re- stricted solely to output text. This capability represents promising advancement, potentially enabling compre- hensive evaluation on diverse vision tasks. However, the current image generation capability exhibit several limitations, some of which we illustrate in Fig. 5 Specifically, we observe that generated outputs suffer from spatial misalignments and hallucinations, as also observed in Chen al. (2025).
This presents challenges for directly applying this model to vision tasks, which we leave to future work. We provide further qualitative examples and preliminary quantitative analyses in App. G
8

---

Published as a conference paper at ICLR 2026
Object detection
Grouping
RGB
Ground Truth
Prediction
Rcai
Ground Truth
Prediction
Depth estimation
Surface Normals
Ground
Prediction
R38
Ground Truth
Prediction
Figure 5: Failure cases of GPT-40 with image generation capability. Despite the model's promising capabilities, limitations remain. Here, we highlight some typical failure modes: hallucinations (marked in dotted blue) and inaccurate predictions (marked in dotted green).
4.2 ANALYSIS
Prompt chaining vs direct prompting. We analyze the impact of using prompt chains versus directly asking the models to solve tasks in a single prompt in Tab. 7 For bounding box regression, we directly query GPT-4o to predict its coordinates; for semantic segmentation, and request corresponding semantic labels. The results indicate a clear performance boost from using the prompt chains. Please App. for detailed discussion, qualitative visuals, and ablations.
mark image regions
Prompt sensitivity. The choice prompt can greatly influence the performance of model, and we have made reasonable efforts to optimize prompts for each model. We evaluate the MFMs across various prompts to assess their sensitivity to word choice and prompt structure. We then select the most effective prompt validation set for the final results presented Sec. comprehensive analysis is provided in App. H showing that there is some variation in performance with different prompts, and the performance is generally less prompt- dependent for better-performing models, e.g., GPT-4o. GPT-4o.
Table 7: Prompt chaining ablation. We compare the performance the prompt chaining algorithm to direct prompting on GPT-4o for semantic segmentation and ob- ject detection. 'Segmentation is on 100 images.
small
Task
Direct prompting
Prompt Chaining (Ours)
Segmentation f
Object Detection (APso 1)
17.69
60.62
Analysis of Failure Modes. While the closed nature of MFMs prevents directly probing internal representations, we observe systematic failure patterns across several axes:
First, in semantic segmentation, providing full images with the superpixels merely outlined yields poor results (see App. D. whereas multi-scale crops improve performance. This supports findings that MFMs suffer from "blurry vision" and struggle with fine-grained localization (Fu et al. 2024).
Second, our blind-guess analysis (App. reveals a "ceiling bias": models tend to assign larger depth to pixels higher in the image. We observe this trend empirically across several images, where models consistently attribute greater depth to pixels higher in the image.
Third, in surface normals, models exhibit coordinate- frame ambiguities, frequently inverting the x and y-axes (e.g., confusing left and right facing surfaces). We find in App. that these errors be partially resolved by CoT prompting.

---

Published as a conference paper at ICLR 2026
Overall, MFMs appear to rely heavily coarse semantic cues while exhibiting unstable fine- tne-grained
geometric understanding.
In-the-wild evaluations. Previously, we used standard vision datasets like ImageNet and COCO in our cvaluations. Given the opaqueness on what training data was used in training the MFMs, one cannot be confident that those images were not used in training. This, so called "data contamination', problem a broad concern for the community regarding most MFMs (Jacovi et al. 2023). To assess to what extent the MFMs generalize to entirely novel data and ensure our evaluations are not distorted by potential data contamination, we curated a collection of images released online after the specific model APIs were launched (Flickr [2024; Unsplash] (024), which the MFMs could not have encountered before their knowledge cutoft date. Results in App. E0 show a good generalization
performance to the in wild samples. Therefore, do not find evidence for data contamination with standard datasets to be a concern for our evaluations.
5 LIMITATIONS AND CONCLUSIONS
We present benchmark to investigate the vision capabilities of MFMs by translating standard computer vision tasks into an API-compatible format that can be solved via prompt chaining. Our results show that current MFMs have relatively stronger performance in semantic tasks compared to geometric tasks, and GPT-4o is generally the best-performing model, followed by Gemini 2.0 Flash and 1.5 Pro, 04-mini, Claude 3.5 Sonnet, Llama 3.2, and Qwen2-VL--22B. Despite recent advances, MFMs still lag significantly behind vision specialists, even when specialists are evaluated under the same constraints of the prompt chain. However, recent reasoning models such o1, o3, and 04-mini show promising performance on geometric tasks, indicating growing capabilities in 3D understanding that complement their already strong semantic performance. We conclude with some limitations and future directions:
Inference cost. The multiple API calls per sample (e.g., for classifying batches of superpixels) can result in a higher computational overhead than standard single-query vision- lision-language evaluations (see App. D.. While this is shortcoming, we emphasize that the framework is designed benchmarking tool for assessing visual capabilities, rather than efficient querying for downstream applications. While efficiency is important, it is orthogonal to objective, and promising direction for future work.
one-time
Optimality of the proposed prompt chains. Research into advanced prompting techniques has the potential to further enhance the performance of MFMs on classical vision tasks, beyond what is shown in this paper. However, demonstrated by our prompt sensitivity analysis, stronger models generally exhibit a lower reliance on specific prompting strategies, becoming less sensitive to exact prompt formulations. Nonetheless, while the final design appears simple, proposed prompt chains are carefully designed, emerge from a vast search space, and consistently improve upon direct prompting (Sec. 4.2 and App. E.. Furthermore, also included careful controls and analyses to disentangle the impact of the prompting method from the benchmarking conclusions.
We emphasize that our framework is for benchmarking, not a production-ready method for solving these tasks. Indeed, we expect that future MFMs with any-to-any capabilities will likely close the gap to specialist vision models by training directly on these tasks. Our framework, however, can benchmark any MFM with image-input and text-output capabilities. Through that, we establish the first benchmark for comparing diverse range of MFMs, both against each other and against specialist vision models, on standard vision tasks.
Generalization of the Framework. We view this benchmark as building block for systematically eval- uating image-based semantic and geometric understanding. While our prompt chaining framework is generic, specific tasks may require different discretization strategies (e.g., sliding windows) rather than superpixel grouping. More broadly, extending this method to encompass video understanding and complex visual reasoning is an exciting direction for future work.
ACKNOWLEDGMENTS
This work was supported as part of the Swiss AI initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID 43 Alps. This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI). We also acknowledge a gift from Google.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to ofate-of-the-ar superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11)22274-2222, 2012.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida. Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08574. 2023.
Meta AI. The Ilama 3 herd of models, 2024.
Haider Al-Tahan, Quentin Garrido, Randall Balestriero, Diane Bouchacourt, Caner Hazirbas, and Mark Ibrahim. Unibench: Visual reasoning requires rethinking vision-language heyond scaling. arXiv preprint 2.xiv:2408.088100 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 237736,,--373,, 2022.
Authropic.
Introducing claude 3,5 sonnet.
c Slaude 3- 5- sonnet 2024. Accessed: 2024-09-23.
Roman Bachmann, Oguzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Grifliths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4m-21: An any-to-any vision madel for of tasks and modalities. arXiv preprint arXiv:2406.0999 09406, 2024.
Andrew J. Davison. Rethinking inductive biases surface normal estimation, 2024. URL Gwangbin http://aaww.....//a////2...... 30712
hittps://arxiss
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vim for transfer. arXiv preprint 220x::2007.07726. 2024.
Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: large benchmark and model for 3d object detection the wild. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13154 13164, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnacve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCY ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt I 16, pages 213-229. Springer, 2020.
Mark Chen, Jemry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
Sixiang Chen, Jinbin Bai, Zhuoran Zhao, Tian Ye, Qingyu Shi, Donghao Zhou, Wenhao Chai, Xin Lin, Jianzong Wu, Chao Tang, et al. An empirical study gpt-4o image generation capabilities. arXiv preprint arXiv:2504.05999 2025.
Wei-Lin Chen, Cheng- Wu, Yun-Nung Chen, and Hsin-Hsi Chen. Self-icl: Zero-shot in-context learning with self- ref-reeeeateed demonstrations, 2023. URL https://arxiv.orgg/ wttp://wwgbbbbbb...o//bb/ 15035
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language- image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-2829, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating Ilms by human preference. arXiv preprint arXiv:2403.04133 04132, 2024.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag. Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateck Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09677 2020.
11

---

Published as a conference paper at ICLR 2026
Wenliang Dai, Jumnan Li. Dongxu Li. Anthony Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https YaZJA
Ainaz Eftekhar, Alexander Sax, Roman Bachmann, Jitendra Malik, and Amir Roshan Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. 2021 IREE/CVF International Conference on Computer Vision (ICCV), pages (0766-10776, 2021.
Flickr. Find your inspiration. attps://www. flickr. com/ 2024. Accessed: 2024-09-23.
Xingyu Fu, Yushi Hu, Bangzheng Li, Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXiv:240.12339, 2024.
Google. Explore vision capabilities with the gemini api. https://al.google..Be bbcs:/visccnnaagggyytttcc 2024. Accessed: 2024-09-23.
Google DeepMind. Gemini 2.0 flash. Flash/| 2024.
Jing He, Haodong Li. Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying-Cong Chen. Lotus: Diffusion- visual foundation model for high-quality dense prediction, 2025.
URL/https: /tattps://axiien org/ abs// 1ttp///229911111.....1 18124
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv:1903. 2019.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-of- distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-884000 03400083342200 2021.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. arXiv preprint arXiv:2406. 09403, 2024.
Huggingface. Llama multiple images.
11B Vision- Instruct/d scussiccccom ins///nnns3333
Alon Jacovi, Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305. /0/60, 2023.
Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One trans- former rule universal image segmentation, 2022. httttp/:/aaaxv..oog//////// httegg/ab///211.. 06220
Yixing Jiang, Jeremy Irvin, Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning multimodal foundation models. arXiv preprint arXiv:2405.09798 2024a.
Yixing Jiang, Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H. Chen, and Andrew Y. Ng. Many-shot in-context learning in multimodal foundation models, 2024b. httpss//////xx
abs 0246.2405.oo0 09798
Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir 7amir. 3d common corruptions and data augmentation. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18963- 18974, 2022a.
Oguzhan Fatih Kar, Teresa Yeo, and Amir Zamir. 3d common corruptions for object recognition. In ICML 2022 Shift Happens Workshop, 2022b.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks, arXiv preprint 210.02220.002406,
2022.
12

---

Published as a conference paper at ICLR 2026
Alexander Kirillov,, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehcad, Alexander Berg, Wen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ArXiv, abs/2304.006433, 2023a. |tttttss:://api
http:///pp...s.a....emmme semant hrt.aaaaissssslo.a..oooooooooooo
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2.04.02644 2023b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2.00. 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
Meta. Llama 90b.
https://wwet www. 11mma.aamm.mmmmmmmmmmmmm docs/ caddaa/c prompt formats/
1lama 3 2/
David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling, In Advances in Neural Information Processing Systems, 2023.
OpenAl. Introducing openai ol. httpp://gpenai.comm/0 2024a
OpenAL. Hello gpt-4o. httpp:///ggnaal.ooiineeee. htch:/hhello-pppp
g-t 40/ 2024b. Accessed: 2024-09-
23.
OpenAL.
Introducing
40 image generation.
ht hc:ps://ppenal.com//eeas
int ratrociing-- 40 4o-1gge--mmmmeeeeee Seraage--eeee wwweeeeeeetine icn/ 2025a.
OpenAl.
Introducing openai o3 and 04-mini.
hc:ps://openal.com/ndeee
int -
2025b.
Ponyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Tacsiri, and Anh Totti Nguyen. Vision language models are blind. arXiv preprint 2000::207.00681, 2024.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao Yuan Wu, Ross Girshick. Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos, 2024. URI http:/taaa///aiii...oo
/2408 00714
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389 5400. PMLR, 2019.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptisa Alayrac, Radu Soricut, Angeliki I Lazaridou, Orhan Firat, Julian Schrittwieser, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
David Rein, Betty Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-proof q&a henchmark. arXiv preprint arXiv:2311.12022 2023.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10912 10922, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 11::211-252, 2014.
David Stutz, Alexander Hermans, and Bastian Leibe. Superpixels: An evaluation of the state-of-the-art. Computer Vision and Image Understanding, 166:1-27, 2018.
13

---

Published as a conference paper at ICLR 2026
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09878 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Sean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2112.1180. 2023.
Shengbang Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision- cenoriccnttic exploration of multimodal lims. arXiv preprint 2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms. arXiv preprint wrXiv:2401.06209 2024b.
Unsplash. The internet's source for visuals.
2024. Accessed: 2024-09-23.
Haohan Wang, Songwei Ge, Zachary L Lipton, and Fric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Jianfeng Wang. Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-ter transformer for vision and language. arXiv preprint arXiv:2205. 14100. 2022.
Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 1XXi:2409.12191,,
Ruicheng Wang. Sicheng Xu, Yue Dong, Yu Deng. Jianfeng Xiang, Zelong Lv. Guangzhong Sun. Xin Tong, and Jiaolong Yang. Moge-2: Accurate monocular geometry with metric scale and sharp details, 2025. URL
hhtpss:/aaxxi..org/aaaaaa
6
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rehecca Roclofs, Raphael Gontijo-Lopes Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. http::/ arxii o abs 2203. C5492
Penghao Wu and Saining Xie. V*: Guided visual search of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 13084-13094, 2024.
mechanism in multimodal 1lms. In Proceedings
Yixuan Wu, Yizhou Wang. Shixiang Tang, Wenhao Wu, Tong He. Wanli Ouyang, Jian Wu, and Philip Tor. Det- toolchain: A new prompting paradigm to unleash detection ability of milm. arXiv preprint arXiv:2403. 12488, 2024.
Jianwei Yang, Hao Zhang, Feng Li. Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 1441, 2023a.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision), 2023b. htRLh:tts://aatt
org/ abs/
2309. 17421
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Delibcrate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun. Tong Xu, and Enhong Chen. survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9556 9567, 2024.
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401. 13601, 2024.
14

---

Published as a conference paper at ICLR 2026
Bolei Zhou, Hang Zhao, Xavier Puig. Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5122-5130, 2017.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe Le. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint 2rXiv:2205.10625, 2022.
Zhuofan Zong, Guanglu Song, and Liu. Dets collaborative hybrid assignments training. 2023. URL
Daniel Zoran, Phillip Isola, Dilip Krishnan, and William Freeman. Learning ordinal relationships for mid-level vision. Proceedings of the IEEE international conference on computer vision, pages 2015.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
TABLE OF CONTENTS
Overview Video
21
Code & Full prompts
21
Qualitative examples
21
D Additional Details on Prompt Chaining
21
D.1 Object detection
D.2 Segmentation
23
D.3 Depth prediction
23
D.4 Surface normal prediction
25
Additional experimental details and results
27
Object detection
27
Semantic segmentation
27
Grouping
29
Depth prediction
31
Surface normal prediction
E,6
Experiments with Llama
31
Blind guess
32
E.8
Finer-grained prompt chain
33
E.9
Performance Ceiling for Geometric Tasks
34
10
In-the-wild evaluations
34
Reasoning results
36
F.1
Ablating the batch size
36
F22 Detailed Experiments
37
Additional experiments with 4o Image Generation
38
Prompting Methodology
38
G.2
Preliminary Quantitative Analysis
40
Prompt sensitivity analysis
40
Prompting Costs
40
16

---

Published as a conference paper at ICLR 2026
Ground Trubs
GPT-4o
Genén Pash
Gambe 1.3Pa
Claude 33 Soreet
Queea 738
Ulmea 3.2
Delecton
Grappigg
Supeumstte
Surfic Rormal
Figure 6: Additional qualitative results for MFM predictions on different tasks.
17

---

Published as a conference paper at ICLR 2026
Ieput
Ground Truth
GPT-4o
Gemini 2,0 Flash
Gemini 15
Claude Sonnet
Owen2VV 728
Liana
Dutaction
Groupii
Dapai
Figure 7: Additional qualitative results for MFM predictions on different tasks.
18

---

Published as a conference paper at ICLR 2026
Detection
Groupina
Segmantaation
SurTAck Normal
Depith
Figure 8: Qualitative results for reasoning model predictions on different tasks next to GPT-40 prediction.
19

---

Published as a conference paper at ICLR 2026
Input Image
Ground Truth
Generation
4o Image
Input Image
Ground Truth
4o Image Generation
Detection
Grouping
Depth
Normal
Figure 9: Additional qualitative results for GPT-4o image generation predictions across tasks. Notice
the various failure cases such as spatial misalignment these examples, as outlined in the main paper
Fig. 5
20

---

Published as a conference paper at ICLR 2026
A
OVERVIEW VIDEO
narrated overview video including the paper's method, quantitative and qualitative results, as well interactive visualizations, is provided in the supplementary material.
B CODE & FULL PROMPTS
We provide our code and full prompts in the supplementary material.
C QUALITATIVE EXAMPLES
We provide additional qualitatives in Figures 6 different tasks.
and 9 to show each model's performance
D ADDITIONAL DETAILS ON PROMPT CHAINING
D.1 OBJECT DETECTION
Different variations of classification for object detection. As discussed in Section 3 the first stage of the object detection pipeline involves identifying all the objects present the image. We attempt the following two strategies for the multi-label classification task:
The first strategy simply provides the model with the entire image, asking it to identify all present classes.
The second strategy divides the image into five regions: four quadrants and a center crop.
The model is asked to identify the classes present the regions in independent queries. With each query, the full image is provided for additional context. The final prediction is obtained by taking the union of the classes identified across all regions (see Algorithm [ in the appendix for detailed pseudocode). This approach typically improves recall but may reduce precision, reflecting a trade-off between the two strategies.
The precision-recall trade-off for the models is described in Tab. 8 To pick the best classification strategy for the models, we run the oracle on the predicted labels on a small subset and pick the one that yields the highest AP.
After we find the object labels, we run the procedure described in Algorithm 2 to regress the bounding boxes. Figure 10 also provides visualization of the mentioned algorithm.
Agaittioe Staps
Predicted Box
Imagee
Stop
Seep 2
Stop
- MFM
MFM
MPM
commin
steep
Figure 10: Object detection algorithm. At each step, we divide the image into 3 grid of crops and query each for the presence of the target object (Sheep in the figure) through the model. Grid cells without the object are discarded, and the process repeated until the object fully located. "Summary of the actual prompt; see the full prompt in the provided Markdown files.
In Alg. 2
we explored batching the grid-cells and querying them independently. While batching didn't 't affect results for most MFMs, it significantly deteriorated the performance for Gemini, opted to use independent queries for Gemini instead.
21

---

end while Published as a conference paper at ICLR 2026
Table 8: Classification for Object Detection: The results clearly show the precision-recall trade-off between using the two strategies for multi-label classification.
Strategy
Model
Precision
Recall
GPT-4o
97.5
75.75
Strategy
Gemini 1.5 Pro
90.5
83.81
Claude 3.5 Sonnet
84.27
81.24
GPT-4o
89.05
88.37
Strategy 2
Gemini 1.5 Pro
84.37
89.3
Claude 3.5 Sonnet
78.18
85.94
Algorithm Region-based Image Classification
1: procedure REGIOA-BBSSOQCASSSHCCCXIOOOOOOO
2: regions +- (ividemmoRegiion((mmqee)))
3: allClasses <
4: for region E regions do
5: classes + QueryMFM region)
6: allClasses +- allClosses classes
7: end for
8: return allClasses
9: end procedure
procedure (IVIDIINTOREGIOOS(mmae)) quadrants &- dividelntooQuaitannssimaagg
11:
12:
center ExtractCenterCroppl (image)
13:
return quadrants U {center)
14: end procedure
Algorithm/ 2 Recursive Grid-Search
1: procedure COARFFGGRIISFARRCHHmmee object, gridStructure)
2:
while search space can be reduced do
3: cells + DividelntoGridí (immggg gridStructurr))
4:
relevantCells +- {c E cells:
5:
QueryMFM(c, object) = TRUE)
6:
image ChopToRdeewntCClllinnng.g relevantCells)
7:
8: return image as bbor
9:
end procedure
10: procedure QUERYMFM(cell, object)
11: return MFM classification of object presence in cell
12: end procedure
22

---

Published as a conference paper at ICLR 2026
Alporitin Steps
Input Image
Superpiani
Sagmentation
Saperponite
Superpisel Multi Scale Pyramid
Superpowel Multi Scale Pyramid
Predicted
Predicted Class
Sky
Locomotive
Input Prompt"
MFM
MFM
Specify the the object in this superpixe!
Figure II: Semantic segmentation algorithm. We divide the image into superpixels and create "multi-scale pyramids" of superpixels. The pyramids are then classified using the MFM in sequential manner to produce the complete segmentation map. multi-scale pyramid consists crop of the superpixel, some context surrounding the crop, and the full image. In practice we batch multiple superpixels into sequences and classify them jointly. "Summary of the actual prompt; see the full prompt in the provided Markdown files.
layers:
Inpal Promutt
indianoo the Aeyyane
MFM
MFM
Predkzed Orogping
Superpiziddy
Choose Asjiooe, Odl's
Me ge Chense Arp.oetttttttoooooo....oo.....ea Cale
Merge and AC
Figure 12: Grouping algorithm. Given an image and query point, first divide the image into superpixels and select the superpixel that the query point falls into. At each step, the model is asked to identify the adjacent superpixels that belong the same object as the one covered by the cluster. The selected superpixels are then merged with the cluster to form the next step's 's input cluster. "Summary of the actual prompt; see the full prompt in the provided Markdown files.
D.2 SEGMENTATION
The procedures for supervised segmentation and grouping described in Algorithm (see Fig. and Algorithm (see Fig. [2 respectively.
11
D.3 DEPTH PREDICTION
The procedure for depth prediction is given in Algorithm 5 Furthermore, a visualization of the depth prediction algorithm given in Fig. [3 crucial part of the algorithm involves optimizing the objective to obtain the overall depth rankings. To formulate the objective for globalizing the pairwise depth rankings, we repurpose the objective Zoran et al. (2015). Given the vector of global rankings E N we first consider instances where superpixel is predicted to be at greater depth than superpixel j. The corresponding objective formulated as:
Cge(x)
Σ (x) - 1Ci-xz-- 1)2
(I)
23

---

Published as a conference paper at ICLR 2026
Algorithm Superpixel Segmentation
1: procedure SEMSECMEENTATTONNNNNe, batchSize, batchSi
scaleList)
2: superpizels +- SLIC(image)
3: classi desssr/iieSueeepiizeell +-0
4: history + 0
5:
for t to wength((suerpireel step batchSize do
6:
batch GetBatch superpirels, batchSize)
7:
pyramid + CreacSemmnnicPPramid
image, batch, scale List)
8:
batchClasses +- ClassifyBatch(
pyramid, history)
9:
classi fiedS uuerppppss +-
classi wea.fiiSSSpppppie U batchClasses history t UpdateHistory( history, batchClasses)
10:
11:
end for
12:
segmentedl mage FloodFillSuperpiiee(( image, classi (eduuppiisiss) return segmentedImmag
13:
14:
end procedure
Algorithm BFS Segmentation
1: procedure INSTAANCEGRRUPPNNNKnnggee..c queryyyinnd batchSSzze, scale
2:
superpizels +- SLIC(image)
3: graph + ConstrutSuueppteeerrrrpeeeeprrre startNode < FindSupperpxeelConannnngg superpizels, query
4:
5:
cluster + [start.Noee)
6:
queue + new Queue()
7:
mueuue.eqqeuucstarbbe ode) visited + {start. ode} while not queueeisEEptt(() do
8:
9:
10:
batch +- QeBBacchrrrnnnuuueeeeeeeeee batchSize) batchP + CreateSemantiiPyyrnndd
11:
image, batch, scale
12:
cluster Pyr CreateSemantticPyraniie
image, cluster, scale
13:
NewM embers QucryMFM batchPyr, clusterPyr)
14:
cluster t- cluster J new embbers queue, visited +- UpdateeuuueendViiiiee graph, new embers, visited)
15:
end while
17:
return cluster
18:
end procedure
24

---

Published as a conference paper at ICLR 2026
Algorthi Steps
Input Image
Supersisel Rankings
Estimeled Depth
API
Supppoers
Rxperptnt ia axprrptat
Superpixol
Superphoal
Depth Order
Dupth
at greater depth than Red
greater depth than Blue
Input Prompt"
MFM
MFM
Whtch of two regions colored bike ond rea is os grwater depit??
Figure 13: Depth prediction algorithm. We randomly select pairs of superpixels. Each pair is given to the model to perform a pairwise depth comparison. The resulting pairwise ranks are then globalized by minimizing an objective function to generate a relative depth map, which can then be scaled to obtain classical evaluation metrics. "Summary of the actual prompt; see the full prompt in the provided Markdown files.
This objective encourages 24, ranked at a greater depth than 2y, to take higher values. Similarly, an analogous objective Cu can be defined for superpixels x; predicted to be at depth less than zz-
Following Zoran et al. (2155), we include smoothness regularization term to stabilize the depth estimates:
(x) Σ (1:=-,z) x,y)
This regularization is applied over pairs of adjacent superpixels and j. promoting smooth transitions between their depth values.
The final objective that needs to be minimized is a weighted sum of the above terms:
a min Cgt + Aucu + A,C.)
(3)
where Àgt, Au. and Å are the weight parameters, For experiments, select Age Au- 1 and Ag based on the performance on a smaller validation set.
To obtain metric depth estimates, assume access to ground-truth depth values for the purpose of scaling. Specifically, after floodfilling the values æ. we generate complete relative depth map d. Given the ground-truth depth map d', we optimize the following objective to determine the appropriate scale and shift parameters:
M
(s.t) arg min 8,
Σ (sd, + - d;
where M the total number of pixels in the image. By solving this optimization problem, we can then scale and shift the relative depth map d to align it with the metric depth.
D.4 SURFACE NORMAL PREDICTION
The procedure for surface normal prediction detailed Algorithm 6 and Fig. 14 While the model makes binary decisions regarding whether depth is lesser greater than another, have found that enabling the model to also consider equality predictions enhances the accuracy surface normal estimates.
25

---

Published as a conference paper at ICLR 2026
Algorithm 5 Depth Prediction
1: procedure ESTIM.ATEDEETHHmmggg.. numPairs)
2: superpixels t- SLIC(image)
3: Shfrrisseanniinge <-
4:
for i to wummaairs do
5:
pair <- SampeeRandmmaaitssuurrrrree ranking &- (porr)
6:
7:
pairwise Rankings
pairwise Rankings U {ranking)
8:
end for
9:
global Rankings +- MinimizeeOjecttive )airuise Rankings)
10:
aap + ossignDephTTPiixllss
tmagge,superpicee superpizels, global Rankings) return Map
11:
12:
end procedure
To incorporate this into our approach, we introduce the following term for cases where superpixels x; and zj are predicted to be at equal depth:
(x) = -- (2; - =,)²
(5)
1.
for pairs of superpixels x; and æj predicted to lie at an equal depth. For weights, we choose Xeg Ate Agt 1 and as before, we pick X based on the performance smaller validation set.
Algorithm 6 Surface Normal Prediction
1: procedure ESTINAYENNORMALLSCnnnne numPairs. bases)
2: superpirels + SLIC(image)
3: pairwise.Align + {
4: for 1 nummaairs do
5: pair t SampleRamdomPairt superppiiiisss
6:
for 6 in bases do
7:
alignment < QueryMFM(pair, b)
8:
pairwriseAlign/f
pairuiiseAAigmbb [b] U {alignment)
9:
end for end for normalMaps + 0
10:
12:
for in bases do global.Align MinimiieeGloballhjjectiie AaiguriseAllgnnb))
normal [ps|b] + AssignAlignmennTToPP image, superpizels, (obballligg) end for 16:
return Mormallapps
end procedure
To visualize surface normals, we take the per-axis predictions and normalize them to [0,1], after which we project them onto the unit sphere. We directly interpret the three channels RGB values. Note that since the per-axis normalized surface normal predictions do not present absolute directional information with respect to the camera, the colors might not match the ground truth visualizations.
26

---

Published as a conference paper at ICLR 2026
Appettx Skips
Imput Image
Superpleel Pairwise
Calls
Bankings per Direction
Ropeptnr icdeprettniin
Globalize
Superpixel
Superpixel
Predicted Surface Direction
equalty towards right
...
egisity
Prompt"
MFM
MFM
Corpone the ciecserr in witch the bue and region
the airectlass
Figure 14: Surface normal prediction algorithm. Similar to depth in Fig. 13 we randomly select superpixels and give them to the model to perform a pairwise comparison. The superpixels are compared based their alignment with the basis vectors relative the camera. The pairwise ranks are globalized to obtain the final result. "Summary of the actual prompt; see the full prompt in the provided Markdown files.
Figure 15: We ablate different ruler types as visual aids for object detection.
E ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
E. OBJECT DETECTION
We evaluate additional baselines for GPT-40 in Tab. In these experiments, the classification component of the pipeline remains unchanged, while the grid search is replaced with alternative methods. The results are clear: GPT-4o struggles with directly regressing bounding box coordinates. To address this, we experimented with overlaying rulers on the images to assist in bounding box regression, following insights from Wu et al. (2024), but found minimal improvement. The various visual prompts we tried are displayed in Fig. 15 and the numbers we obtained on subset of 100 COCO images summarised in Tab. 10
Additionally, we evaluate direct bounding box regression with Gemini, Owen2-VLL. and Claude (see Tabl) some of these models have demonstrated this capability (Google 2024). The results indicate substantial variance in performance: while Gemini and Qwen2- VL localize bounding boxes effectively. GPT-4o and Claude underperform considerably. Interestingly, despite improvements in Gemini and Qwen2-VLL they still lag behind the specialist models and do not surpass GPT-4o when using the chain algorithm.
E.2 SEMANTIC SEGMENTATION
We depict various marker types used for segmentation in Fig. 16 Furthermore, we conduct an ablation study on the marker type and the context provided during classification, shown in Tab. 11 The
27

---

Published as a conference paper at ICLR 2026
Table 9: Additional experiments with MFMs on object detection. Direct bounding box regression ineffective for GPT-4o and Claude 3.5 Sonnet, while Gemini 1.5 Pro and Qwen2- VL. perform better. For all models, the most effective prompt was selected from set of options based on validation set performance, similar to the approach used for prompt chains.
Method
AP50
AP75
AP
GPT-4o (Direct Regression)
17.69
1.69
5.08
Gemini 2.0 Flash (Direct Regression)
38.77
10.80
15.66
Gemini 1.5 Pro (Direct Regression)
55.11
31.23
31.33
Claude 3.5 Sonnet (Direct Regression)
17.97
2.13
6.03
Qwen2-VL (Direct Regression)
44.10
23.71
24.36
GPT-4o (Regression with Ruler)
13.95
2.60
4.99
Table 10: Rulers for Object Detection: The results indicate that visual markers such as rulers are ineffective in aiding GPT-4o for bounding box regression. Numbers obtained are on subset 100 COCO Images.
Visual Prompt
AP50
AP?5
AP
Ruler
21.19
4.09
7.60
Ruler 2
22.59
7.85
9.20
Ruler 3
19.06
4.86
8.09
Point marker
Rectangle marker
Curve marker
Figure 16: The curve, rectangle, and point marker types we ablated for the segmentation task.
Reference RGB
All removed
Top removed
Context removed
Full Pyramid
Figure 17: Semantic Segmentation predictions with different layers of the semantic pyramid. From left to right: 1. The RGB Image. 2. The predicted mask when no crops are given, and markings on the full image are directly used. The model is unable to make out fine details. 3. The predicted mask when the top of the semantic pyramid is removed. The model misses out on predicting some finer details (for instance, the gaps in the bench and the handbag). 4. The predicted mask when the middle layer (the context) is removed. The model makes some wrong predictions. 5. The mask with full pyramid of information.
28

---

Published as a conference paper at ICLR 2026
Number markers
Rectangle markers
Curve markers
Figure 18: The marker styles used for directly querying semantic entities from the full image.
Table 11: Ablation study on semantic segmentation. The results show that GPT-4o is robust to the choice of visual prompt. The substantial performance drop (16 mloU) observed upon removal of the semantic pyramid shows the critical role of the contextual information used in the sub-task.
Category
Ablation
mloU
Pixel Accuracy
Curve
41.62
67.43
Visual Prompts
Rectangle
41.67
69.74
Point
41.68
68.83
Without Crop
40.97
69.84
Contextual Ablations
Without Context
31.25
61,66
Best Direct
25.79
55.42
numbers highlight the importance of contextual information within the semantic pyramid. Removing the context layer leads to a performance drop of over 10 mloU. Additionally, the direct strategy of marking directly the image and then classifying results in 16 mloU difference, indicating that MFMs currently lack the ability to localize precisely. We also investigate the impact of omitting the finest level of the semantic pyramid-the crop. While the mloU value does not decrease much, qualitative analysis reveals that this omission hampers the model's ability to capture finer image details. This is shown in Fig. 17
We also conduct ablation studies on the effect of the model's performance when the semantic pyramid omitted. The visual markers in Fig. do work well and do not allow batching,
borrow visual marker similar to the one used in Yang et al. (2023a) (see Fig. T8. Table 12 shows the results for different marker types (Yang al. (023a). It clear that the model's performance greatly drops when it is deprived the crops. We note that the marks we use differ from the ones used Yang al. a023a) in two ways:
The marks obtained in Yang et al. (2023a) already correspond to semantic entities, while we use superpixels as proxy for this.
Extracting a full semantic mask requires discerning finer-grained details, so the marks we use typically correspond smaller regions in the image.
E.3 GROUPING
For the grouping task, filter out 100 COCO images that contain instances which well-posed for this task. The well-posedness of an instance for grouping is measured by how consistent the SAM predictions are for the instance. To calculate the consistency of predictions for instance, we sample random points inside the instance and use SAM to obtain an instance mask for each point individually, well global mask by querying all points together. The mloU between individual masks and
29

---

Published as a conference paper at ICLR 2026
Table 12: Ablation on Direct Segmentation: The numbers clearly show that omitting the extra information provided by the crops greatly impacts the model's performance. The numbers shown for subset of 30 images. The prompt selected from set of options based validation performance.
Visual Marker
mloU
Pixel Accuracy
Curve
20.70
50.34
Rectangle
18.24
47.80
Number
21.13
50.00
Figure 19: Ambiguous instances: If cat's car marked, is the object the cat or the cat's car? Images on the left: Grouping without explicit reference to "objectness". Images the right: Grouping obtained using the "apostrophe-s" test.
the global mask is used as the consistency metric. Finally, the images that contain instances with consistency value above given threshold are selected and randomly sampled to create the evaluation set.
E.3.1 WHAT CONSTITUTES AN OBIECT?
Determining the granularity of what qualifies as object in grouping task is often ambiguous. For instance, if a person's nose is highlighted, should the object be considered the nose alone, or the entire person? Both interpretations are valid, leading potential inconsistencies.
30

---

Published as a conference paper at ICLR 2026
Table 13: Ablation study on depth prediction. GPT-4o performs the best when curves are used as the visual marker.
Higher hetter 1
Lower is better !
Method
61
82
03
Accuracy
AbsRel
Curve
0.550
0.822
0.935
53.75
70,43
0.332
Rectangle
0.534
0.807
0.931
51.68
69.28
0.341
Point 0.525
0.802
0.928
51.89
62.07
0.366
Table 14: Oracle depth results with different numbers of superpixels and comparisons made during chaining.
Higher is better f
Lower is better |
Superpixels
Samples
81
82
83
p
AbsRel
100
200
0.571
0,774
0.863
0.83
0,528
100
400
0.597
0.785
0,867
0.86
0,514
200
200
0.571
0.773
0.867
0.83
0,501
200
400
0.593
0.788
0.869
0.86
0.502
To address this, we propose prompting method that refines the granularity of "objectness." By instructing the model to interpret the highlighted instance possessive noun-expressed through the "apostrophe-s" structure-tth the model is encouraged group coarser objects. For example, when prompted with "person's 's nose," the model is guided to interpret the object as the person, rather than the nose alone. This approach is illustrated in Fig. 19
While this method is not universally effective, it often resolves ambiguity by clarifying the relationship between parts and wholes. We provide the full prompt in the supplementary material.
E.4 DEPTII PREDICTION
We conduct an ablation study on the choice of visual markers in Tab. 13 additional oracle evaluations.
Please also see Tab. f4r
E.5 SURFACE NORMAL PREDICTION
We conduct ablation study the choice of visual markers in Tab. 15
E.6
EXPERIMENTS WITH LLAMA
Unlike the other models, Llama employs different prompt chains for object detection, grouping and segmentation due to its current limitations with handling multiple images (Huggingface).. Specifically:
For object detection, we provide the full image with the corresponding grid cell marked instead of providing crop of the grid cell with the full image.
For semantic segmentation, we provide the full image with the corresponding superpixel marked, instead of providing a set of crops per superpixel.
Table 15: Ablation study on surface normal prediction. GPT-4o is relatively robust to different visual marker choices.
Method
Pr.
Py
Pz
Curve
-4.99
58.00
39.28
Rectangle
-13999
58.84
39,65
Point
2.42
51.26
39.59
31

---

Published as a conference paper at ICLR 2026
Detection
Semantic Segmentation
Depth
Surface normal (x-axis)
Surface normal (y-axis)
Surface normal (z-axis)
Figure 20: The blind guesses made by GPT-4o on different tasks.
For grouping, we highlight the superpixel corresponding the initial cluster in red and the superpixel of the query point in blue. The model is tasked with determining whether the blue region belongs the same entity as the red region.
As with the other models, experimented with multiple prompts and selected the best best-performing one based on a smaller validation set across all tasks.
Surprisingly, Llama does well in segmentation despite not being provided with any crops. A comparative evaluation against other MFMs on smaller subset using identical prompts is shown in Tab. 16 Notably, Llama surpasses all other models in this setup.
additional interesting finding is Llama's unique capability to achieve a positive correlation in the a-direction for the surface normals task, result not observed with the other non-reasoning MFMs.
Table 16:
The performance of Llama compared with the other models on smaller subset, in the absence of crops.
Model
mIoU
Pixel Accuracy
GPT-4o
19.77
54.31
Gemini 1.5 Pro
22.98
61.04
MFMs
Claude 3.5 Sonnet
20.00
55.69
Llama-3.2-90B
25.86
61.66
E.7 BLIND GUESS
As mentioned in Section 4 useful way to analyze the potential biases of the MFM, and to gauge the degree to which uses the visual content blind guess, or prompting the image with blank image. In particular:
For object detection, we ask the model to imagine the classes present. After this, we ask provide reasonable coordinates for the objects based on its world knowledge.
For semantic segmentation, we mark a rectangle in white image and force the model to predict class. We ask the model to use the location to make an educated guess.
For depth, ask the model imagine force the model to predict that one is at a greater depth than the other.
indoor setting. We mark two rectangles and
For normals, we repeat the procedure for depth for each direction.
The results for GPT-4o visualized in Fig. 20 and reveal several interesting insights.
32

---

Published as a conference paper at ICLR 2026
Object Detection
Segmentation
90
80
mlou
70
60
60
1/8
1/12
1/16
400
600
800
1000
Outer Grid-Cell Thickness
Number of Superpixels
Depth
Normals
0.92
0.725
0.700
0.88
0.675
0.86
0.650
0.84
0.625
100/200
200/400
400/800
100/200
200/400
400/800
Superpixels Comparisons
Superpixels Comparisons
Specialist Chain
Performance improvements the "Oracle Chain" and "Specialist Chain" baselines on Figure 21:
full datasets when using a finer- finer-grained prompt chain. Ten iterations were used for object detection.
For object detection, the model chooses common classes like person car. Additionally, it seems to grasp the relative sizes of objects reasonably well, indicated by its tendency to make the car and the bench longer.
For semantic segmentation, the model makes reasonable guesses. For instance, it guesses "sky-merged" and "airplane" at the top of the image, "person" near the middle, "dog," "cat," and "floor" near the bottom.
For depth prediction, GPT-4o exhibits "ceiling bias" and consistently infers that the top right corner located at a greater relative depth. We observe that this bias is reflected in several of the model's predictions well, where the ceiling consistently assumed to be at a greater depth.
For surface normals, the model uses the relative locations of the rectangles to form judgments. For instance, in the direction, it infers that the right rectangle aligns more towards the right. the y direction, it consistently infers that the bounding box a greater y coordinate aligns more with the positive direction. While Chain-of-Thhugee (CoT) reasoning is able to break this bias along the direction for GPT-4o, the left-right bias persists when actual images are presented.
E.8 FINER-GRAINED PROMPT CHAIN
natural question is whether the performance of MFMs be further enhanced by refining the granularity the prompt chain. In other words, can we improve performance by increasing the number of superpixels and comparisons first examine the effect of a finer-grained prompt chain on the "Oracle + Chain" and "Specialist + Chain" baselines. As shown in Fig. 21 these baselines exhibit steady performance improvements as the prompt chain is refined.
by using thinner outer grid cells? To explore this,
To determine whether this trend extends the MFMs, conducted small-scale experiment using GPT-40 on the same tasks. As illustrated in Fig. 22 although GPT-4o shows modest improvements with finer-grained prompt chain, its performance quickly saturates due to misclassifications. This
33

---

Published as a conference paper at ICLR 2026
Object Detection
Segmentation
80
80
70
70-
mlou
60
50
40
50
30
1/8
1/12
1/16
400
500
600
800
Outer Grid-Cell Thickness
Number Superpixels
Depth
Normas
0.75
0.9
0.70
0.65
0.8
0.60
0.55
0.7
0.50
100/200
200/400
400/800
100/200
200/400
400/800
Superpixels Comparisons
Superpixels Comparisons
Oracle Chain
Specialist Chain
4o
Figure 22:
GPT-4o's performance improvements with finer-grained prompt chain plateau, showing that our original settings are adequate.
observation supports our decision to adopt a coarser granularity for the MFMs, confirming that our original settings are sufficient capture the performance gap.
We reiterate here that the control baselines in our paper serve as calibration tools rather than perfor- mance ceilings. They mitigate the sub-optimality of the prompting method and prevent exaggerated conclusions about MFMs significantly lagging behind specialists. Our benchmark is designed to be accurate in a relative and ordinal sense.
E.9 PERFORMANCE CEILING FOR GEOMETRIC TASKS
To investigate whether the pipeline imposes a performance ceiling for the geometric tasks, we evaluated the depth and surface normal algorithms using increasingly fine-grained superpixelations. We varied the number superpixels from 100 to 2000. To ensure sufficient number pairwise comparisons for the optimization, we set the number of pairwise comparisons Npeirs 32 N.
As shown in Tab. 17 increasing the granularity consistently improves the performance of the control baselines. Notably, the correlation metrics converge towards the "Dense" values (representing the performance obtained using raw pixel predictions without superpixel discretization). This confirms that the pipeline is capable of high-fidelity geometric recovery given sufficient granularity. We report correlation (p) as our algorithm reconstructs geometry purely from ordinal rankings.
We provide qualitative comparisons between the Specialist Chain and Oracle + Chain reconstruc- tions in Figures 23 and 24 The visuals are in agreement with the quantitative results: fine- nre-grained geometric details progressively recovered the number of superpixels and comparisons increases, demonstrating that the prompt chaining mechanism itself does not inherently prevent high-fidelity estimation.
E.10
IN-THE-WILD EVALUATIONS
Please see Fig.
27
28 and 29 for qualitative evaluation of MFMs on in-the-wild samples (Flickr
2024
Unsplash
202
34

---

Published as a conference paper at ICLR 2026
Figure 23: Increasing superpixel granularity in the depth prompt chain leads to the recovery of fine-grained features.
Figure 24: Similar to depth, increasing superpixel granularity in the surface normals prompt chain leads to a sharper normal map.
35

---

Published as a conference paper at ICLR 2026
(a) Depth
Metric: P (1)
Number of Superpixels
Baseline
100
200
400
1000
2000
Dense
Oracle Chain
0.938
0.958
0.964
0.978
0.982
1,000
Specialist + Chain
0.924
0.943
0.952
0.958
0.962
0.969
(b) Surface Normals
Metric: P= (1)
Number of Superpixels
Baseline
100
200
400
1000
2000
Dense
Oracle + Chain
0.743
0.774
0.802
0.831
0.848
1.0
Specialist + Chain
0.713
0.735
0.751
0.762
0.769
0.805
Table 17: Effect of superpixel granularity on the Depth and Surface Normal Prediction baselines. We observe that the correlations approach the dense (raw pixel) performance ceiling.
F REASONING RESULTS
F.1 ABLATING THE BATCH SIZE
As noted in Sec. 04-mini especially sensitive to the batch size used during inference. While our main evaluations used a batch size of 100 for consistency across models, we ablate this choice on a subset of 500 ImageNet samples Fig. 25
75
Accuracy (%)
70
65
60
55
50
20
40
60
80
100
Batch Size
40
03
04-mini-medium
ol O
04-mini-low
04-mini-high
Figure 25: Effect of batch size on classification accuracy. We ablate the classification performance of ol, 03, 04-mini and GPT-4o on subset of 500 ImageNet images under varying batch sizes. While ol and benefit from larger batch sizes, 04-mini shows a pronounced drop in accuracy.
As shown, o4-mini suffers a substantial degradation in classification accuracy as the batch size increases, across all reasoning effort settings. In contrast and surprisingly, both ol and o3 demonstrate improved performance with larger batch sizes, consistent with trends observed in prior work on in-context learning for L.MSs (Chen et al. 2023 Jiang al. 2024b).
36

---

Published as a conference paper at ICLR 2026
F.2 DETAILED EXPERIMENTS
As noted in the main paper, we evaluate ol and o3 on smaller, representative subsets of data. To construct informative subsets for classification, object detection, segmentation, and grouping. compute the Kendall rank correlation between the performance rankings of non-reasoning models candidate subsets and their full-dataset rankings, we choose the smallest subset size that meets task-specific correlation threshold, yielding 1,000 samples for classification, 200 for detection, 50 for segmentation, and 30 for grouping. These subsets are deemed informative because they preserve the relative ranking of non-reasoning models.
multiple bootstrap
For each task,
For depth and surface normal prediction, select the 10 most challenging samples for GPT-40 (those where it achieves the lowest Spearman correlation) due to higher evaluation costs.
Table 18: Classification. Accuracy scores on the standard classification datasets. o3 consistently outperforms GPT-4o, and achieves the highest overall scores. contrast, 04-mini lags behind.
Model
ImageNet
ImageNet-Vi V2
2DCC
3DCC
ImageNet-R
ImageNet-Sketch
04-mini (low)
50.7
04-mini (medium)
58.4
49.2
35.6
36.4
58.2
46.4
04-mini (high)
59.6
ol
77.90
70,50
62,40
60.20
85.3
65.50
o3
78.00
74.10
62.60
61.70
87.2
69.10
GPT-4o
76.70
72.10
63.80
61.40
86.2
66.10
Table 19: Object Detection. All models other than 04-mini achieve comparable performance, with ol and o3 slightly outperforming GPT-4o.
Table 20: Segmentation. ol and o3 outperform GPT-4o both mloU and pixel accuracy. 04- mini underperforms across all reasoning levels.
Model
AP (f)
AP7s (1)
AP (f)
Model
mloU (f)
Pixel acc. (1)
04-mini (low)
48.40
29.71
27.65
04-mini (low)
29.67
63.94
o4-mini (medium)
47.56
29.77
27.11
04-mini (medium)
28.86
63.82
o4-mini (high)
48.10
26.49
26.07
04-mini (high)
28.66
62.81
ol o3
66.70
40.52
37.77
ol
40.24
72.92
64.89
40.73
38.57
o3
38.18
72.75
GPT-4o
64.11
42.61
38.17
GPT-4o
35.66
69.59
Table 21: Depth Prediction. All models struggle with this difficult subset. However, o3 consistently outperforms the others across most metrics.
Model
81(1)
82(1)
8s(t)
p(t)
Accuracy(1)
AbsRel(()
04-mini (low)
0.467
0.726
0.857
-0.009
55.15
1.07
04-mini (medium)
0.465
0.711
0.852
0.070
56.85
0.953
04-mini (high)
0.462
0.724
0.864
-0.029
53.65
1.090
ol
0.456
0.716
0.873
0.079
57.20
0.914
o3
0.490
0.738
0.855
0.250
63.50
0.819
GPT-4o
0.460
0.732
0.881
-0.050
52.00
1.121
The performance of ol, o3, GPT-4o, and o4-mini, under varying levels of reasoning effort, is summarized in [8b covering classification, object detection, segmentation, grouping. depth, and surface normal prediction.
On semantic tasks, all models perform comparably, with ol and o3 showing slightly stronger results in classification, object detection, and segmentation. For geometric tasks, performance drops across the board due to the difficulty of the selected samples. However, the reasoning models consistently outperform GPT-4o. In particular, all reasoning models achieve positive correlation along the horizontal axis in surface normal prediction, correcting a common failure mode in GPT-40 (see Fig, where the horizontal gradient is flipped). Qualitative comparisons for all tasks shown in Fig. 8
37

---

Published as a conference paper at ICLR 2026
Table 22: Grouping. GPT-4o outper- forms all reasoning-based models on this semantic task. Among the rea- soning models, 03 performs best.
Table 23: Surface Normals. Reasoning models out- perform GPT-4o, particularly along the horizontal (æ) direction, where GPT-4o shows strong negative corre- lation. 03 achieves the highest scores across all axes.
Model
mI)U(1)
Model
Pr
Py
P=
04-mini (low)
47.41
04-mini (low)
0.18
0.12
0.30
04-mini (medium)
44.59
04-mini (medium)
0.39
0.24
0.32
04-mini (high)
46.36
04-mini (high)
0.38
0.23
0.31
ol
52.61
ol
0.24
0.23
0.19
o3
55.06
03
0.48
0.36
0.28
GPT-4o
59.64
GPT-4o
-0.30
0.09
0.05
G ADDITIONAL EXPERIMENTS WITH 4o IMAGE GENERATION
G.1 PROMPTING METHODOLOGY
As discussed in the main paper, GPT-4o generates full image recreations rather than edits, often resulting in spatial misalignments. To enable consistent comparisons, we first zero-pad all input images square dimensions and, after generation, crop the relevant regions to align with the original input. Below, we outline task-speeific prompting details. Preliminary quantitative results for grouping. depth, and surface normals, evaluated at full scale, are provided in App. G.2
The prompt used for abject-based segmentation is shown in Listing The model given an image with a red point marking a location on an object and is instructed to return the same image with the entire object filled solid red. To extract the predicted mask, we use HSV thresholding and post- process it by retaining only the largest connected component, which effectively removes small artifacts and hallucinated regions. Although this simple postprocessing often yields reasonable masks, more advanced postprocessing techniques could improve results further. We leave such refinements future work.
Grouping.
38

---

Published as a conference paper at ICLR 2026
You are given an input RGB image where a small red circle marks point on an object.
Your task is to return the exxcct same image*., but with the *entire object that contains the marked point filled in solid red*.. Do not add any other markings, text, overlays; only apply the red fil1 the object.
Listing Grouping prompt for GPT-4o image generation
Depth. The prompt used for depth prediction is provided in Listing 2 The model is instructed to produce a grayscale rendering the imput image, where closer regions appear darker and farther regions lighter.
Generate a **pure grayscale depth nap** from the input image. The grayscale values must encode depth as follows:
...hite (255 intensity) .. represents points that are *cclosest** the camera (minimum depth).
**Black (0 intensity) represents points that are **farthest** from the camera (maximum depth).
All other points must be shaded imonotonically between black and white **, based solely their distance from the camera.
This map must not contain any colors, textures, or artistic effects: only smooth grayscale transitions that accurately reflect increasing deptn, with darker shades at greater distances.
Listing 2: Depth prediction prompt for GPT-4o image generation
Surface Normals. The prompt used for surface normal prediction is shown in Listing 3 The model asked to generate surface normal map in the standard RGB encoding used in computer graphics. Conventionally. red corresponds to the left-right (æ) axis (left= 0, right = I), green to the up-down (y) axis (down up 1), and blue the depth (z) axis (inward O, outward = 1).
You are a vision model that, given an input RGB image, must predict per -pixel surface normal map and render it as RGB image using the standard normal-map color scheme.
Requirements:
1. **Output format**
rirtly generate an image.
Procuce raw image (same dimensiens input) whose pixel colors
encode the normals above.
Do **oo*** add any annotations, text overlays, or alpha channels: only the RG2 channels.
2. **Normal-map encoding**
For each pixel estimate its surface normal vector
Display the orientation of the surface normal vector using the standard color scheme used in computer graphics.
Listing 3: Normals prediction prompt for GPT-4o image generation
G.1.1 PROMPT SENSITIVITY
We observe that GPT-4o's image generation is highly sensitive to changes in the prompt. To illustrate this brittleness, test two modified prompts. For depth prediction, invert the color mapping in Listing 2 asking the model to render near points as dark and far points as white. For surface normals, rather than producing single RGB normal map, images, each representing alignment with one of the x, y. or axes, similar to our prompt-chaining setup. As shown in Tab. 24 these small changes lead substantial degradation in performance.
prompt the model for three separate grayscale
39

---

Published as a conference paper at ICLR 2026
Table 24: Performance for different prompts. Comparison between the original and an altered prompt for both depth prediction and surface normal prediction.
(a) Depth prediction
Prompt 81 (1)
82 (1)
63 (1)
p (1)
AbsRel (1)
Original
0.562
0.849
0.942
0.448
0.303
Altered
0.399
0.694
0.854
-0.12
0.474
(b) Surface normal prediction
Prompt Px.
Py
Pz
Original
0.31
0.35
0.14
Altered
-0.15
0.38
-0,23
G.2 PRELIMINARY QUANTITATIVE ANALYSIS
The full-scale quantitative results shown in Tab. 25
While the performance is non-trivial, it currently falls short of what is achieved by GPT-4o using the prompt chain. We view the refinement of prompts and decoding strategies for image generation an important direction for future work.
Notably, grouping predictions are impacted by spatial misalignment, hallucinated regions, and incorrect markings. Depth predictions, the other hand, occasionally suffer from an inverted rendering of the depth gradient, which significantly affects correlation praeeedtio---aedd metrics. We showcase representative qualitative results for grouping, depth, and surface normals in Fig. 9 highlighting both successes and common failure modes.
Table 25: GPT-4o image generation performance across three tasks.
Task
Metric
Value (f)
Grouping
mloU
28.
Depth Prediction
81
0.485
82
0.735
83
0,848
0.52
AbsRel (4)
0.575
Surface Normal Prediction
Px
0,09
Pg
0.44
0.17
H PROMPT SENSITIVITY ANALYSIS
In Fig. 26 we evaluate the non-reasoning models for each task considering different prompting techniques. We observe that GPT-4o generally shows lower sensitivity different prompts most of the tasks compared to other MFMs. For surface normals, we interestingly observe that the predictions greatly improve in the y and directions, when GPT-4o and Claude asked to reason in the prompt (see Tab. 26)
I PROMPTING COSTS
At the time of writing, the API pricing for GPT-4o (gpt--0--2-022-00--005 was $2.50 per million input tokens and $10.00 per million output tokens. For Gemini 1.5 Pro (gemini-1. serpiiii-..-pp---- 5-pro-001), the corresponding rates were $3.50 (input) and $10.50 (output), and for Claude 3.5 Sonnet $3.00 and $15.00, respectively. In contrast, lower-cost mod-
aude-3-55 (c1ende------omnee-- S--sonnett222406222 els like Gemini 2.0 Flash (gemi -- and o4-mini (o4-min1-2225-04--
40

---

Published as a conference paper at ICLR 2026
Table 26: Prompt sensitivity for surface normal prediction. Correlations under five different
prompts for GPT-4o and Claude 3.5 Sonnet. CoT prompting (in bold) greatly improves Py and Ps.
Model
Prompt
Px. (t)
Py (f)
Pz (f)
Prompt
-0.36
-0.63
0,04
Prompt
-0.29
-0.45
-0.10
GPT-4o
Prompt 3
0.07
0.55
0.44
Prompt
-0.08
0.65
0.43
Prompt -0.34
0.45
0.37
Prompt -0.34
-0.56
0.00
Prompt -0.18
-0.08
-0.11
Claude 3.5 Sonnet
Prompt -0.09
0.68
0.41
Prompt -0.06
0.66
0.35
Prompt -0.06
0.56
0.35
were priced at 10/00/00.40 and $1.10/S4.40, respectively. The reasoning models o 22024-12-1 and o3-2025-044-11 were substantially more expensive at $15.00/$60.00 and $10.00/540.00, respectively.
The costs for the scaled-up experiments are documented in Tab. 27 and the prompting costs for the reasoning model are presented in Tab. 28 The primary reason for cost fluctuations across tasks the way each MFM tokenizes images. The notably higher costs for object detection with Gemini 1.5 Pro stem from independent calls required the prompt chain. For reasoning models, especially on the surface normal task, major contributor is the large number of generated reasoning tokens. The availability of highly affordable models like Gemini 2.0 Flash, for which our entire evaluation cost approximately $50, demonstrates that such benchmarking is becoming increasingly accessible.
These costs reflect the constraints current APIs and are not indicative how such tasks would be solved in practical deployments. As discussed in the main paper, our framework is intended for standardized one-time evaluation (and for efficient task execution) with MFMs.
Table 27: Prompting costs for scaled-up experiments (in S).
Task
GPT-4o
Gemini 1.5 Pro
Claude 3.5 Sonnet
Gemini 2.0 Flash
Classification
223.8
298.6
142.8
9.7
Object Detection
185.8
610.8
155.0
18.1
Semantic Segmentation
232.1
450.1
227.9
14.0
Grouping
22.1
47.4
42.0
1.0
Depth
57.4
52.4
198.2
3.6
Normals
130.1
50.1
209.9
3.9
41

---

Published as a conference paper at ICLR 2026
Classification
Object Detection
Accuracy
65
14
25
OFT-4)
Gerva 1.5 Coude 3-5 Socner Gwes7-st 728
Off-dt
Cenera Daste 35 Sereet Qeent-V 778
Models
Models
Depth Prediction
Surface Normal Prediction
Rank Correlation Coefficintt
0.42
0.8
0.44
-00
-64
Models
Models
Segmentation
30
mloU
24
Models
Figure 26: Sensitivity of MFMs to different prompting techniques. We observe that GPT-4o showcases lower sensitivity on most tasks compared to other MFMs.
Table 28:
Prompting costs for the reasoning models (in $) 'Reported costs are for experiments conducted on the subset.
Task
ol*
03*
04-mini
Classification
41.0
22.5
50.0
Object Detection
220.0
104.0
102.2
Semantic Segmentation
200.0
96.0
115.0
Grouping
82.2
48.9
25.0
Depth
96.4
31.5
62.0
Normals
306.2
85.4
194.0
42

---

Published as a conference paper at ICLR 2026
legud Image
40-21
OP7-0
Denent Pro
Claske Seeret
teag?
60-21
OP7-40
Oened
Claake &d Sceret
Figure 27: Qualitative results of evaluating MFMs for object detection on in-the-wild examples (Fickr
2024 Unsplash 2024). We compare against 4M-21 (Bachmann al. 2024) as vision specialist.
43

---

Published as a conference paper at ICLR 2026
49-01
OPT-0
Clade Seenet
60-21
OPT-40
Oured 1-1
Clasko
Figure 28: Qualitative results of evaluating MFMs for semantic segmentation on in-the-wild exam-
ples (Flickr) 2024 Unsplash] 2024). We compare against 4M-21 (Bachmann et al.) 2024) a vision specialist.
44

---

Published as a conference paper at ICLR 2026
Wallen
OPT-
Genera
Clasdo Beenet
Vasen spec sout
OPT-30
Gasle
Figure 29: Qualitative results of evaluating MFMs for depth prediction on in-the-wild exam- ples (Flickr) 2024 Unsplash] 2024). We compare against the Omnidata (Eftekhar al. 2021 depth estimator as vision specialist.
45


Reviews:
[{"review_id": "1fkS3CuGpF", "reviewer": "Reviewer_J3EM", "summary": "This paper presents a comprehensive benchmark evaluating multimodal foundation models (MFMs) on standard computer vision tasks including classification, object detection, semantic segmentation, grouping, depth estimation, and surface normal prediction. The key contribution is a prompt chaining framework that decomposes complex vision tasks into text-solvable sub-tasks, enabling API-level evaluation of closed-source models. The study evaluates GPT-4o, o4-mini, Gemini 2.0 Flash, Gemini 1.5 Pro, Claude 3.5 Sonnet, Qwen2-VL, and Llama 3.2 on established datasets. Main findings show that: (1) MFMs lag significantly behind specialist models on all tasks, (2) they perform better on semantic tasks than geometric ones, (3) GPT-4o performs best among non-reasoning models, and (4) reasoning models (o1, o3, o4-mini) show promising improvements on geometric tasks.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Extensive Model Coverage: The evaluation includes a diverse set of models (7 main MFMs plus reasoning models), providing a thorough landscape of current multimodal model capabilities across both open and closed-source systems.\n\n- Multiple Task Evaluation: The breadth of tasks evaluated (6 core vision tasks) spanning semantic to geometric understanding provides valuable insights into where MFMs excel and struggle.", "weaknesses": "- Potential Data Contamination: While the authors conduct in-the-wild evaluations to address this, the use of standard benchmarks (ImageNet, COCO) raises concerns about training data leakage for closed-source models, which could inflate performance estimates.\n\n- Limited Analysis of Failure Modes: While the paper shows that MFMs struggle with geometric tasks, there is limited investigation into why they fail (e.g., lack of 3D training data, architectural limitations, reasoning deficits). The \"blurry vision\" hypothesis is mentioned but not systematically explored.\n\n- Task Selection Bias: The choice of tasks favors dense prediction problems that can be decomposed into classification. Other important vision capabilities (e.g., visual reasoning, fine-grained recognition, video understanding) are not evaluated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oBknGdjwDc", "reviewer": "Reviewer_hd9G", "summary": "This paper benchmarks multimodal foundation models for standard CV tasks. They proposed an interesting way to avoid using API calls for promptable computer vision tasks. The whole evaluation tasks covers 2 categories, semantic and geometrical understanding, and there are 6 sub-tasks in total. GPT-4o achieves an overall best performance, but is still lagging far behind CV models. For geometric tasks, under the promtable format, all the MFM failed to work.", "questions": "- How will random superpixelizations/grid seeds affect the performance?\n- Could other CV tasks also be evaluated in the same manner, such as pose estimation, edge detection?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The promptable task design itself is interesting and is effective for semantic understanding tasks. \n- This paper is well written and easy to follow\n- Experimental evaluations are comprehensive.", "weaknesses": "The biggest problem is, the pipeline of the geometric understanding tasks (depth and surface normal) does not work at all for MFMs:\n- In Figure 3, neither depth nor surface normal works as expected, making it not surprising that all the models lagged far behind CV models. \n- The root cause of the mismatch is the ranking algorithm. It is impossible to get the numeric results for depth and surface normal purely from ranks. \n- Without these two geometric tasks, all 4 tasks are basically semantic understanding tasks, with the performance of all the compared models highly correlated. \n\nOther minor weaknesses: \n- The observations did not provide key findings, unless the ranking. Some interesting points may be: why the reasoning models work worse on semantic understanding? \n- The metrics for surface normal and depth are not the most popular ones. \n- Results with GPT-4o image generation is confusing and did not fit the whole story.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xYrHX0GXof", "reviewer": "Reviewer_9qaY", "summary": "This paper presents a comprehensive benchmark evaluating the visual understanding capabilities of state-of-the-art multimodal foundation models (MFMs), including GPT-4o, Gemini 1.5/2.0, Claude 3.5 Sonnet, Qwen2-VL, and Llama 3.2, on a suite of standard computer vision tasks: classification, object detection, semantic segmentation, grouping, depth estimation, and surface normal prediction. Recognizing the challenge that most MFMs are text-based and do not natively output dense or structured visual predictions, the authors introduce a systematic “prompt chaining” framework that decomposes vision tasks into API-compatible sub-tasks. The results demonstrate that while MFMs are decent generalists—especially on semantic tasks like classification and segmentation—they lag substantially behind state-of-the-art specialist vision models, and struggle most with geometric (3D) tasks. The paper includes both quantitative and qualitative analyses, explores prompt sensitivity, and openly acknowledges the limitations of the approach.", "questions": "Please see weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**1) Thorough and Timely Benchmarking:** The paper provides one of the most exhaustive and systematic evaluations to date of leading MFMs' visual understanding, moving well beyond the customary VQA or captioning settings to address a broader spectrum of fundamental computer vision tasks.\n\n**2) Task Translation via Prompt Chaining:** The “prompt chaining” framework is thoughtfully designed, enabling apples-to-apples comparisons between text-based MFMs and vision specialists through structured sub-task decomposition. Figure 2 effectively visualizes the design for converting complex tasks like depth and segmentation into textual queries, making the framework transparent and reproducible.\n\n**3) Comprehensive Model and Task Coverage:** The analysis involves a diverse set of MFMs (both closed- and open-weight) across six core vision tasks and multiple datasets, situating the current state of MFMs within established benchmarks (COCO, ImageNet, Hypersim, etc.), including corruption and robustness variants. Table 1 and Table 3–6 substantiate this comprehensiveness.\n\n**4) Control Baselines and Calibration:** The empirical results are carefully calibrated with various baselines—(a) top specialist models, (b) specialists subjected to the same chaining and superpixel constraints, (c) oracle variants, and (d) blind guess. This tightens the attribution of observed deficits to either model limitations or task translation artifacts.", "weaknesses": "**1) Prompt Chaining Overhead and Realism:** The proposed evaluation relies on decomposing tasks into a large number of textual API calls, which is computationally expensive (as noted in App. I). While the paper claims this is for benchmarking only, the translation introduces additional sources of potential error (task granularity, superpixel boundaries, etc.), and the extent to which these reflect real “model” limitations is not exhaustively disentangled. For instance, in Figure 2 and related descriptions, there is acknowledgment that chaining is not optimal—yet the effect of granularity and design choices is mostly probed by coarse ablations, potentially underestimating the ceiling performance of some MFMs.\n\n**2) Limited Exploration of Advanced Prompting or Visual Tools:** There is a missed opportunity to rigorously evaluate more advanced or recently proposed visual prompt engineering techniques (e.g., interactive alignment, visual rulers, or interactive markers for geometric tasks, as briefly mentioned in App. E.1, Figure 15) across all models as a systematic solution for object localization or dense prediction. The decision to use superpixel-based batch querying is pragmatic but may limit the apparent ability of models on fine-structured tasks.\n\n**3) Insufficient Error Analysis on Geometric Tasks:** While the paper establishes that MFMs fare significantly worse on geometric tasks (Tables 5 and 6; Figure 3), there is minimal deep dive into why—for example, what kinds of 3D/normal ambiguities (left-right, scale, out-of-plane rotation) are most problematic, or whether cues are absent due to model design, pretraining data, or prompt interface. Figure 3 and Table 6 show low/negative correlations for some directions, but the causal factors or failure modes are not dissected in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NdZHhBZdD2", "reviewer": "Reviewer_GMJw", "summary": "This paper benchmarks VLMs on standard computer vision tasks (semantic segmentation, object detection, image classification, depth and surface normal prediction) using established datasets. It developed a novel method to translate vision tasks into text-promptable, API-compatible formats via prompt chaining. It draws a few interesting findings: VLMs are not close to the state-of-the-art specialist\nmodels at any tasks; GPT-4o performs the best among non-reasoning models, securing the top position in 4 out of 6 tasks.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors develop a very clever way to prompt the VLMs to perform dense prediction tasks by using superpixels. They also find a better way to prompt object detection through iterative cropping and splitting the image into grids. This new technique creates the new possibility of systematically benchmarking the spatial and semantical understanding capability of VLMs, and the results are interesting.\n2. The paper is well-written. The experiments are extensive, covering 6 fundamental vision tasks and the state-of-the-art VLMs, both closed and open-sourced ones.", "weaknesses": "1. Some of the specialist models are outdated. This could make the readers think that the gaps between VLMs and SOTA models are smaller than they actually are. Examples: the authors should compare with SAM2 in semantic segmentation (Tab 4); compare with Lotus [1] and Moge2 [2] for depth and normal map estimation.\n2. For depth and normal map estimation, it is not clear how many random pairs are needed for a good convergence of the optimization algorithm. The results of depth and normal in Fig 3 also look very coarse, probably due to the granularity of the superpixels. It would be interesting to see if higher quality depth/normal map will emerge if we use more superpixels and/or more pairs.\n\n[1] Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction, ICLR 2025\n\n[2] MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details, Arxiv 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)