AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models

## Abstract

Abstract The rise of vision foundation models (VFMs) calls for systematic evaluation. A common approach pairs VFMs with large language models (LLMs) as general-purpose heads, followed by evaluation on broad Visual Question Answering (VQA) benchmarks. However, this protocol has two key blind spots: (i) the instruction tuning data may not align with VQA test distributions, meaning a wrong prediction can stem from such data mismatch rather than a VFM’ visual shortcomings; (ii) VQA benchmarks often require multiple visual abilities, making it hard to tell whether errors stem from lacking all required abilities or just a single critical one.
To address these gaps, we introduce AVA-Bench , the first benchmark that explicitly disentangles 14 Atomic Visual Abilities (AVAs)—foundational skills like localization, depth estimation, and spatial understanding that collectively support complex visual reasoning tasks. By decoupling AVAs and matching training and test distributions within each, AVA-Bench pinpoints exactly where a VFM excels or falters.
Applying AVA-Bench to leading VFMs thus reveals distinctive “ability fingerprints,” turning VFM selection from educated guesswork into principled engineering. Notably, we find that a 0.5B LLM yields similar VFM rankings as a 7B LLM while cutting GPU hours by 8 × \times , enabling more efficient evaluation. By offering a comprehensive and transparent benchmark, we hope AVA-Bench lays the foundation for the next generation of VFMs.

## 1 Introduction

Figure: Figure 2: Visual Question Answering (VQA) often requires multiple atomic visual abilities (AVAs) to answer a question. As such, when a model makes an incorrect prediction, it can be difficult to determine whether it stems from a failure to capture all required AVAs or just a single critical one.
Refer to caption: x1.png

Vision Foundation Models (VFMs), pre-trained on large and diverse datasets, have become central to AI by providing transferable features for a wide range of downstream tasks khan2022transformers; awais2025foundation; chowdhury2025prompt; bommasani2021opportunities. The variety of pre-training objectives and supervision signals has led to a proliferation of specialized VFMs—such as DINOv2 oquab2023dinov2, CLIP radford2021learning, and SAM kirillov2023segment—each excelling in distinct visual capabilities while often exhibiting interesting emergent properties goldblum2023battle; caron2021emerging; naseer2021intriguing.
Consequently, establishing a systematic and effective evaluation protocol for VFMs has become increasingly crucial.

Existing evaluation protocols can generally be categorized into two groups. The first focuses on task-specific capabilities, typically attaching tailored heads to VFMs, followed by fine-tuning and evaluation on dedicated datasets such as ImageNet for classification han2022survey; mai2024fine and COCO for detection or segmentation thisanke2023semantic; balachandran2024eureka. To better capture the diverse and complex perception challenges of the real world, recent studies advocate a more generic approach that leverages large language models (LLMs) as general-purpose heads, evaluating VFMs on broad Visual Question Answering (VQA) benchmarks liu2023visual; zhu2023minigpt; chowdhery2023palm.

While increasingly adopted, this generic protocol suffers from two potential blind spots: (i) discrepancies between the instruction tuning data and test VQA data can cause performance drops due to data mismatch rather than genuine visual limitations in VFMs; and (ii) existing VQA benchmarks typically require multiple visual abilities simultaneously, making it difficult to determine whether a failure arises from the absence of multiple abilities or merely a single critical one.

To address these challenges, we introduce AVA-Bench, the first VFM evaluation benchmark explicitly designed to disentangle Atomic Visual Abilities (AVAs)—the fundamental visual capabilities that combine to solve complex visual reasoning tasks.
For instance, answering typical VQA questions shown in [Figure 2](https://arxiv.org/html/2506.09082v2#S1.F2) necessitates integrating several AVAs.

Specifically, AVA-Bench evaluates VFMs across 14 carefully identified AVAs (see [Figure 3](https://arxiv.org/html/2506.09082v2#S2.F3)), including localization, counting, spatial reasoning, orientation, absolute and relative depth estimation, and recognition of textures, colors, objects, actions, emotions, optical characters (OCR), and scenes.
Each AVA comes with distribution-matched train and test splits and is probed *in isolation*, eliminating the two aforementioned ambiguities.
This enables AVA-Bench to pinpoint exactly where a VFM excels or falters, providing a clear picture of its strengths and weaknesses.

We systematically benchmark leading VFMs trained under diverse objectives and data (see [Figure 1](https://arxiv.org/html/2506.09082v2#S0.F1)), covering language-supervised (e.g., SigLIP-1/2 tschannen2025siglip; zhai2023sigmoid, CLIP radford2021learning, InternVL-2.5 chen2024expanding), multimodal autoregressive (AIMv2 fini2024multimodal), segmentation-supervised (SAM kirillov2023segment), depth-supervised (MiDas ranftl2020towards), contrastive self-supervised (DINOv2 oquab2023dinov2), and agglomerative models (RADIO ranzinger2024radio). We follow the standard protocol of adding an LLM on top, but fine-tune it separately for each AVA.

Our extensive analyses reveal the following main findings: (1) SigLIP-1/2 and AIMv2 emerge as the most versatile VFMs, achieving the highest average rank across all AVAs—*highlighting the critical role of language supervision in enhancing general visual capability*; (2) For vision-centric AVAs such as localization, absolute depth estimation, and orientation, the SSL-based DINOv2 performs comparably or better than language-supervised counterparts; (3) Conversely, language-centric tasks such as OCR strongly favor language-supervised VFMs; (4) Last but not least, we observe that VFMs universally excel at low- to mid-level AVAs (e.g., texture, relative depth estimation, object recognition), regardless of their training objectives—suggesting that VQA failures typically stem from *deficiencies in specific critical AVAs* rather than broad visual incompetence. These insights turn VFM selection from educated guesswork into principled engineering, enabling practitioners to choose (or ensemble) VFMs based on the specific AVA strengths required by their downstream tasks.

Alongside our main study, we identify a more resource-efficient evaluation strategy. Existing LLM-based VFM evaluations typically rely on heavyweight models such as Vicuna-1.5(7B/13B) tong2024cambrian; huang2024survey, aiming for high absolute accuracy but incurring significant computational overhead.
However, when the goal is to compare VFMs, we advocate *prioritizing relative performance over absolute accuracy.*
We demonstrate that a lightweight 0.5B LLM preserves reliable VFM rankings while reducing evaluation costs by 8$\times$, making large-scale analysis substantially more practical. We also contextualize our findings within related work goldblum2023battle; tong2024cambrian to offer a holistic understanding of existing VFMs and highlight directions for future research (see [section 6](https://arxiv.org/html/2506.09082v2#S6)).

Contributions. Our key contributions are three-fold:

- •
We identify critical blind spots in existing evaluation protocols and introduce AVA-Bench, a systematic, diagnostic, and comprehensive VFM evaluation benchmark covering 14 atomic visual abilities (AVAs), clearly highlighting VFMs’ fundamental strengths and weaknesses.
- •
We conduct a detailed evaluation and insightful analysis of diverse leading VFMs, deriving actionable guidance for VFM selection in downstream applications such as customized MLLMs.
- •
We release a resource-efficient evaluation protocol along with an open-source codebase to facilitate the development of the next generation of accountable and versatile VFMs.

## 2 Vision Foundation Models (VFMs) and Evaluation

Figure: Figure 3: AVA-Bench consists of 14 Atomic Visual Abilities (AVAs) that can be combined to address more complex visual reasoning tasks.
Refer to caption: x2.png

### 2.1 Vision Foundation Models (VFMs)

A wide range of architectures and learning strategies have been explored to develop vision foundation models (VFMs). In general, they fall into two categories based on the nature of their training data. On one hand, VFMs are pre-trained purely on visual data. For instance, the Vision Transformer (ViT)  dosovitskiy2020image is trained on labeled images using the supervised signal to capture visual representations. DINOv2  oquab2023dinov2, in contrast, adopts a self-supervised learning approach, enabling the model to learn those features without labeling. Some VFMs are designed for specific vision tasks—for example, SAM kirillov2023segment specializes in open-vocabulary segmentation, while MiDaS ranftl2020towards focuses on monocular depth estimation. RADIO  ranzinger2024radio instead introduces a multi-teacher distillation framework that unifies the strengths of different VFMs (e.g., CLIP, DINOv2, and SAM) into a single efficient student model.

Another family of VFMs leverages image-text pairs. CLIP  radford2021learning uses contrastive learning to align image and textual descriptions, while SigLIP  tschannen2025siglip; zhai2023sigmoid replaces the contrastive loss with a sigmoid one for more efficient training. Unlike the conventional language-guided VFMs that process images and text separately, AIMv2  fini2024multimodal integrates visual and textual understanding into a single auto-regressive framework, which is simple yet effective. Due to their different architectures and learning objectives, VFMs may have varying strengths and limitations in visual understanding.

### 2.2 LLM-based Evaluation for VFMs

Figure: Figure 4: Evaluation pipeline for AVA-Bench: The standard LLaVA-style two-stage training prepares the connector and LLM for VFM evaluation. For each AVA, only connector and LoRA is trained.
Refer to caption: x3.png

Unlike the traditional paradigm, where different perception tasks, such as classification, detection or segmentation, necessitate task-specific models han2022survey; awais2025foundation, the widespread adoption of Large Language Models (LLMs) as versatile interfaces has significantly shifted this paradigm zhang2024mm; wu2023multimodal. Reflecting this shift, recent studies advocate leveraging large language models (LLMs) as general-purpose heads and evaluating VFMs on broad Visual Question Answering (VQA) benchmarks tong2024cambrian. Specifically, following the LLaVA approach, these studies utilize a two-stage framework: (i) pre-training a connector between frozen LLM and VFMs with image-text pairs for feature alignment (ii) fine-tuning both the connector and the LLM using instruction-tuning data, while keeping VFMs frozen during both stages (see [Figure 5](https://arxiv.org/html/2506.09082v2#S2.F5)). This LLM-based evaluation protocol has rapidly gained popularity as it closely mirrors the contemporary multimodal LLM setting and effectively captures diverse real-world perception challenges.

However, this evaluation protocol suffers from two potential blind spots. First, discrepancies may exist between instruction-tuning datasets and the test VQA datasets. Consequently, a miss-prediction might arise from data mismatch rather than genuine visual deficiencies in VFMs. Second, typical VQA benchmarks typically require multiple visual abilities simultaneously to produce correct answers. This multi-ability requirement obscures the exact cause behind a model’s incorrect predictions. For instance, if a VFM is proficient in almost all visual abilities except orientation recognition (forward or backward), it would fail on the question in [Figure 2](https://arxiv.org/html/2506.09082v2#S1.F2) (left). In such scenarios, current evaluation methods only provide relative comparisons between VFMs without elucidating the specific missing visual capabilities. Thus, our AVA-Bench aims to complement the popular LLM-based evaluation by pinpointing exactly where a VFM excels or falters, yielding a holistic understanding of their strengths and shortcomings.

## 3 AVA-Bench

Figure: Figure 6: Performance comparison of VFMs across all AVAs. (Left) Language-Supervised VFMs with DINOv2 as a reference. (Right) Other VFMs with the SigLIP-2 as a reference.
Refer to caption: x4.png

To address the blind spots discussed earlier, we introduce AVA-Bench, the first systematic evaluation suite explicitly designed to disentangle 14 fundamental perceptual skills–Atomic Visual Abilities (AVAs)–for VFMs. We first introduce the concept of AVAs ([subsection 3.1](https://arxiv.org/html/2506.09082v2#S3.SS1)), followed by detailing our dataset curation pipeline ([subsection 3.2](https://arxiv.org/html/2506.09082v2#S3.SS2)). Lastly, we outline the overall dataset statistics and discuss quality control ([subsection 3.3](https://arxiv.org/html/2506.09082v2#S3.SS3)).

### 3.1 A tomic V isual A bilities ( AVAs )

AVAs are fundamental perceptual capabilities that can be combined to address more complex visual reasoning tasks. Rather than treating VQA as a monolithic task, we break it down into 14 fundamental perceptual abilities–such as counting, depth estimation, localization and spatial reasoning–each of which can be composed to answer more complex questions. For example, answering the question, “How many yellow dogs are facing backward on the left-hand side of the stop sign?”, requires multiple AVAs: counting, color recognition, localization, and spatial reasoning (see [Figure 2](https://arxiv.org/html/2506.09082v2#S1.F2)). By explicitly defining the set of AVAs required to interpret a question, our benchmark quantitatively characterizes where a VFM excels or falters across each core ability.

The AVAs selected for AVA-Bench are grounded in a thorough literature analysis, including compositional text-to-images benchmarks huang2023t2i; kil2024mllm; wu2024conceptmix and VQA questions goyal2017making; RealWorldQA; ainslie2023gqa (details in Appendix), focusing strictly on pure perceptual tasks and excluding non-perceptual reasoning skills (e.g., historical context, mathematical reasoning). Examples of each AVA can be found in [Figure 3](https://arxiv.org/html/2506.09082v2#S2.F3) and detailed definitions can be found in Appendix.

### 3.2 Dataset Curation

Constructing AVA-Bench required carefully isolating image–question pairs that specifically test individual AVAs. Existing MLLM and VQA datasets often blend multiple perceptual abilities, complicating direct assessments tong2024eyes; zhang2024mme; yu2023mm. To achieve clear isolation, we assembled a comprehensive suite comprising image–question pairs from 26 diverse datasets, explicitly aligning each pair with a single targeted AVA. These datasets span a broad range of domains—including general scenes, wildlife (e.g., birds, fungi, plants), vehicles, indoor/outdoor settings, and remote-sensing imagery.

To construct the benchmark for an AVA, image–question pairs were carefully designed or adapted to focus solely on that AVA. For instance, the original question “What’s the depth of the car from the camera?” involves both localization and depth estimation. To isolate depth estimation, we explicitly provide the car’s bounding box. This approach enables fine-grained diagnostic insights into VFMs’ AVA-specific strengths and weaknesses. Detailed examples are illustrated in [Figure 3](https://arxiv.org/html/2506.09082v2#S2.F3), with further details specific below and in the Appendix.

#### 3.2.1 Localization

Localization is a crucial foundational AVA, essential for various VQA questions. We curate 34.8K localization-focused image–question pairs sourced from diverse datasets: Objects365 shao2019objects365 and LVIS gupta2019lvis (open-domain), iNaturalist-2021 van2021benchmarking (birds and animals), and DIOR li2020object (remote-sensing). To ensure clarity, we include images with a single instance of the target object and exclude objects with bounding box area smaller than 0.2% of the image.

#### 3.2.2 Counting

We collect 11.3K image-question pairs from five datasets: VQAv2 goyal2017making (open-domain), FSC ranjan2021learning (open-domain counting), CARPK hsieh2017drone (cars), Crowd Surveillance Dataset li2022video (people), and LVIS gupta2019lvis (open-domain). FSC, CARPK, and Crowd datasets explicitly cater to counting tasks, ensuring clear and distinct instance counts. LVIS instance masks are utilized to generate additional counting questions, complemented by counting-focused selections from VQAv2.

#### 3.2.3 Spatial Reasoning

We curate 15.9K pairs from NYU-Depth V2 silberman2012indoor for indoor scenes with object segmentation, and LVIS gupta2019lvis and Objects365 shao2019objects365 for diverse open-domain scenarios. For each image, we select two non-overlapping objects—one marked with a blue bounding box (the reference object) and another with a red bounding box (the target). The model is then asked to determine the relative spatial position of the target object with respect to the reference one, choosing from multiple-choice options: *top-left*, *top-right*, *bottom-left*, and *bottom-right*. We ensure that the two boxes do not overlap either horizontally or vertically to avoid ambiguity.

#### 3.2.4 Visual Attribute

Orientation. We compile 10.1K pairs from two specialized datasets, CURE-OR Temel2018_ICMLA and EgoOrientBench jung2024right, providing uncluttered images of objects from nine distinct orientations: *front*, *back*, *left*, *right*, *top*, *front left*, *front right*, *back left*, and *back right*.

Color. We curate 14K pair sourcing from from open-domain datasets, such as Objects365 shao2019objects365 and LVIS gupta2019lvis. To isolate the color recognition from object semantics or background clutter, we provide bounding boxes that tightly localize regions with minimal color variance, ensuring that the model focuses on RGB prediction.

Texture. We collect 13.2K pairs from four open-domain datasets: DTD cimpoi14describing, Kylberg kylberg2011kylberg, KTH-TIPS kth-tips_dataset and KTH-TIPS2 article with diverse texture types, such as *striped*, *aluminum foil*, and *zigzagged*.

Emotion. We gather 13.9K pairs from two publicly available datasets, RAF-DB li2019reliable and ExpW lian2020expression, covering 7 annotated human emotional states: *surprise*, *neutral*, *disgust*, *fear*, *happy*, *sad*, and *angry*.

#### 3.2.5 Depth Estimation

We curate depth-focused image-question pairs from NYU-Depth V2 silberman2012indoor for indoor scenes and KITTI geiger2013vision for outdoor scenes.

Absolute Depth. We assemble 9K pairs and for each sample, we place a bounding box on a target object and ask the model to estimate its distance from the camera. We ensure that each object class spans at least five distinct depth bins and that the samples within each bin are balanced.

Relative Depth. We collect 11.5K pairs. Each image is annotated with two non-overlapping bounding boxes for two distinct objects. The model is asked to determine which object is closer to the camera. To prevent annotation bias, we ensure that each object class is evenly distributed between being the nearer or farther object to avoid cases where certain objects are always closer.

#### 3.2.6 Recognition

Action. We construct 15K pairs from the Moments in Time monfort2019moments, a video dataset with diverse human actions. We extract the middle frame from each 3-second video for 302 distinct actions

Fine-grained.
We curate 9K pairs from three diverse datasets: CUB-200-2011 wah2011caltech (birds), iNaturalist-2021 van2021benchmarking (fungi, plants, animals), and FGVC-Aircraft maji2013fine (objects). We randomly select 50 species across fungi, plants, and animals from iNaturalist van2021benchmarking and 100 bird species from CUB, resulting in a total of 300 fine-grained classes.

Object.
We curate 34.8K pairs from 4 datasets with diverse domains across 70 unique objects: Objects365 shao2019objects365 and LVIS gupta2019lvis (open-domain), iNaturalist-2021 van2021benchmarking (birds and animals), and DIOR li2020object (remote sensing). For each image, we provide a bounding box to eliminate the need for localization.

Scene.
We curate 13.9K pairs from two datasets: Places434 zhou2017places (open-domain) and AID xia2017aid (remote sensing). The model is required to select the correct scene class from 30 randomly sampled options. The full set includes 464 classes spanning a wide range of scenes.

Figure: Figure 7: The ranks of VFMs across all AVAs. Detailed results in Appendix.
Refer to caption: x5.png

### 3.3 Quality Control and Dataset Statistics

We emphasize rigorous quality control to ensure fair, balanced, and unbiased assessments. Every AVA follows an 80/20 split in which the exact object classes and answer bins that appear in training are mirrored in testing. This ensures that performance differences truly reflect the VFM’s perceptual capabilities rather than train-test distribution mismatches, effectively addressing the concerns highlighted in [subsection 2.2](https://arxiv.org/html/2506.09082v2#S2.SS2). Moreover, we try to avoid potential bias in the benchmark. For example, in counting, we explicitly balance the number of samples per counting bin and per object type in both training and testing sets. This approach avoids biases wherein certain counts (e.g., “7 apples”) might dominate the training data, artificially inflating accuracy for specific numerical predictions. In another example, for localization, we set a clear threshold for minimum bounding-box area to ensure object visibility. More details about how we ensure the quality of AVA-Bench can be found in Appendix.

To further improve the generalizability within each AVA, we aggregate data from multiple diverse datasets. This ensures that models encounter the same core visual ability across varied scenes, object types, and data distributions, making the assessment more robust and less dataset-specific. Lastly, image-question pairs are intentionally crafted or selected to be simple, clear, and explicitly focused on testing only one AVA at a time. In summary, AVA-Bench comprises 218K meticulously curated image-question pairs that robustly isolate individual AVAs, carefully control dataset balance and visibility, and systematically prevent annotation biases. Detailed statistics are provided in [Figure 5](https://arxiv.org/html/2506.09082v2#S2.F5), with additional details in Appendix.

## 4 Evaluation Pipeline of AVA-Bench

To evaluate VFMs using AVA-Bench, we adopt the established LLM-based VFM evaluation protocol, employing the standard LLaVA-style two-stage training procedure to prepare the connector and LLM for VFM evaluation (details in [subsection 2.2](https://arxiv.org/html/2506.09082v2#S2.SS2) and [Figure 5](https://arxiv.org/html/2506.09082v2#S2.F5)). For each AVA in AVA-Bench, we fine-tune the connector and LLM while keeping the VFM frozen. Given the modest size of the training sets per AVA (typically around 6K–10K), we employ Parameter-Efficient Fine-Tuning (PEFT) mai2025lessons; houlsby2019parameter; tu2023visual, specifically Low-Rank Adaptation (LoRA) hu2022lora, to mitigate potential overfitting. Subsequently, the fine-tuned model is evaluated on the corresponding AVA-specific test sets.

### 4.1 Is a Heavyweight LLM Evaluator Necessary?

As discussed in [subsection 2.2](https://arxiv.org/html/2506.09082v2#S2.SS2), traditional LLM-based VFM evaluations, following the LLaVA protocol, predominantly rely on heavyweight models such as Vicuna-1.5 (7B/13B)liu2023visual., aiming for high absolute accuracy but incurring considerable computational costs. Nevertheless, a heavyweight LLM may not be mandatory for reliable comparative evaluations. When the goal is to compare VFMs, we advocate *prioritizing relative performance over absolute metrics.* As depicted in [Figure 8](https://arxiv.org/html/2506.09082v2#S4.F8), a significantly smaller 0.5B LLM (Qwen2) achieves similar relative VFM rankings comparable to a 7B Vicuna-1.5, while dramatically reducing evaluation costs by approximately 8$\times$ (additional details in the Appendix), making large-scale analysis substantially more practical. Thus, we utilize the lightweight 0.5B LLM for all subsequent experiments.

Figure: Figure 8: A much smaller 0.5B LLM achieves similar relative VFM rankings comparable to a 7B LLM, while dramatically reducing evaluation costs by approximately 8$\times$.
Refer to caption: x6.png

## 5 Experiments

### 5.1 Setup

All experiment setup follow the evaluation pipeline described in [section 4](https://arxiv.org/html/2506.09082v2#S4). More experiment details can be found in Appendix.

Metrics for AVAs.
For absolute depth and counting AVAs, we utilized a normalized mean absolute error (MAE) relative to the ground-truth. This normalization ensures that errors involving greater distances or counts, which are inherently more challenging, are proportionally penalized less severely. Localization performance is evaluated using Generalized Intersection-over-Union (GIoU rezatofighi2019generalized), color via CIEDE2000 luo2001development, and OCR through Average Normalized Levenshtein Similarity (ANLS) biten2019scene. All other AVAs employ standard accuracy metrics. Please refer to Appendix for more metric details.

### 5.2 Observations and Analyses

Figure: Figure 9: Impact of bounding boxes on spatial reasoning performance. With bounding boxes, all VFMs perform perfectly in spatial AVA; without them, models with weaker localization (MiDaS, SAM) perform worse.
Refer to caption: x7.png

Bounding boxes help isolate and assess specific AVAs. Bounding boxes serve as a crucial tool for disentangling AVAs, allowing us to isolate and evaluate specific capabilities rather than compounded performance on complex tasks. For example, when assessing spatial reasoning, VFMs exhibit uniformly strong and similar performance when provided with ground-truth bounding boxes, indicating a shared understanding of object relationships. However, removing the bounding box input transforms this into a composite task requiring both localization and spatial reasoning. As shown in [Figure 9](https://arxiv.org/html/2506.09082v2#S5.F9), this removal leads to a significant divergence in model performance. While SigLIP-2 and DINOv2 maintain high performance, Midas and SAM perform considerably worse in this composite setting. Analyzing their localization performance independently reveals that Midas and SAM possess significantly weaker localization capabilities compared to SigLIP-2 and DINOv2. This stark contrast confirms that performance degradation on composite tasks is often attributable to deficiencies in specific, untrained or underdeveloped constituent AVAs, such as localization in the case of Midas and SAM, rather than a fundamental failure in all visual understanding capacities.

Subgroup analyses reveal exceptions hidden by aggregate metrics. Aggregate metrics can sometimes obscure important nuanced insights. To gain a deeper understanding, we conduct detailed analyses by partitioning test samples based on specific criteria (e.g., object size in localization tasks) and examining whether these subgroup trends align with overall performance. We split localization testing samples based on normalized bounding box sizes (relative to image size), where 0.1 indicates an object occupies 10% of the image area. As illustrated in [Figure 10](https://arxiv.org/html/2506.09082v2#S5.F10), VFMs surprisingly exhibit minimal performance differences when localizing large objects (0.3–0.5). Conversely, performance disparities amplify as object size decreases, revealing significant weaknesses in MiDas and SAM for smaller objects. More subgroup analyses for other AVAs can be found in the Appendix.

Niche mastery of a modest model. Interestingly, almost all VFMs, including those with generally lower performance, excel in at least one specific AVA ([Figure 7](https://arxiv.org/html/2506.09082v2#S3.F7)). For example, SAM achieves exceptional results in color recognition, and DINOv2 excels notably in orientation.

Consistently good VFM performance across lower and mid-level AVAs. All VFMs, regardless of their training strategies, demonstrate good performance in low- to mid-level AVAs such as texture recognition, relative depth estimation, and object recognition ( [Figure 6](https://arxiv.org/html/2506.09082v2#S3.F6)). This uniformity implies that failures in complex visual reasoning predominantly arise from deficiencies in specific, critical AVAs rather than general shortcomings in visual understanding.

Language-aligned pretraining boosts performance on language-centric AVAs. AVAs that involve understanding visual information intertwined with text, such as OCR, substantially benefit from language-aligned pretraining. Non-language-aligned VFMs significantly underperform in these tasks, as illustrated in [Figure 6](https://arxiv.org/html/2506.09082v2#S3.F6), highlighting the importance of language alignment in VFM training.

The role of language supervision in VFM versatility. Language-supervised VFMs, specifically SigLIP-1/2 and AIMv2, demonstrate broad competency across AVAs ([Figure 7](https://arxiv.org/html/2506.09082v2#S3.F7) & [Figure 6](https://arxiv.org/html/2506.09082v2#S3.F6)). Their consistent high ranking highlights that language supervision is crucial to developing general-purpose visual reasoning abilities.

Figure: Figure 10: Detail results for localization for overall and different splits based on the ground-truth’s bounding box size normalized by image size. For example, 0.1 means the bounding box size is 10% of the image size. Higher GIoU is better.
Refer to caption: x8.png

## 6 Discussion

Struggles of Non-Language-Aligned VFMs. [Figure 7](https://arxiv.org/html/2506.09082v2#S3.F7) and [Figure 6](https://arxiv.org/html/2506.09082v2#S3.F6) indicate that non-language-aligned VFMs, despite exhibiting strengths in certain vision-centric or low-to-mid-level AVAs (e.g., DINOv2 excelling in orientation and SAM leadning in color), typically underperform in most AVAs. To investigate the underlying cause of these shortcomings, we conducted a preliminary analysis focusing on the visual feature representations before and after the LLM connector. Using linear probing on max-pooled visual features from DINOv2, we observed a substantial accuracy degradation from 66.3% (pre-connector) to 25.67% (post-connector) for the fine-grained recognition AVA, aligning with findings reported in previous work finer. This sharp performance drop suggests that critical visual information is often compromised during modality alignment processes. Recent studies have shown that partially unfreezing and fine-tuning the last few layers of VFMs can enhance performance chen2024sharegpt4v; however, this approach risks eroding the generalizability that initially makes these VFMs valuable. Alternatively, agglomerative models ranzinger2024radio; heinrich2024radio such as RADIO-2.1 demonstrate relatively robust performance on AVA-Bench, suggesting potential in combining specialized VFMs. Nevertheless, effectively aligning non-language-aligned VFMs to language modalities without sacrificing their inherent visual strengths remains a challenge, highlighting an important avenue for future research.

Platonic Representation Hypothesis Holds? Recent research suggests that large-scale training may lead VFMs toward converging onto similar representational spaces–Platonic Representation Hypothesis platonic. Our findings partially support this hypothesis: for certain AVAs such as object recognition, texture, and relative depth estimation, VFMs demonstrate universally strong performance, indicative of similar underlying visual representations irrespective of training objectives. Conversely, significant performance disparities among VFMs in most other AVAs, as revealed by our evaluation, indicate limitations to the generality of this hypothesis. Thus, our study suggests that the Platonic Representation Hypothesis might only hold in certain cases and warrants further empirical scrutiny in more diverse and challenging contexts.

What Are Things Going From Here? While MLLM have demonstrated remarkable versatility, they are not universally effective in all scenarios, especially in specialized domains cheng2024domain; liang2024comprehensive. Thus, there is a growing necessity for developing specialized MLLMs kumar2024diagnostics; li2025visual. Currently, selecting appropriate VFMs for such customized MLLMs remains largely heuristic tong2024cambrian; sun2025multi; li2025openvision. Our work provides actionable insights that transform this selection process from heuristic guesswork into principled engineering. By clearly identifying AVA-specific strengths and weaknesses, practitioners can now systematically choose VFMs to precisely address the particular visual demands of targeted downstream tasks. Moreover, AVA-Bench represents a critical step towards developing next-generation VFMs by providing a systematic, diagnostic, and comprehensive evaluation framework. This benchmark enables VFM developers to accurately pinpoint specific deficiencies and implement targeted improvements, fostering the creation of more robust, versatile, and well-rounded VFMs in the future.

## References

- (1)
Joshua Ainslie, James Lee-Thorp, Michiel De Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai.
Gqa: Training generalized multi-query transformer models from multi-head checkpoints.
arXiv preprint arXiv:2305.13245, 2023.
- (2)
Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan.
Foundation models defining a new era in vision: a survey and outlook.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
- (3)
Vidhisha Balachandran, Jingya Chen, Neel Joshi, Besmira Nushi, Hamid Palangi, Eduardo Salinas, Vibhav Vineet, James Woffinden-Luey, and Safoora Yousefi.
Eureka: Evaluating and understanding large foundation models.
arXiv preprint arXiv:2409.10566, 2024.
- (4)
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas.
Scene text visual question answering.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019.
- (5)
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.
On the opportunities and risks of foundation models.
arXiv preprint arXiv:2108.07258, 2021.
- (6)
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
Emerging properties in self-supervised vision transformers.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
- (7)
Hyunsik Chae, Seungwoo Yoon, Chloe Yewon Chun, Gyehun Go, Yongin Cho, Gyeongmin Lee, and Ernest K Ryu.
Decomposing complex visual comprehension into atomic visual skills for vision language models.
In The 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24.
- (8)
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin.
Sharegpt4v: Improving large multi-modal models with better captions.
In European Conference on Computer Vision, pages 370–387. Springer, 2024.
- (9)
Tianle Chen, Zheda Mai, Ruiwen Li, and Wei-lun Chao.
Segment anything model (sam) enhanced pseudo labels for weakly supervised semantic segmentation.
arXiv preprint arXiv:2305.05803, 2023.
- (10)
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.
arXiv preprint arXiv:2412.05271, 2024.
- (11)
Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Wayne Xin Zhao, Zhongzhi Luan, Bo Dai, and Zhenliang Zhang.
On domain-specific post-training for multimodal large language models.
arXiv preprint arXiv:2411.19930, 2024.
- (12)
Ming-Chang Chiu, Shicheng Wen, Pin-Yu Chen, and Xuezhe Ma.
Megacoin: Enhancing medium-grained color perception for vision-language models.
arXiv preprint arXiv:2412.03927, 2024.
- (13)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways.
Journal of Machine Learning Research, 24(240):1–113, 2023.
- (14)
Arpita Chowdhury, Dipanjyoti Paul, Zheda Mai, Jianyang Gu, Ziheng Zhang, Kazi Sajeed Mehrab, Elizabeth G Campolongo, Daniel Rubenstein, Charles V Stewart, Anuj Karpatne, et al.
Prompt-cam: A simpler interpretable transformer for fine-grained analysis.
arXiv preprint arXiv:2501.09333, 2025.
- (15)
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.
Describing textures in the wild.
In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
- (16)
Dawei Dai, Long Xu, Yutang Li, Yuanhui Zhang, and Shuyin Xia.
Humanvlm: Foundation for human-scene vision-language model.
Information Fusion, page 103271, 2025.
- (17)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.
An image is worth 16x16 words: Transformers for image recognition at scale.
arXiv preprint arXiv:2010.11929, 2020.
- (18)
Sagi Eppel, Mor Bismut, and Alona Faktor.
Shape and texture recognition in large vision-language models.
arXiv preprint arXiv:2503.23062, 2025.
- (19)
Miguel Espinosa, Chenhongyi Yang, Linus Ericsson, Steven McDonagh, and Elliot J Crowley.
There is no samantics! exploring sam as a backbone for visual understanding tasks.
arXiv preprint arXiv:2411.15288, 2024.
- (20)
Jiaqi Fan, Jianhua Wu, Jincheng Gao, Jianhao Yu, Yafei Wang, Hongqing Chu, and Bingzhao Gao.
Mllm-sul: Multimodal large language model for semantic scene understanding and localization in traffic scenarios.
arXiv preprint arXiv:2412.19406, 2024.
- (21)
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, et al.
Multimodal autoregressive pre-training of large vision encoders.
arXiv preprint arXiv:2411.14402, 2024.
- (22)
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna.
Blink: Multimodal large language models can see but not perceive.
In European Conference on Computer Vision, pages 148–166. Springer, 2024.
- (23)
Paul Gavrikov, Jovita Lukasik, Steffen Jung, Robert Geirhos, Bianca Lamm, Muhammad Jehanzeb Mirza, Margret Keuper, and Janis Keuper.
Are vision language models texture or shape biased and can we steer them?
arXiv preprint arXiv:2403.09193, 2024.
- (24)
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.
Vision meets robotics: The kitti dataset.
The international journal of robotics research, 32(11):1231–1237, 2013.
- (25)
Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al.
Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks.
Advances in Neural Information Processing Systems, 36:29343–29371, 2023.
- (26)
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the v in vqa matter: Elevating the role of image understanding in visual question answering.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
- (27)
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.
In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- (28)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
arXiv preprint arXiv:2407.21783, 2024.
- (29)
Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo, Matthew J Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E White, James Balhoff, et al.
Bioclip 2: Emergent properties from scaling hierarchical contrastive learning.
arXiv preprint arXiv:2505.23883, 2025.
- (30)
Agrim Gupta, Piotr Dollar, and Ross Girshick.
Lvis: A dataset for large vocabulary instance segmentation.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
- (31)
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.
A survey on vision transformer.
IEEE transactions on pattern analysis and machine intelligence, 45(1):87–110, 2022.
- (32)
Greg Heinrich, Mike Ranzinger, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, Pavlo Molchanov, et al.
Radio amplified: Improved baselines for agglomerative vision foundation models.
arXiv preprint arXiv:2412.07679, 2024.
- (33)
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
Parameter-efficient transfer learning for nlp.
In International conference on machine learning, pages 2790–2799. PMLR, 2019.
- (34)
Meng-Ru Hsieh, Yen-Liang Lin, and Winston H Hsu.
Drone-based object counting by spatially regularized regional proposal network.
In Proceedings of the IEEE international conference on computer vision, pages 4145–4153, 2017.
- (35)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
ICLR, 1(2):3, 2022.
- (36)
Jiaxing Huang and Jingyi Zhang.
A survey on evaluation of multimodal large language models.
arXiv preprint arXiv:2408.15769, 2024.
- (37)
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu.
T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.
Advances in Neural Information Processing Systems, 36:78723–78747, 2023.
- (38)
Mingxin Huang, Yongxin Shi, Dezhi Peng, Songxuan Lai, Zecheng Xie, and Lianwen Jin.
Ocr-reasoning benchmark: Unveiling the true capabilities of mllms in complex text-rich image reasoning.
arXiv preprint arXiv:2505.17163, 2025.
- (39)
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola.
Position: The platonic representation hypothesis.
In Forty-first International Conference on Machine Learning, 2024.
- (40)
Ji Hyeok Jung, Eun Tae Kim, Seo Yeon Kim, Joo Ho Lee, Bumsoo Kim, and Buru Chang.
Is’ right’right? enhancing object orientation understanding in multimodal language models through egocentric instruction tuning.
arXiv preprint arXiv:2411.16761, 2024.
- (41)
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.
Transformers in vision: A survey.
ACM computing surveys (CSUR), 54(10s):1–41, 2022.
- (42)
Jihyung Kil, Zheda Mai, Justin Lee, Arpita Chowdhury, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, and Wei-Lun Harry Chao.
Mllm-compbench: A comparative reasoning benchmark for multimodal llms.
Advances in Neural Information Processing Systems, 37:28798–28827, 2024.
- (43)
Jeonghwan Kim and Heng Ji.
Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models.
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6187–6207, 2024.
- (44)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.
Segment anything.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023.
- (45)
Aman Kumar, Mahbubul Alam, Ahmed Farahat, Maheshjabu Somineni, and Chetan Gupta.
Diagnostics-llava: A visual language model for domain-specific diagnostics of equipment.
In Annual Conference of the PHM Society, volume 16, 2024.
- (46)
Gustaf Kylberg.
Kylberg texture dataset v. 1.0.
Centre for Image Analysis, Swedish University of Agricultural Sciences and …, 2011.
- (47)
Boyan Li, Shengyi Ding, Deen Ma, Yixuan Wu, Hongjie Liao, and Kaiyuan Hu.
Llmcount: Enhancing stationary mmwave detection with multimodal-llm.
arXiv preprint arXiv:2409.16209, 2024.
- (48)
Deng Li, Xin Liu, Bohao Xing, Baiqiang Xia, Yuan Zong, Bihan Wen, and Heikki Kälviäinen.
Eald-mllm: Emotion analysis in long-sequential and de-identity videos with multi-modal large language model.
arXiv preprint arXiv:2405.00574, 2024.
- (49)
Haopeng Li, Lingbo Liu, Kunlin Yang, Shinan Liu, Junyu Gao, Bin Zhao, Rui Zhang, and Jun Hou.
Video crowd localization with multifocus gaussian neighborhood attention and a large-scale benchmark.
IEEE Transactions on Image Processing, 31:6032–6047, 2022.
- (50)
Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han.
Object detection in optical remote sensing images: A survey and a new benchmark.
ISPRS journal of photogrammetry and remote sensing, 159:296–307, 2020.
- (51)
Shan Li and Weihong Deng.
Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition.
IEEE Transactions on Image Processing, 28(1):356–370, 2019.
- (52)
Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, and Cihang Xie.
Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning.
arXiv preprint arXiv:2505.04601, 2025.
- (53)
Yifan Li, Zhixin Lai, Wentao Bao, Zhen Tan, Anh Dao, Kewei Sui, Jiayi Shen, Dong Liu, Huan Liu, and Yu Kong.
Visual large language models for generalized and specialized applications.
arXiv preprint arXiv:2501.02765, 2025.
- (54)
Zheng Lian, Ya Li, Jian-Hua Tao, Jian Huang, and Ming-Yue Niu.
Expression analysis based on face regions in real-world conditions.
International Journal of Automation and Computing, 17:96–107, 2020.
- (55)
Chia Xin Liang, Pu Tian, Caitlyn Heqi Yin, Yao Yua, Wei An-Hou, Li Ming, Tianyang Wang, Ziqian Bi, and Ming Liu.
A comprehensive survey and guide to multimodal large language models in vision-language tasks.
arXiv preprint arXiv:2411.06284, 2024.
- (56)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
Advances in neural information processing systems, 36:34892–34916, 2023.
- (57)
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai.
Ocrbench: on the hidden mystery of ocr in large multimodal models.
Science China Information Sciences, 67(12):220102, 2024.
- (58)
Vincenzo Lomonaco, Lorenzo Pellegrini, Pau Rodriguez, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vazquez, et al.
Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions.
Artificial Intelligence, 303:103635, 2022.
- (59)
M Ronnier Luo, Guihua Cui, and Bryan Rigg.
The development of the cie 2000 colour-difference formula: Ciede2000.
Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, 26(5):340–350, 2001.
- (60)
Zheda Mai, Arpita Chowdhury, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, et al.
Fine-tuning is fine, if calibrated.
Advances in Neural Information Processing Systems, 37:136084–136119, 2024.
- (61)
Zheda Mai, Hyunwoo Kim, Jihwan Jeong, and Scott Sanner.
Batch-level experience replay with review for continual learning.
arXiv preprint arXiv:2007.05683, 2020.
- (62)
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner.
Online continual learning in image classification: An empirical survey.
Neurocomputing, 469:28–51, 2022.
- (63)
Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner.
Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3589–3599, 2021.
- (64)
Zheda Mai, Ping Zhang, Cheng-Hao Tu, Hong-You Chen, Quang-Huy Nguyen, Li Zhang, and Wei-Lun Chao.
Lessons and insights from a unifying study of parameter-efficient fine-tuning (peft) in visual recognition.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14845–14857, 2025.
- (65)
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.
Fine-grained visual classification of aircraft.
arXiv preprint arXiv:1306.5151, 2013.
- (66)
P Mallikarjuna, Alireza Targhi, Mario Fritz, Eric Hayman, Barbara Caputo, and J.-O Eklundh.
The kth-tips2 database.
07 2006.
- (67)
Wenfeng Mi, He Chen, and Weipeng Liu.
Hierarchical interpretable vision reasoning driven through a multi-modal large language model for depth estimation.
In 2024 China Automation Congress (CAC), pages 829–834. IEEE, 2024.
- (68)
A. Mishra, K. Alahari, and C. V. Jawahar.
Scene text recognition using higher order language priors.
In BMVC, 2012.
- (69)
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al.
Moments in time dataset: one million videos for event understanding.
IEEE transactions on pattern analysis and machine intelligence, 42(2):502–508, 2019.
- (70)
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.
Intriguing properties of vision transformers.
Advances in Neural Information Processing Systems, 34:23296–23308, 2021.
- (71)
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.
Dinov2: Learning robust visual features without supervision.
arXiv preprint arXiv:2304.07193, 2023.
- (72)
Dipanjyoti Paul, Arpita Chowdhury, Xinqi Xiong, Feng-Ju Chang, David Carlyn, Samuel Stevens, Kaiya Provost, Anuj Karpatne, Bryan Carstens, Daniel I Rubenstein, et al.
A simple interpretable transformer for fine-grained image classification and analysis.
2023.
- (73)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In International conference on machine learning, pages 8748–8763. PmLR, 2021.
- (74)
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.
Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.
IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020.
- (75)
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai.
Learning to count everything.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3394–3403, 2021.
- (76)
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov.
Am-radio: Agglomerative vision foundation model reduce all domains into one.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024.
- (77)
Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.
Generalized intersection over union: A metric and a loss for bounding box regression.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658–666, 2019.
- (78)
Ranjan Sapkota and Manoj Karkee.
Object detection with multimodal large vision-language models: An in-depth review.
Available at SSRN 5233953, 2025.
- (79)
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.
Objects365: A large-scale, high-quality dataset for object detection.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 8430–8439, 2019.
- (80)
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang.
Online class-incremental continual learning with adversarial shapley value.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9630–9638, 2021.
- (81)
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.
Indoor segmentation and support inference from rgbd images.
In Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746–760. Springer, 2012.
- (82)
Li Sun, Chaitanya Ahuja, Peng Chen, Matt D’Zmura, Kayhan Batmanghelich, and Philip Bontrager.
Multi-modal large language models are effective vision learners.
In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 8617–8626. IEEE, 2025.
- (83)
Andrew Szot, Bogdan Mazoure, Harsh Agrawal, R Devon Hjelm, Zsolt Kira, and Alexander Toshev.
Grounding multimodal large language models in actions.
Advances in Neural Information Processing Systems, 37:20198–20224, 2024.
- (84)
D. Temel, J. Lee, and G. AlRegib.
Cure-or: Challenging unreal and real environments for object recognition.
In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018.
- (85)
texture classification and segmentation.
Kth-tips dataset.
[https://universe.roboflow.com/texture-classification-and-segmentation/kth-tips](https://universe.roboflow.com/texture-classification-and-segmentation/kth-tips), feb 2024.
visited on 2025-05-16.
URL: [https://universe.roboflow.com/texture-classification-and-segmentation/kth-tips](https://universe.roboflow.com/texture-classification-and-segmentation/kth-tips).
- (86)
Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, and Damayanthi Herath.
Semantic segmentation using vision transformers: A survey.
Engineering Applications of Artificial Intelligence, 126:106669, 2023.
- (87)
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al.
Cambrian-1: A fully open, vision-centric exploration of multimodal llms.
Advances in Neural Information Processing Systems, 37:87310–87356, 2024.
- (88)
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.
Eyes wide shut? exploring the visual shortcomings of multimodal llms.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568–9578, 2024.
- (89)
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al.
Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.
arXiv preprint arXiv:2502.14786, 2025.
- (90)
Cheng-Hao Tu, Hong-You Chen, Zheda Mai, Jike Zhong, Vardaan Pahuja, Tanya Berger-Wolf, Song Gao, Charles Stewart, Yu Su, and Wei-Lun Harry Chao.
Holistic transfer: Towards non-disruptive fine-tuning with partial target data.
Advances in Neural Information Processing Systems, 36:29149–29173, 2023.
- (91)
Cheng-Hao Tu, Zheda Mai, and Wei-Lun Chao.
Visual query tuning: Towards effective usage of intermediate representations for parameter and memory efficient transfer learning.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7725–7735, 2023.
- (92)
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha.
Benchmarking representation learning for natural world image collections.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12884–12893, 2021.
- (93)
Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie.
Coco-text: Dataset and benchmark for text detection and recognition in natural images, 2016.
URL: [https://arxiv.org/abs/1601.07140](https://arxiv.org/abs/1601.07140), [arXiv:1601.07140](https://arxiv.org/abs/1601.07140).
- (94)
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.
The caltech-ucsd birds-200-2011 dataset.
2011.
- (95)
Zhihua Wang, Yu Long, Qiuping Jiang, Chao Huang, and Xiaochun Cao.
Harnessing multi-modal large language models for measuring and interpreting color differences.
IEEE Transactions on Image Processing, 2025.
- (96)
Diankun Wu, Fangfu Liu, Yi-Hsin Hung, and Yueqi Duan.
Spatial-mllm: Boosting mllm capabilities in visual-based spatial intelligence.
arXiv preprint arXiv:2505.23747, 2025.
- (97)
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al.
Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks.
Advances in Neural Information Processing Systems, 37:69925–69975, 2024.
- (98)
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu.
Multimodal large language models: A survey.
In 2023 IEEE International Conference on Big Data (BigData), pages 2247–2256. IEEE, 2023.
- (99)
Xindi Wu, Hee Seung Hwang, Polina Kirichenko, and Olga Russakovsky.
Compact: Compositional atomic-to-complex visual capability tuning.
arXiv preprint arXiv:2504.21850, 2025.
- (100)
Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora.
Conceptmix: A compositional image generation benchmark with controllable difficulty.
arXiv preprint arXiv:2408.14339, 2024.
- (101)
xAI.
grok, 2024.
URL: [https://x.ai/news/grok-1.5v](https://x.ai/news/grok-1.5v).
- (102)
Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu.
Aid: A benchmark data set for performance evaluation of aerial scene classification.
IEEE Transactions on Geoscience and Remote Sensing, 55(7):3965–3981, 2017.
- (103)
Zhongyi Xia and Tianzhao Wu.
Large language models can understanding depth from monocular images.
arXiv preprint arXiv:2409.01133, 2024.
- (104)
Hanchen Xie, Rose Ma, Jiageng Zhu, Zheda Mai, Wael Abd-Almageed, and Zubin Abraham.
Efficiently mitigating video content misalignment on large vision model with time-series data alignment.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3301–3307, 2025.
- (105)
Qu Yang, Mang Ye, and Bo Du.
Emollm: Multimodal emotional understanding meets large language models.
arXiv preprint arXiv:2406.16442, 2024.
- (106)
Ziyu Yao, Xuxin Cheng, Zhiqi Huang, and Lei Li.
Countllm: Towards generalizable repetitive action counting via large language model.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19143–19153, 2025.
- (107)
Hang Yin, Zhifeng Lin, Xin Liu, Bin Sun, and Kan Li.
Do multimodal language models really understand direction? a benchmark for compass direction reasoning.
In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025.
- (108)
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang.
Mm-vet: Evaluating large multimodal models for integrated capabilities.
arXiv preprint arXiv:2308.02490, 2023.
- (109)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training.
In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023.
- (110)
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu.
Mm-llms: Recent advances in multimodal large language models.
arXiv preprint arXiv:2401.13601, 2024.
- (111)
Huanyu Zhang, Chengzu Li, Wenshan Wu, Shaoguang Mao, Ivan Vulić, Zhang Zhang, Liang Wang, Tieniu Tan, Furu Wei, et al.
A call for new recipes to enhance spatial reasoning in mllms.
arXiv preprint arXiv:2504.15037, 2025.
- (112)
Jinchang Zhang and Guoyu Lu.
Vision-language embodiment for monocular depth estimation.
In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29479–29489, 2025.
- (113)
Ping Zhang, Zheda Mai, Quang-Huy Nguyen, and Wei-Lun Chao.
Revisiting semi-supervised learning in the era of foundation models.
arXiv preprint arXiv:2503.09707, 2025.
- (114)
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al.
Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans?
arXiv preprint arXiv:2408.13257, 2024.
- (115)
Ziheng Zhang, Jianyang Gu, Arpita Chowdhury, Zheda Mai, David Carlyn, Tanya Berger-Wolf, Yu Su, and Wei-Lun Chao.
Finer-cam: Spotting the difference reveals finer details for visual explanation.
arXiv preprint arXiv:2501.11309, 2025.
- (116)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.
Judging llm-as-a-judge with mt-bench and chatbot arena.
Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
- (117)
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.
Places: A 10 million image database for scene recognition.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
- (118)
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
Minigpt-4: Enhancing vision-language understanding with advanced large language models.
arXiv preprint arXiv:2304.10592, 2023.

## Appendix

## Appendix A AVA-Bench Details

**Table 1: Atomic Visual Abilities (AVAs). We identify 14 AVAs, serving as the foundational capabilities that can be combined to tackle complex visual reasoning tasks. For each AVA, we provide the definition and an example question in ourbench.**
| Atomic Visual Ability | Definition | Example Question |
| --- | --- | --- |
| Counting | Determining the number of instances of an object | How many apples are in the image? |
| Localization | Identifying the location of an object in the image | Provide bounding box coordinate for bicycle. |
| Fine-Grained | Differentiating between similar sub-categories of objects | What species of fungi is in the image? |
| OCR | Reading and interpreting text visible in the image | What is written in the red bounding box in the image? |
| Absolute Depth | Estimating how far an object is from the camera | From the camera’s perspective, estimate how far the closest point of the car (red box) is from the camera in real-world distance, in meters. |
| Relative Depth | Comparing distances of two objects from the camera | Which object is closer to the camera, the car (red box) or the cyclist (blue box) to the camera? |
| Orientation | Determining the facing direction or angle of an object | What is the orientation of the toy bus in the image? |
| Spatial | Inferring layout and spatial relations | Considering the relative positions of two objects in the image, where is the bicycle (red box) located with respect to the towel (blue box)? |
| Object Recognition | Identifying objects present in the image given bounding box | What is in the red bounding box in the image? |
| Scene Recognition | Identifying the broader environment or type of setting | What is the scene class of the image? |
| Action Recognition | Determining what action is being performed | Which action or activity is shown in the image? |
| Texture | Describing surface appearance or material of objects | What is the texture attribute of image? |
| Color | Identifying colors of objects given bounding box | What color is shown within the bounding box? |
| Emotion | Recognizing emotional expressions in humans given bounding box | What emotion is being shown in the image? |

### A.1 A tomic V isual A bilities ( AVAs )

As mentioned in Section 3.1, AVAs are elemental visual capabilities that can be combined to address more complex visual reasoning tasks. The definitions and representative questions for each AVA can be found in [Table 1](https://arxiv.org/html/2506.09082v2#A1.T1). Additional qualitative illustrations are provided in Figure 2 and Figure 5.

The 14 AVAs selected for AVA-Bench are grounded in a thorough literature analysis.

- 1.
Compositional text-to-image (T2I) benchmarks. Studies on controllable generation motivate core visual primitives—number, colour, texture, object identity, spatial relations, and more—used to construct compositional prompts huang2023t2i; wu2024conceptmix. These primitives form an initial pool of candidate abilities.
- 2.
VQA question analysis. We employ GPT-4 to summarize the visual skills demanded by VQA questions in various commonly-used datasets (VQAv2 goyal2017making, RealWorldQA RealWorldQA, GQA ainslie2023gqa, etc.), thereby enriching the pool with abilities emphasized by real-world questions.

Intersecting these two sources yields a concise yet crucial set of AVAs. Moreover, we focus strictly on pure perceptual tasks and exclude non-perceptual reasoning skills (e.g., historical context, mathematical reasoning). We provide more related work discussion in [Appendix D](https://arxiv.org/html/2506.09082v2#A4).

### A.2 Dataset Curation

Spatial Reasoning wu2025spatial; zhang2025call.
We curate 15.9K image pairs from NYU-Depth V2 silberman2012indoor (indoor scenes) and LVIS gupta2019lvis and Objects365 shao2019objects365 (open-domain scenes), all containing instance segmentation annotations. In each image, two distinct, non-overlapping objects are selected—one highlighted with a blue bounding box (reference object) and another with a red bounding box (target object). The model must identify the relative spatial position of the red box with respect to the blue box, choosing from four multiple-choice options: *Left above*, *Left below*, *Right above*, and *Right below* ([Figure 11](https://arxiv.org/html/2506.09082v2#A1.F11)). The preprocessing steps for dataset creation are summarized below:

- •
To prevent ambiguity in interpretation, we restrict each image to contain only one instance of the target and reference objects.
- •
Object pairs whose bounding boxes overlapped either horizontally or vertically were excluded, ensuring unambiguous assignment to the four spatial categories.
- •
Extremely small bounding boxes complicating localization were filtered out. Specifically, object instances covering at least 2% of the image area for NYU-Depth V2, and at least 0.2% for LVIS and Objects365, were retained.
- •
Ambiguous object categories were excluded via a manually curated list of label IDs that often lack clear boundaries or meaningful depth interpretations from NYU-Depth V2 (e.g., wall, floor, ceiling, etc.). GPT-o1 was used to unify labels across datasets, identifying common object names conveying the same meaning, and merging these samples accordingly.
- •
Each object class was ensured to appear in multiple spatial positions, with 40 samples per spatial position category. This prevents models from memorizing fixed layouts.
- •
For each object class and spatial position category, an 80% train and 20% test split was ensured, for uniform distribution and fair evaluation.
- •
The question for each pair: “Considering the relative positions of two objects in the image, where is the microphone (annotated by the red box) located with respect to the speaker (annotated by the blue box)? Choose from A. Left above, B. Left below, C. Right above, D. Right below.”

Figure: Figure 11: Examples of Spatial Reasoning AVA Samples.
Refer to caption: x9.png

Counting yao2025countllm; li2024llmcount.
We curate a total of 11.3K images from five datasets— VQAv2 goyal2017making, FSC-147 ranjan2021learning, CARPK hsieh2017drone, LVIS gupta2019lvis , and CrowdHuman li2022video —to evaluate object counting abilities across diverse domains, including open-domain scenes, natural objects, structured environments, and densely crowded contexts. Each image is paired with a question prompting the model to count the number of instances of a specified object category, and the model must return an integer-valued answer([Figure 12](https://arxiv.org/html/2506.09082v2#A1.F12)). The preprocessing steps for dataset creation are summarized below:

- •
To ensure valid supervision, we filter all samples to retain only those with non-zero object counts and with object_count $\leq$ 40.
- •
For each object category (object id), we require at least 4–5 distinct object count values to be represented, preventing overfitting to static object layouts.
- •
For each object count, we sample a fixed range of images per object_id—between 6 and 30 depending on the dataset—to balance frequency and diversity.
- •
Dataset-specific sampling rules are applied:
–
VQAv2 and LVIS: 15–30 images per object count; $\geq$5 count values per object id.
–
FSC-147: 6–12 images per object count; $\geq$4 count values per object id.
–
CARPK: 10–20 images per object count; $\geq$5 count values per object id.
–
CrowdHuman: object count capped at 40; 25–50 images per count level.
- •
An 80% train / 20% test split is maintained for each object id and count level to ensure balanced distribution during evaluation.
- •
The question for each image is: “How many [object] are there in the image?”, where [object] refers to the annotated target category.

Figure: Figure 12: Examples of Counting AVA Samples.
Refer to caption: x10.png

**Table 2: Detailed statistics of AVA-Bench.**
| Atomic Visual Abilities (AVA) | Dataset | Domain | # Train Samples | # Test Samples |
| --- | --- | --- | --- | --- |
| Localization | Objects365 shao2019objects365 | Open | 27.8K | 6.9K |
|  | LVIS gupta2019lvis | Open |  |  |
|  | iNaturalist-2021 van2021benchmarking | Bird, Animal |  |  |
|  | DIOR li2020object | Remote-Sensing |  |  |
| Counting | VQAv2 goyal2017making | Open | 9K | 2.4K |
|  | FSC ranjan2021learning | Open |  |  |
|  | CARPK hsieh2017drone | Car |  |  |
|  | Crowd Surveillance li2022video | People |  |  |
|  | LVIS gupta2019lvis | Open |  |  |
| Fine-grained | CUB-200-2011 wah2011caltech | Bird | 7.2K | 1.8K |
|  | iNaturalist-2021 van2021benchmarking | Fungi,Plant, Animal |  |  |
|  | FGVC-Aircraft maji2013fine | Object |  |  |
| Absolute Depth | NYU-Depth V2 silberman2012indoor | Indoor Scene | 6.9K | 1.8K |
|  | KITTI geiger2013vision | Outdoor Scene |  |  |
| Relative Depth | NYU-Depth V2 silberman2012indoor | Indoor Scene | 9.2K | 2.4K |
|  | KITTI geiger2013vision | Outdoor Scene |  |  |
| OCR | COCO-Text veit2016cocotextdatasetbenchmarktext | Open | 8.8K | 2.2K |
|  | IIIT5K  MishraBMVC12 | Open |  |  |
|  | TextVQA balanced_vqa_v2 | Open |  |  |
| Orientation | EgoOrientBench jung2024right | Open | 6.9K | 1.6K |
|  | CURE-OR Temel2018_ICMLA | Indoor |  |  |
| Object Recognition | Objects365 shao2019objects365 | Open | 27.9K | 6.9K |
|  | LVIS gupta2019lvis | Open |  |  |
|  | iNaturalist-2021 van2021benchmarking | Bird, Animal |  |  |
|  | DIOR li2020object | Remote-Sensing |  |  |
| Action Recognition | MiT monfort2019moments | Open | 12.1K | 3K |
| Texture | DTD cimpoi14describing | Open | 10.6K | 2.7K |
|  | Kylberg kylberg2011kylberg | Open |  |  |
|  | KTH-TIPS kth-tips_dataset | Open |  |  |
|  | KTH-TIPS2 article | Open |  |  |
| Spatial Reasoning | Objects365 shao2019objects365 | Open | 13K | 2.9K |
|  | LVIS gupta2019lvis | Open |  |  |
|  | NYU-Depth V2 silberman2012indoor | Indoor Scene |  |  |
| Scene Recognition | Places434 zhou2017places | Open | 11.1K | 2.8K |
|  | AID xia2017aid | Remote-Sensing |  |  |
| Emotion | RAF-DB li2019reliable | Human | 11.9K | 5.1K |
|  | ExpW lian2020expression | Human |  |  |
| Color | Objects365 shao2019objects365 | Open | 11.2K | 2.8K |
| Total | - | - | 173.5K | 45.3K |

Fine-grained paul2023simple; gu2025bioclip; zhang2025finer.
We curate a total of 9K images from five fine-grained recognition domains—Bird, Animal, Fungi, Plant, and Object—to assess species-level recognition capabilities. The dataset sources include CUB-200-2011 wah2011caltech for birds, iNat21 van2021benchmarking for animals, fungi, and plants, and FGVC Aircraft maji2013fine for objects. Each sample contains an image and a question prompting the model to identify the specific species or object type([Figure 13](https://arxiv.org/html/2506.09082v2#A1.F13)). Construction details are as follows:

- •
We select 100 bird species from CUB-200-2011, and 50 random classes each from the Animal, Fungi, and Plant categories of iNat21, as well as 50 classes from FGVC Aircraft for the Object category. All random selections use a fixed random seed to ensure reproducibility.
- •
This results in 300 total object ids. For each class, we uniformly sample 30 images.
- •
For iNat21 entries, we format species names by retaining only the last two words of their taxonomic labels for clarity and consistency.
- •
For each object class, an 80% training and 20% testing split was established to ensure balanced per-class evaluation.
- •
For each multiple-choice question, the candidate list includes all 50 or 100 (Object only) class names used in that task split, ensuring consistent, closed-set evaluation.
- •
The question for each image is: “What species of bird is in the image? Choose one from below: 1. Cerulean_Warbler, 2. American_Crow, …, 100. Pine_Warbler ”

Figure: Figure 13: Examples of Fine-grained recognition AVA Samples.
Refer to caption: x11.png

OCR huang2025ocr; liu2024ocrbench.
We curate 10.9K images from three OCR datasets—COCO-Text veit2016cocotextdatasetbenchmarktext, IIIT5K MishraBMVC12, and TextVQA balanced_vqa_v2—each containing word-level bounding box annotations. In each image, a red bounding box highlights the word to be transcribed. The model is prompted to recognize the textual content inside the box based on visual context([Figure 14](https://arxiv.org/html/2506.09082v2#A1.F14)). The preprocessing steps for dataset creation are summarized below:

- •
For COCO-Text, we retain only word-level boxes with an area greater than 1500 pixels to ensure sufficient visual resolution.
- •
For TextVQA, we apply a stricter filtering criterion by retaining only word boxes with area larger than 2000 pixels.
- •
For IIIT5K, we randomly sample 2,000 images from the original dataset without applying any area-based filtering.
- •
Each dataset is split into 80% training and 20% validation subsets individually, before merging the resulting splits to form the final OCR benchmark.
- •
A red bounding box is rendered on each image to highlight the target word location during inference.
- •
The question for each image: “What is written in the red bounding box in the image?”

Figure: Figure 14: Examples of OCR AVA Samples.
Refer to caption: x12.png

Localization wu2024visionllm; sapkota2025object.
We curate 34.8K localization-focused image–question pairs sourced from Objects365 shao2019objects365 and LVIS gupta2019lvis (open-domain), iNaturalist-2021 van2021benchmarking (birds and animals), and DIOR li2020object (remote-sensing). Each image contains a single object instance, and the model is prompted to identify its location by providing the bounding box coordinates([Figure 15](https://arxiv.org/html/2506.09082v2#A1.F15)). The preprocessing steps for dataset creation are summarized below:

- •
We only retain object instances whose category appears exactly once in an image, to avoid ambiguity in localization supervision.
- •
We filter out objects with extremely small or large bounding boxes, retaining only those whose normalized area falls within the range of $0.002<\text{area}<0.5$ relative to the image.
- •
For Objects365 and LVIS, we manually select 20 target object categories each. If the number of valid images for a category exceeds 700, we randomly sample 700 using a fixed seed for reproducibility.
- •
For iNaturalist-2021, we select 10 categories from the aves (birds) class and 10 from the mammalia (mammals) class, following the same filtering and sampling strategy. All scientific names are mapped to common names to improve interpretability and model alignment.
- •
For DIOR, we follow the same filtering steps and manually select 10 target categories, sampling up to 700 images per category as needed.
- •
All images are padded to square format using a consistent background color computed as the mean RGB value across multiple image processors:
$\text{background color}=RGB(124,120,111)$
The padding preserves content aspect ratio and ensures uniform input dimensions across models.
- •
For each object category, an 80% training and 20% testing split was performed after filtering and sampling, ensuring balanced and fair evaluation.
- •
The question for each image follows the format: “Provide bounding box coordinate for red-tailed hawk.” The object name is dynamically replaced depending on the image.

Figure: Figure 15: Examples of Localization AVA Samples.
Refer to caption: x13.png

Recognition fu2024blink; zhang2025revisiting; tu2023holistic.
We curate 34.8K recognition samples from four datasets spanning diverse visual domains—Objects365 shao2019objects365 and LVIS gupta2019lvis (open-domain objects), iNaturalist-2021 van2021benchmarking (birds and animals), and DIOR li2020object (remote sensing). These samples are derived from the same images and object instances used in the localization benchmark. However, instead of asking for bounding box prediction, the recognition task requires the model to identify the object within a visually highlighted region ([Figure 16](https://arxiv.org/html/2506.09082v2#A1.F16)).

The preprocessing steps for dataset creation are summarized below:

- •
We apply the same curation strategy as in localization: only one valid instance per image, with normalized bounding box area between $0.2\%$ and $50\%$ of the image. For each dataset, 10 or 20 object categories are manually selected.
- •
From Objects365 and LVIS, we select 20 object categories each, and randomly sample up to 700 images per category (using a fixed seed for reproducibility).
- •
From iNaturalist-2021, we retain 10 species from the Aves (birds) and 10 from Mammalia (mammals) branches. Scientific names are converted to common English names for accessibility. Each species contributes up to 700 images.
- •
From DIOR, we select 10 object categories and apply the same filtering and sampling strategy (max 700 images per class).
- •
All images are padded to square shape using a consistent background color, computed from the average mean pixel values across nine vision-language processors, to ensure uniform input dimensions.
- •
For each object category, an 80% training and 20% testing split was performed after filtering and sampling, ensuring balanced and fair evaluation.
- •
Unlike localization, where bounding boxes are not rendered and must be predicted, in recognition the red bounding box is explicitly drawn onto each image to guide the model’s attention.
- •
The question format is: “What is in the red bounding box? Choose from the following option: 1. airport, 2. american robin, …, 70. vulpes vulpes” The 70 object categories are shared across datasets and randomly shuffled for each question instance.

Figure: Figure 16: Examples of Recognition AVA Samples.
Refer to caption: x14.png

Color wang2025harnessing; chiu2024megacoin.
We curate 14K images from Objects365 shao2019objects365 and LVIS gupta2019lvis, each contributing 7K samples. This AVA focuses on assessing color perception in natural scenes. For each image, we extract a coherent color region using the following pipeline:

- •
We apply SLIC superpixel segmentation and convert the image to LAB color space.
- •
Superpixels with similar color values are merged to form larger regions of consistent color.
- •
Among all candidate regions, we select the top-1 region with the lowest internal color variance as the final choice.
- •
A red bounding box is drawn on the selected region, and the most frequent RGB color within this region is used as the answer.
- •
For each object category, an 80% training and 20% testing split was performed after filtering and sampling, ensuring balanced and fair evaluation.
- •
The question format for each sample is: “What color is shown within the red bounding box?”

Action szot2024grounding; wang2025harnessing; xie2025efficiently.
We curate a total of 15K image–question pairs from the Moments in Time monfort2019moments dataset, covering a wide range of human actions and activities. Each sample is derived from a short video clip annotated with a specific action label ([Figure 17](https://arxiv.org/html/2506.09082v2#A1.F17)). Construction details are as follows:

- •
Similar action labels are merged into a unified category for clarity.
- •
For each class, we randomly sample up to 500 training videos and 100 validation videos, ensuring broad yet balanced category coverage.
- •
The middle frame of each selected video is extracted and used as the image representing the associated action.
- •
Since listing all classes may exceed the token limits of many vision–language models, we randomly sample 100 action options per question, ensuring the ground-truth answer is always included.
- •
The question format for each sample is: “Which action or activity is shown in the image? Choose from the following option: 1. buying, 2. catching, …, 100. boxing”

Figure: Figure 17: Examples of Action AVA Samples.
Refer to caption: x15.png

Emotion yang2024emollm; li2024eald.
We curate 16K image-question pairs from two large-scale facial expression datasets: RAF-DB li2019reliable and ExpW lian2020expression. These datasets consist of human portraits labeled with one of seven basic emotions: happy, sad, angry, fear, surprise, neutral, and disgust. Each image is annotated with a bounding box localizing the face of interest.

The preprocessing steps for dataset creation are summarized below:

- •
Emotion labels across datasets were unified by consolidating synonymous terms (e.g., happiness and happy, anger and angry) to ensure consistent categorization across all samples.
- •
Bounding box annotations provided in the datasets were used to highlight the specific individual in multi-person scenes.
- •
An 80/20 train-test split was applied independently per emotion category to maintain class balance during evaluation.
- •
The question for each image is framed as: “Which of the following best describes the person’s emotion in the red box? 1. happy, 2. sad, 3. angry, 4. fear, 5. surprise, 6. neutral, 7. disgust.”

Scene dai2025humanvlm; fan2024mllm.
We curate 13.9K image-question pairs from two diverse datasets: Places434 zhou2017places (open-domain) and AID xia2017aid (remote sensing). Each image is paired with a multiple-choice question, where the model selects the correct scene category from a pool of 30 randomly sampled options. The full set includes 464 unique scene classes spanning a wide range of environments([Figure 18](https://arxiv.org/html/2506.09082v2#A1.F18)). The preprocessing steps for dataset creation are summarized below:

- •
GPT-4o was utilized to standardize labels across datasets by converting fine-grained labels into single, unified labels. Humans carefully checked each conversion to merge the newly converted labels conveying the same meaning with existing labels, ensuring semantic clarity and fluency.
- •
A uniform distribution was maintained by extracting exactly 30 images per scene category, preventing category imbalance and ensuring consistent representation across classes.
- •
For each scene category within both datasets, an 80% training and 20% testing split was established, ensuring balanced and fair evaluation conditions.
- •
The question for each pair: “What is the scene class of the image? Choose one from below: 1. Entrance hall, 2. Lawn, … 30. Snowy Mountain.” These 30 options were selected by randomly sampling from the complete set of scene classes within each respective dataset, maintaining diversity and preventing predictable patterns.

Figure: Figure 18: Examples of Scene AVA Samples.
Refer to caption: x16.png

Texture eppel2025shape; gavrikov2024vision.
To assess a VFM’s ability to distinguish fine-grained visual patterns, we curate 13.2K image-question pairs from diverse surface textures using close-up images from four open-domain datasets: DTD cimpoi14describing , Kylberg kylberg2011kylberg , KTH-TIPS kth-tips_dataset and KTH-TIPS2 article. These datasets encompass a diverse array of texture types—such as *striped*, *aluminum foil*, and *zigzagged*—capturing subtle visual patterns that are essential for accurate texture recognition. Each image is paired with a multiple-choice question, requiring the model to select the correct texture label from a set of options([Figure 19](https://arxiv.org/html/2506.09082v2#A1.F19)). The preprocessing steps for dataset creation are summarized below:

- •
GPT-4o was used to unify labels across datasets, identifying common texture attributes conveying similar meanings. These unified labels were manually verified to retain only one representative for synonymous terms.
- •
Images where textures appeared as part of larger objects in cluttered scenes or within complex real-world photographs were discarded, to ensure that textures were clearly localized and recognizable without contextual interference.
- •
Each texture attribute was represented by multiple images, with a minimum of 120 and a maximum of 480 samples per attribute, ensuring diversity and preventing memorization of fixed patterns by the models.
- •
For each texture attribute, an 80% train and 20% test split was ensured, reaching uniform distribution and fair evaluation.
- •
The question for each pair:“What is the texture attribute of the image? Choose one from below: 1. banded, 2. blotchy, …, 47. veined.” The provided options exactly match the entire option pool from each respective dataset and were shuffled to avoid bias.

Figure: Figure 19: Examples of Texture AVA Samples.
Refer to caption: x17.png

Orientation yin2025multimodal; jung2024right.
To evaluate viewpoint understanding, we curate 10.1K image-question pairs from two specialized datasets: CURE-OR Temel2018_ICMLA and EgoOrientBench jung2024right . These datasets provide uncluttered images of objects captured from nine distinct orientations—*front*, *back*, *left*, *right*, *top*, *front left*, *front right*, *back left*, and *back right*—allowing models to learn pose-specific cues without requiring bounding boxes([Figure 20](https://arxiv.org/html/2506.09082v2#A1.F20)). The preprocessing steps for dataset creation are summarized below:

- •
For EgoOrientBench, each object class was ensured to appear in multiple orientations, with at least 10 and at most 40 samples per orientation label. This encourages models to learn generalized representations rather than memorizing specific arrangements.
- •
From CURE-OR, we selected object instances photographed against two different background conditions using three distinct capture devices, ensuring variation in imaging style without compromising clarity.
- •
For each object, 80% of the images from each orientation were assigned to the training set, and the remaining 20% to the test set, ensuring balanced representation and fair evaluation across orientations.
- •
The question for each pair: "What is the orientation of the toy plane in the image? Choose one from below: 1. front, 2. front right, 3. right, 4. back right, 5. back, 6. back left, 7. left, 8. front left, 9. top". These nine options represent the common orientation labels provided by both datasets.

Figure: Figure 20: Examples of Orientation AVA Samples.
Refer to caption: x18.png

Absolute Depth xia2024large; mi2024hierarchical; zhang2025vision.
We curate 9K image-object pairs from NYU-Depth V2 silberman2012indoor for indoor scenes and KITTI geiger2013vision for outdoor scenes. These datasets contain aligned RGB and depth information. Each image includes an object annotated with a bounding box, and the task requires estimating the absolute depth of the object in meters. This value is then matched against discretized ground-truth bins for evaluation([Figure 21](https://arxiv.org/html/2506.09082v2#A1.F21)).

The preprocessing steps for indoor dataset creation are summarized below:

- •
Ambiguous object categories were excluded via a manually curated list of label IDs that often lack clear boundaries or meaningful depth interpretations (e.g., wall, floor, ceiling, etc.).
- •
Images were resized to a fixed resolution of $384\times 384$, padding vertically as needed to preserve the original aspect ratio.
- •
A minimum bounding box area threshold was enforced after resizing all images. Specifically, we filtered out bounding boxes smaller than 500 pixels to ensure sufficient spatial resolution for the model.
- •
To ensure depth variation and avoid trivial samples, only object classes with at least 3 distinct depth bins (i.e., meaningful distribution over depth) were retained.
- •
For each label, depth bins were required to have a minimum of 10 and a maximum of 30 image samples. We removed bins with insufficient data to meet this requirement and capped those with excess samples by sorting instances based on bounding box area.
- •
After filtering, we retained 45 object classes, resulting in 4.4K unique image-object pairs. The dataset was split into 80% train and 20% test, preserving label and bin balance.
- •
The question for each pair: “From the camera’s perspective, estimate how far the closest point of the cabinet (highlighted by a red box) is from the camera in real-world distance, in meters. Select the best answer from the options below: A. 1-2, B. 2-3, C. 3-4, D. 4-5, E. 5-6, F. 6-7”.

The preprocessing steps for outdoor dataset creation are summarized below:

- •
To ensure the depth estimation task remains non-trivial, only objects with a minimum distance of 8 meters from the camera were considered. This avoids bias toward near-field predictions and better evaluates model precision in far-range perception.
- •
Depth values were discretized into bins, and for each class, we selected between 20 and 60 samples per bin to ensure coverage while avoiding overrepresentation. Bins with fewer than 20 samples were discarded. When bins exceeded 60 samples, selection was sorted by bounding box area to prioritize larger, more reliable objects.
- •
Image crops were extracted per object while maintaining the following aspect ratio constraint to preserve visual consistency: the width of the crop must be within the range $[\text{height},2\times\text{height}]$. This was enforced using the original image dimensions before padding or resizing.
- •
Objects touching any edge of the image were excluded to reduce the likelihood of partial occlusion or clipping.
- •
Images were padded vertically as needed to preserve the original aspect ratio.
- •
The final outdoor absolute depth set contains approximately 6K samples. For each object class and depth bin, an 80/20 train-test split was applied to maintain consistency in evaluation.
- •
The question for each outdoor sample: “Estimate the distance from the camera to the closest part of the cyclist (highlighted by a red box) in meters. Choose the best option: A. 8-9, B. 10-11,…, H. 30-31.”

Figure: Figure 21: Examples of Absolute Depth AVA Samples.
Refer to caption: x19.png

Relative Depth xia2024large; mi2024hierarchical; zhang2025vision.
We curate 11.6K image-object pairs from NYU-Depth V2 silberman2012indoor for indoor scenes and KITTI geiger2013vision for outdoor scenes, targeting the task of identifying which of two objects in an image is closer to the camera. Each image contains two distinct objects, each annotated with a bounding box. The model is asked to compare their absolute depth and choose the object that appears closer to the camera ([Figure 22](https://arxiv.org/html/2506.09082v2#A1.F22)).

The preprocessing steps for indoor dataset creation are summarized below:

- •
Ambiguous object categories were excluded via a manually curated list of label IDs that often lack clear boundaries or meaningful depth interpretations (e.g., wall, floor, ceiling, etc.).
- •
Only object pairs with valid bounding boxes (i.e., non-overlapping, fully inside image boundaries) were considered.
- •
To ensure perceptual clarity, candidate object pairs were filtered by requiring an absolute depth difference of at least 0.5 meters between them.
- •
For a given object class, only those with at least 10 valid pairings were retained, and a maximum of 30 total pairings per label were allowed.
- •
After filtering and sampling, we retained 131 object pairs across 5.7K total questions. Images were padded vertically as needed to preserve their original aspect ratio. These were split into 80% train and 20% test sets while maintaining object label and depth-difference balance.
- •
Each question is posed as: “Estimate the real-world distances between the objects in this image. Which object is closer the camera, the sink (highlighted by a red box) or the towel (highlighted by a blue box) to the camera? Choose one option from below: 1. red, 2. blue”.

The preprocessing steps for outdoor dataset creation are summarized below:

- •
Object annotations were sourced for the following categories: Car, Van, Pedestrian (merged with Person sitting), and Cyclist.
- •
Pairs were formed using both intra-class (e.g., Car vs. Car) and inter-class (e.g., Car vs. Pedestrian) combinations.
- •
Pairs were retained only if the depth difference between the two objects was at least 0.5 meters, ensuring a meaningful perceptual gap.
- •
To avoid ambiguity and incomplete visual evidence, the following filters were applied:
–
Pairs with occluded objects were excluded.
–
Pairs where either object was touching the image edge were discarded.
–
Only crops satisfying the aspect ratio constraint $\text{height}\leq\text{width}\leq 2\times\text{height}$ were included, ensuring visual consistency.
–
Images were padded vertically as needed to preserve their original aspect ratio.
- •
After filtering, approximately 6K valid image-object pairs were retained. For each pair type, an 80/20 train-test split was applied while maintaining distributional balance over object categories and depth separations.
- •
Each question is posed as: “Which object is closer to the camera, the van (highlighted in red) or the cyclist (highlighted in blue)? Choose one: 1. red, 2. blue.”

Figure: Figure 22: Examples of Relative Depth AVA Samples.
Refer to caption: x20.png

## Appendix B Experiment Details

The hyperparameters of Stage-1 vision-language alignment pretraining, Stage-2 visual instruction
tuning and Stage-3 AVA-Bench evaluation are show in [Table 3](https://arxiv.org/html/2506.09082v2#A2.T3).

**Table 3: Hyperparameters of TinyLLaVa and AVA-Bench Evaluation Stage**
|  | TinyLLaVa | AVA-BENCH |  |
| --- | --- | --- | --- |
| Hyperparameter | Stage 1 | Stage 2 | Stage 3 |
| batch size | 16 | 4 | 4 |
| grad accum steps | 4 | 8 | 1 |
| LR | 1e-3 | 2e-5 | 1e-4 |
| LR schedule | cosine decay | cosine decay |  |
| LR warmup ratio | 0.03 | 0.03 |  |
| weight decay | 0 | 0 |  |
| epoch | 1 | 10 (20 for localization AVA) |  |
| optimizer | AdamW | AdamW |  |
| DeepSpeed stage | 3 | 3 |  |
| components finetuned | Connector | Connector + LLM | Connector + LoRA on LLM |
| sample size | 558K | 665K | – |

### B.1 Metric Details

##### Color Recognition.

We use CIEDE2000 luo2001development to calculate the color differences using the colour Python library. Specifically, we convert the predictions and ground-truths from CIE XYZ tristimulus format to CIE L*a*b* colour space with colour.XYZ_to_Lab, followed by the colour.delta_E(pred, gt, method="CIE 2000") for color differences.

##### Absolute Depth & Counting.

We use the mean absolute error relative to the ground-truth (see [Equation 1](https://arxiv.org/html/2506.09082v2#A2.E1)). This normalization ensures that errors involving greater distances or counts, which are inherently more challenging, are proportionally penalized less severely.

$$ $\mathrm{MAE/GT}\;=\;\frac{1}{N}\sum_{i=1}^{N}\frac{\bigl{|}\,y_{i}-\hat{y}_{i}\,\bigr{|}}{y_{i}}.$ (1) $$

where N is the number of test samples, $y_{i}$ is the ground-truth (depth or count) for sample $i$ and $\hat{y}_{i}$ the model prediction.

##### Localization.

We use the Generalized Intersection-over-Union (GIoU rezatofighi2019generalized, [Equation 2](https://arxiv.org/html/2506.09082v2#A2.E2)):

$$ $\mathrm{GIoU}(A,B)=\frac{|A\cap B|}{|A\cup B|}\;-\;\frac{\bigl{|}\,C\setminus(A\cup B)\bigr{|}}{|C|}\,.$ (2) $$

where $A$ and $B$ are the prediction and ground-truth bounding boxes and $C$ is the smallest (axis–aligned) enclosing box of $A\cup B$.

##### OCR.

We evaluate OCR performance with Average Normalized Levenshtein Similarity (ANLS) biten2019scene:

$$ $\displaystyle\mathrm{NLS}(p,g)$ $\displaystyle=1-\frac{\mathrm{Lev}(p,g)}{\max\bigl{(}|p|,|g|\bigr{)}},$ (3) $\displaystyle\mathrm{ANLS}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathrm{NLS}(p_{i},g_{i})=\frac{1}{N}\sum_{i=1}^{N}\Biggl{(}1-\frac{\mathrm{Lev}(p_{i},g_{i})}{\max\bigl{(}|p_{i}|,|g_{i}|\bigr{)}}\Biggr{)}.$ $$

where $\mathrm{Lev}(p,g)$ is the (Levenshtein) edit distance and $|.|$ denotes string length and N is the number of testing samples.

##### Others.

All other AVAs employ standard accuracy metrics.

## Appendix C More Results and Analyses

### C.1 Detailed Overall Results

The results used for plotting Figure 7 and Figure 8 are presented in [Table 4](https://arxiv.org/html/2506.09082v2#A3.T4) where the best performance for each is AVA is bold and the second best is in italics. For each VFM, the first row is the performance, and the second row is the rank.

**Table 4: The detailed overall results where the best performance for each is AVA is bold and the second best is in italics. For each VFM, the first row is the performance, and the second row is the rank. Arrows indicate whether lower ($\downarrow$) or higher ($\uparrow$) is better.**
| AVA | Abs. Depth | Rel. Depth | OCR | Counting | Localization | Object | Fine-grained | Scene | Action | Spatial | Emotion | Orientation | Texture | Color | Average<br>Ranking |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metric | MAE/GT$\bm{\downarrow}$ | ACC$\bm{\uparrow}$ | ANLS$\bm{\uparrow}$ | MAE/GT$\bm{\downarrow}$ | GIOU$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | ACC$\bm{\uparrow}$ | CIEDE2000$\bm{\downarrow}$ |  |
| SigLIP-2 | 0.0843 | 97.38 | 81.18 | 0.225 | 0.6738 | 88.19 | 90.17 | 72.60 | 45.33 | 99.17 | 58.39 | 79.2 | 94.08 | 12.25 | 2.4 |
| 3 | 2 | 1 | 1 | 1 | 1 | 3 | 3 | 2 | 4 | 5 | 4 | 2 | 2 |  |  |
| AIMv2 | 0.1008 | 95.71 | 62.44 | 0.254 | 0.5896 | 85.04 | 91.78 | 72.86 | 43.61 | 98.93 | 60.68 | 79.71 | 94.11 | 19.61 | 3.9 |
| 9 | 8 | 3 | 3 | 5 | 4 | 1 | 2 | 3 | 5 | 1 | 3 | 1 | 6 |  |  |
| SigLIP-1 | 0.0953 | 96.58 | 80.3 | 0.229 | 0.6103 | 87.84 | 90.94 | 73.4 | 45.76 | 98.66 | 59.6 | 78.91 | 93.66 | 12.64 | 3.6 |
| 6 | 7 | 2 | 2 | 4 | 2 | 2 | 1 | 1 | 8 | 4 | 5 | 3 | 3 |  |  |
| CLIP | 0.08461 | 97.04 | 60.44 | 0.290 | 0.5787 | 85.02 | 86.83 | 72.32 | 41.59 | 99.28 | 60.42 | 77.77 | 93.25 | 19.85 | 4.9 |
| 4 | 4 | 6 | 7 | 7 | 5 | 4 | 5 | 4 | 3 | 2 | 6 | 5 | 7 |  |  |
| InternVL-2.5 | 0.08212 | 97.00 | 60.88 | 0.269 | 0.5850 | 83.19 | 72.83 | 71.63 | 36.13 | 99.31 | 59.99 | 75.71 | 91.21 | 20.09 | 5.4 |
| 2 | 5 | 5 | 5 | 6 | 7 | 7 | 6 | 7 | 2 | 3 | 7 | 7 | 8 |  |  |
| RADIOv2.1 | 0.07645 | 97.92 | 62.44 | 0.257 | 0.6617 | 84.90 | 85.44 | 72.50 | 38.77 | 99.48 | 56.59 | 83.71 | 93.32 | 17.16 | 3.6 |
| 1 | 1 | 4 | 4 | 2 | 6 | 6 | 4 | 5 | 1 | 6 | 2 | 4 | 5 |  |  |
| DINOv2 | 0.08469 | 97.25 | 9.97 | 0.272 | 0.6598 | 86.31 | 85.5 | 70.99 | 37.45 | 98.90 | 54.44 | 85.54 | 93.06 | 21.54 | 5.3 |
| 5 | 3 | 7 | 6 | 3 | 3 | 5 | 7 | 6 | 6 | 7 | 1 | 6 | 9 |  |  |
| SAM | 0.09792 | 94.13 | 9.79 | 0.313 | 0.5216 | 76.68 | 40.06 | 58.28 | 17.25 | 98.76 | 36.88 | 69.37 | 81.02 | 9.87 | 7.8 |
| 8 | 9 | 8 | 8 | 8 | 8 | 8 | 9 | 9 | 7 | 9 | 8 | 9 | 1 |  |  |
| MiDas-3.0 | 0.09563 | 96.63 | 7.72 | 0.336 | 0.4490 | 75.05 | 32.83 | 60.19 | 18.25 | 96.86 | 40.40 | 67.37 | 86.38 | 13.28 | 8 |
| 7 | 6 | 9 | 9 | 9 | 9 | 9 | 8 | 8 | 9 | 8 | 9 | 8 | 4 |  |  |

### C.2 Detailed Analyses for Each AVA

In the main results, we reported the overall performance of VFMs across various AVAs. However, aggregate metrics can sometimes obscure important nuanced insights. To gain deeper understanding, we conduct detailed analyses by partitioning test samples based on specific criteria (e.g., object size in localization tasks) and examining whether these subgroup trends align with overall performance. Generally, the detailed analyses affirm overall trends, but notable exceptions exist, particularly in localization.

##### Localization.

We split localization testing samples based on normalized bounding box sizes (relative to image size), where 0.1 indicates an object occupies 10% of the image area. As illustrated in [Figure 10](https://arxiv.org/html/2506.09082v2#S5.F10), VFMs surprisingly exhibit minimal performance differences when localizing large objects (0.3–0.5). Conversely, performance disparities amplify as object size decreases, revealing significant weaknesses in MiDas and SAM for smaller objects. Consequently, the lower overall performance of MiDas and SAM is predominantly due to poor handling of small targets. Practitioners should thus consider object size distributions when selecting VFMs; SAM and MiDas remain viable if target objects are predominantly large .

##### Counting.

Counting performance is generally consistent across different datasets and count ranges ([Figure 23](https://arxiv.org/html/2506.09082v2#A3.F23)). A notable exception is SAM, whose accuracy notably improves in denser scenarios.

Figure: Figure 23: Detail results for counting for overall and different splits based on datasets and ground-truth count range. Lower MAE/GT is better.
Refer to caption: x21.png

##### Emotion.

Emotion recognition results exhibit remarkable consistency, with rankings and relative performances highly stable across emotion categories (see [Figure 24](https://arxiv.org/html/2506.09082v2#A3.F24)).

Figure: Figure 24: Detail results for emotion for overall and different splits based on emotion types.
Refer to caption: x22.png

##### Orientation.

Orientation performance remains consistent overall, with some intriguing exceptions. Specifically, VFMs universally achieve near-perfect accuracy for top-view images, presumably due to the distinctive nature of this viewpoint compared to side or frontal views (see [Figure 25](https://arxiv.org/html/2506.09082v2#A3.F25)).

Figure: Figure 25: Detail results for orientation for overall and different splits based on viewpoint directions.
Refer to caption: x23.png

##### Absolute Depth.

Overall, absolute depth performance is stable, though specific VFMs exhibit distinctive trends. SAM notably struggles with near objects but improves significantly with increased distance. Conversely, RADIO demonstrates an opposite pattern, excelling with nearer objects but deteriorating with greater distances (see [Figure 26](https://arxiv.org/html/2506.09082v2#A3.F26)).

Figure: Figure 26: Detail results for absolute depth for overall and different splits based on scene type and object count range. Lower MAE/GT is better.
Refer to caption: x24.png

##### Relative Depth.

Relative depth estimation shows uniformly high performance across VFMs, consistently surpassing 90% accuracy. SAM, however, underperforms notably in interior scenes, consistent with the earlier observation in absolute depth and counting that SAM handles smaller, exterior objects better (see [Figure 27](https://arxiv.org/html/2506.09082v2#A3.F27)).

Figure: Figure 27: Detail results for relative depth for overall and different splits based on scene type.
Refer to caption: x25.png

##### Fine-grained Classification.

Fine-grained classification results are consistently robust across datasets, with the exception of SAM and MiDas, both of which are known to lack semantically rich features espinosa2024there; chen2023segment, resulting in poorer performance (see [Figure 28](https://arxiv.org/html/2506.09082v2#A3.F28)).

Figure: Figure 28: Detail results for fine-grained for overall and different splits based on dataset type.
Refer to caption: x26.png

##### Scene Recognition.

Scene recognition performance is uniformly consistent across all evaluated datasets, echoing the patterns observed in fine-grained classification, where SAM and MiDas again lag behind other VFMs (see [Figure 29](https://arxiv.org/html/2506.09082v2#A3.F29)).

Figure: Figure 29: Detail results for scene for overall and different splits based on datasets.
Refer to caption: x27.png

##### OCR.

OCR results show consistent patterns with those reported in Section 5.2, highlighting significant underperformance by non-language-aligned VFMs, such as DINOv2 and SAM. Notably, we observe that relative performances across VFMs are stable on short texts (length $<$ 20). However, performance for CLIP and AIM sharply declines with longer text sequences (length $>$ 20), indicating potential limitations in handling extensive textual information (see [Figure 30](https://arxiv.org/html/2506.09082v2#A3.F30)).

Figure: Figure 30: Detail results for counting for overall and different splits based on dataset domain and character length. Higher ANLS is better.
Refer to caption: x28.png

## Appendix D Related Works

### D.1 VFM Evaluation

Existing evaluation VFM protocols generally fall into two categories. The first focuses on task-specific capabilities, typically attaching tailored heads to VFMs, followed by fine-tuning and evaluation on dedicated datasets such as ImageNet for classification han2022survey and COCO for detection or segmentation thisanke2023semantic. For example, DINOv2 oquab2023dinov2 is evaluated on image and video classification, instance recognition, image retrieval, semantic segmentation, and depth estimation. For each task, a task-specific head is trained.

To better capture the diverse and complex perception challenges of the real world, recent studies advocate a more generic approach that leverages large language models (LLMs) as general-purpose heads, evaluating VFMs on broad Visual Question Answering (VQA) benchmarks liu2023visual; zhu2023minigpt; chowdhery2023palm. For example, in addition to the traditional task-specific evaluation, AIMv2 fini2024multimodal and RADIO  ranzinger2024radio follow the LLM-based evaluation and use a Llama-3.0(8B) grattafiori2024llama and a Vicuna-1.5(7B) zheng2023judging respectively, demonstrating a shift towards generalized multimodal evaluation.

### D.2 Atomic Visual Abilities

As discussed in [subsection A.1](https://arxiv.org/html/2506.09082v2#A1.SS1), foundational visual concepts—such as number, color, texture, object identity, and spatial relations—have long been recognized as crucial building blocks in compositional Text-to-Image (T2I) benchmarks. Given their foundational role in generation tasks, these primitives naturally underpin perceptual tasks as well. For example, a concept like ’number’ directly translates into the perceptual task of counting.

A recent work chae24decomposing introduced AVSBench to evaluate whether MLLMs understand basic geometric features, including angle, boundary, orthogonality, and curvature, which they refer to as atomic visual skills. However, AVSBench primarily targets geometric comprehension abilities required for geometric diagrams arising in high-school level mathematics. Moreover, AVSBench provides only test data for MLLMs without addressing potential mismatches between training and test data distributions—an issue highlighted in Section 2.2. Consequently, mispredictions in AVSBench evaluations may result from data distribution mismatches rather than genuine visual deficiencies in VFM. In contrast, AVA-Bench explicitly emphasizes atomic visual abilities essential for general visual reasoning tasks commonly encountered in real-world scenarios. By aligning training and evaluation data distributions, AVA-Bench ensures that evaluation outcomes reliably reflect genuine visual perceptual capabilities of VFMs.

Additionally, a concurrent work wu2025compact defines a set of atomic visual capabilities analogous to ours. However, their goal fundamentally differs from ours: wu2025compact aims to build a visual compositional tuning data recipe that builds complex capabilities from simple atomic capabilities, which can significantly reduce instruction-tuning data volume while maintaining strong performance. In contrast, AVA-Bench’s objective is to systematically evaluate VFMs against atomic visual abilities, pinpointing their exact strengths and weaknesses, and providing a comprehensive diagnostic tool to advance the continual development mai2022online; lomonaco2022cvpr; mai2021supervised; shim2021online; mai2020batch of robust vision foundation models.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: There is another paper with a similar motivation that presents a dataset named "Atomic Visual Skill Bench" while this paper is titled "Atomic Visual Ability Bench", essentially the same name. When I first saw the title, I admittedly suspected possible plagiarism, but after reading the paper, I found no actual impropriety.

That said, using an almost identical name creates an appearance of impropriety, and I question whether this naming choice was prudent.
soundness: 3
strengths: This work introduces a framework for evaluating Vision Foundation Models (VFMs) at the level of atomic perception capabilities. Through the 14 Atomic Visual Abilities (AVAs), such as localization, spatial reasoning, and color recognition,  a diagnostic benchmark that can identify the exact source of a model’s strengths or weaknesses is provided.
summary: This work introduces a benchmark that evaluates Vision Foundation Models (VFMs) through 14 Atomic Visual Abilities (AVAs) that serve as the building blocks of visual reasoning. Unlike traditional Visual Question Answering (VQA) benchmarks that mix multiple skills, AVA-Bench isolates each ability using over 218K curated samples from 26 datasets, providing clear diagnostic insight into model strengths and weaknesses.

Across leading VFMs, language-supervised models like SigLIP and AIMv2 show broad competence, while self-supervised models like DINOv2 excel in geometric perception. Most VFMs handle low-level tasks well, but struggle with specific abilities such as small-object localization or OCR. By quantifying these atomic abilities this work provides an effective evaluation and selection of VFMs for downstream multimodal systems.
weaknesses: The deficiencies of vision foundation models that this work points out are well known. That is, vision models often struggle with precisely perceiving basic information despite being able to perceive higher-level objects well.

Also, the 14 atomic visual abilities seem to have been chosen somewhat arbitrarily. As an example, "motion or temporal understanding" could conceivably be an entry in the list of atomic visual abilities, but it is not included. Also, it is questionable whether "emotion" is an "atomic" ability.

While it seems that this benchmark is measuring useful abilities, it is unclear whether this benchmark must exist. While it is true that existing VQG benchmarks often require multiple visual abilities, perhaps the existing benchmarks are sufficient to drive progress in VFM, and the need to have a disentangled "atomic" benchmark is not clear to me.

I believe a good benchmark should measure something that existing benchmarks do not, but existing VQA benchmarks measure a superset of abilities that this benchmark is measuring.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 4
questions: 1. Have the authors evaluated VFMs using simpler non-LLM decoders (e.g., linear probes or task-specific MLPs) to check whether the LLM-based head is introducing bias or performance bottlenecks—especially for AVAs like texture, where language grounding is not required? If so, do any contradictory trends emerge compared to the LLM-based evaluation?
soundness: 4
strengths: 1. The benchmark isolates 14 atomic visual abilities from standard VQA tasks, enabling diagnostic-level insight into VFM performance.
2. It aggregates 218K image-question pairs curated from 26 datasets.
3. It reveals ability-specific performance "fingerprints" across diverse VFMs (e.g., SigLIP, AIMv2, DINOv2, SAM).
4. Demonstrates that 0.5B LLMs retain VFM ranking accuracy while cutting GPU cost by 8× compared to 7B models.
summary: This paper introduces AVA-BENCH, a novel benchmark aimed at evaluating Vision Foundation Models (VFMs) based on 14 Atomic Visual Abilities (AVAs), such as localization, counting, depth estimation, orientation, and others. Unlike traditional VQA-based evaluation, which conflates multiple abilities and suffers from dataset misalignment, AVA-BENCH isolates and evaluates each perceptual skill independently. The framework uses a lightweight LLM (0.5B) as the multimodal head and fine-tunes it with LoRA over VFM features to measure performance on each AVA.

The study evaluates a diverse suite of VFMs (DINOv2, CLIP, SigLIP, SAM, MiDaS, AIMv2, etc.), revealing model-specific strengths and weaknesses across abilities, offering actionable insights for model selection. A key finding is that smaller VFM-LLM combinations can preserve relative performance rankings, significantly reducing computation without sacrificing fidelity.
weaknesses: 1. **Lack of Connector Ablations**: A core component of the AVA-BENCH protocol is the LLM-based evaluation pipeline that attaches a lightweight language model to each VFM via a connector, trained using LoRA. However, the paper does not provide a systematic exploration or justification of the connector design choices, such as the LoRA rank, or whether certain VFMs require different connector hyperparameters to retain their perceptual strengths post-alignment. Since AVA-BENCH aims to serve as a generalizable diagnostic tool across all VFMs, it is critical to establish that connector-induced artifacts do not bias the evaluation. Without ablations on connector design, it is unclear whether performance drops are due to inherent VFM limitations or insufficient preservation of the VFM’s features by the connector. This uncertainty weakens confidence in the benchmark's comparative fidelity. For AVA-BENCH to be replicable and broadly valid, connector components and their training strategies (e.g., LoRA rank, learning rate, frozen depth, etc.) must be validated and stress-tested across diverse VFMs. Otherwise, one risks conflating the alignment's shortcomings with actual perceptual deficiencies of VFMs, leading to misleading performance fingerprints.

2. **Lack of Correlation Analysis with Standard VQA Benchmarks**: While AVA-BENCH offers granular diagnostic insights by decoupling visual abilities into 14 atomic dimensions, the paper does not investigate how these isolated ability scores correlate with performance on more complex, real-world VQA benchmarks. This limits our understanding of whether strong performance on individual AVAs (e.g., spatial reasoning or OCR) meaningfully translates into improved results on holistic tasks that require composition of multiple abilities, such as counting objects with specific attributes or relating text to visual cues in context. Without demonstrating such correlations, it is unclear if AVA-BENCH is predictive of real-world VQA competence or merely measuring isolated skills in a vacuum. 

3. **Similarity to Prior Work (Novelty Concerns):** A significant limitation of the paper lies in its conceptual and empirical overlap with prior diagnostic studies of VFMs. Notably, many of the main findings in AVA-BENCH—such as the superior general performance of SigLIP-based models, the consistent baseline competence of CLIP variants, or the strong vision-only performance of DINOv2—have been previously documented in Cambrian. This raises questions about the extent of empirical novelty provided by the benchmark beyond revalidating known results in a more fine-grained setup. How does AVA training compare to FastVLM [1]?

[1] Vasu, P. K. A., Faghri, F., Li, C. L., Koc, C., True, N., Antony, A., ... & Pouransari, H. (2025). Fastvlm: Efficient vision encoding for vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 19769-19780).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See above
soundness: 3
strengths: 1. The problem tackled—how to fairly, transparently, and diagnostically evaluate VFMs—is a central challenge in the current multimodal and computer vision fields. The authors' analysis of the two blind spots in existing VQA evaluation is spot-on.

2. The core idea of AVA-BENCH ("ability decomposition") is excellent. Decomposing complex visual tasks into 14 atomic abilities and providing independent, distribution-matched datasets for each is a highly systematic and diagnostic approach. It shifts VFM evaluation from "what score does it get?" to "what is it good at, and what is it bad at?".

3. The paper covers VFMs from diverse training paradigms (e.g., language-supervised, self-supervised, segmentation-supervised), making the experiments very comprehensive. The analysis (as seen in Figures 5 & 6) goes beyond reporting numbers to truly reveal the "ability fingerprints" and trade-offs between different VFMs, such as DINOv2's strength in vision-centric tasks versus SigLIP's versatility.
summary: This paper addresses a prevalent and critical problem in the evaluation of Vision Foundation Models (VFMs). Current evaluation methods typically rely on pairing VFMs with Large Language Models (LLMs) and testing them on general-purpose Visual Question Answering (VQA) benchmarks. The authors astutely identify two key blind spots in this approach: Data Distribution Mismatch and Ability Entanglement. To address these issues, the paper introduces AVA-BENCH, an innovative benchmark that explicitly disentangles complex visual tasks into 14 Atomic Visual Abilities (AVAs), such as localization, counting, depth estimation, spatial understanding, OCR, and more.
weaknesses: 1. The paper selects 14 AVAs. Although the authors explain their origins in the appendix (based on literature and VQA analysis), the "completeness" or "orthogonality" of this set remains open for discussion. For instance, why these 14? Are other critical atomic abilities missing? A more in-depth discussion of the selection criteria for this set would make the paper more persuasive.

2. The 14 AVAs vary in their level of "atomicity." For example, "color recognition" or "texture recognition" are arguably more "atomic" than "action recognition" or "fine-grained recognition." The latter (like action recognition) may inherently involve multiple implicit abilities such as localization and pose estimation. The paper could more clearly discuss these differences in complexity and why these "more complex" tasks are still classified as "atomic."

Minor suggestion: The main paper lacks a whole table to present the detailed evaluation results.

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. It remains unclear whether VFMs possess genuine reasoning abilities or can contribute meaningfully to reasoning-related tasks.
2. AVA-Bench relies entirely on an LLM as a universal prediction head. Although results show that a smaller 0.5B LLM (Qwen2) can replace a 7B LLM (Vicuna-1.5) to reduce cost, the scaling behavior and impact on VLM performance remain insufficiently explored.
3. The paper should more clearly explain the differences between this work and general VLM benchmarks, such as MMMU and MMStar, which would help highlight and strengthen its contributions.
soundness: 2
strengths: 1.	The paper first proposes a novel evaluation approach that decomposes visual tasks into Atomic Visual Abilities (AVAs).
2.	AVA-Bench covers 14 AVAs, ranging from low-level texture recognition to high-level spatial reasoning and emotion recognition, offering a broad evaluation scope. The dataset is constructed with attention to diversity and balance, ensuring reliable and fair assessment results.
3.	By revealing the performance of different VFMs across various AVAs, the benchmark helps MLLMs select the most suitable VFM based on specific task requirements.
summary: The paper introduces AVA-Bench, an innovative and practical benchmark designed to evaluate the Atomic Visual Abilities (AVAs) of Visual Foundation Models (VFMs). This tool fills a key gap in existing evaluation methods by decomposing complex visual tasks into fundamental abilities, providing a clear view of VFMs’ strengths and weaknesses. The study is well-designed, and its findings offer valuable guidance for improving future VFMs.
weaknesses: Non-language-aligned VFMs (e.g., DINOv2, SAM) may suffer from visual information loss during alignment with language modalities, causing evaluation results to underrepresent their true visual capabilities. For example, in fine-grained recognition tasks, DINOv2’s accuracy drops sharply from 66.3% (pre-connector) to 25.67% (post-connector), indicating that the alignment process can degrade critical visual representations.