Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
Authors: Hongtao Yu
Yuxin Peng
Serge Belongie
Xiu-Shen Wei
Venue: ICLR 2026 Poster (2026)
Keywords: Empirical Study
Large Vision-Language Model
Benchmark
Evaluation
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal perception capabilities, garnering significant attention. While numerous evaluation studies have emerged, assessing LVLMs both holistically and on specialized tasks, fine-grained image tasks—fundamental to computer vision—remain largely unexplored. To fill this gap, we introduce a comprehensive fine-grained evaluation benchmark, i.e., FG-BMK, comprising 1.01 million questions and 0.28 million images. Our evaluation systematically examines LVLMs from both human-oriented and machine-oriented perspectives, focusing on their semantic recognition and fine-grained feature representation capabilities. Through extensive experiments on twelve representative LVLMs/VLMs, we uncover key findings regarding the influence of training paradigms, modality alignment, perturbation susceptibility, and fine-grained category reasoning on task performance. This work provides critical insights into the limitations of current LVLMs and offers guidance for future data construction and model design in the development of more advanced LVLMs. Our code is open-source and available at https://github.com/SEU-VIPGroup/FG-BMK.

Published as a conference paper at ICLR 2026
BENCHMARKING LARGE VISION-LANGUAGE MODELS ON FINE-GRAINED IMAGE TaSks: A COMPREHENSIVE EVALUATION
Hong-Tao Yu', Yuxin Peng", Serge Belongie", Xiu-Shen Wei**
1 School of Computer Science and Engineering and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China Wangxxan Institute of Computer Technology. Peking University, China 3 University of Copenhagen, Denmark
ABSTRACT
Recent advancements in Large Vision- SaiionLanguage Models (LVLMs) have demon- strated remarkable multimodal perception capabilities, garnering significant at- tention. While numerous evaluation studies have emerged, assessing LVLMs both holistically and on specialized tasks, fine-grained image tasks-fundamental to computer vision-remail largely unexplored. To fill this gap, introduce a comprehensive fine-grained evaluation benchmark, i.e., FG- BMK, comprising 1.01 million questions and 0.28 million images. Our evaluation systematically oriented and maachine-oriented perspectives,
examines LVLMs from both focusing their semantic recognition and enaiinaieed feature representation ca- pabilities. Through extensive experiments on twelve representative L.L.Ms/VLMs, we uncover key findings regarding the influence of training paradigms, modality alignment, perturbation susceptibility, and fine-grained category reasoning on task performance. This work provides critical insights into the limitations of current LVLMs and offers guidance for future data construction and model design in the development of more advanced LVLMs. Our code open-source and available httpp://g wtt.:a/o/////OOGGG..T.... WEw--VIGGoop/EEEEEEEEE BMK
INTRODUCTION
Large language models have recently achieved remarkable advancements, with models like GPT- 4 (OpenAi 2023) surpassing human-level performance across diverse tasks. Building on this, Large SsiionLanguage Models (LVLMs) have rapidly evolved. with models such as GPT-4o (OpenAl
2023). Qwen (Bai al., 2023). InternVL (Chen al. 2024), and VA-1.5 (Liu al. 2024a) demonstrating strong multmodal reasoning and perception capabilities.
In response to these advancements, various holistic and specialized evaluations have emerged to assess the capabilities of LVLMs. For instance, LVLM-eHub (Xu et al. (2025) and MMBench (Yuan) al., 2024) provide broad evaluations of multimodal perception and reasoning, while specialized evaluations like VQA (Mathew al. 2021) and GQA (Hudson & Manning. 2019) focus on specific tasks, such as document visual perception and visual reasoning. While these evaluations have provided valuable insights, critical gap remains: there is no systematic evaluation of LVLMs fine-grained image tasks, which focus on analyzing visual objects at the subordinate category level and are fundamental computer vision (Wei et al. 2022b). Consequently, the capability boundaries of LVLMs in fine- -raiged tasks remain poorly understood.
* *Corresponding author. This work was supported by National Natural Science Foundation of China under Grant (62522602, 62272231, 62525201, 62132001, 62432001), Basic Research Program of Jiangsu under Grant (BK20250073), Beijing Natural Science Foundation (L247006), the Pioneer Centre for AI. DNRF grant number P1, the Fundamental Research Funds for the Central Universities (4009002401, 2242025K33024), and the Big Data Computing Center of Southeast University.

---

Published as a conference paper at ICLR 2026
To address this gap, propose comprehensive fine-grained evaluation benchmark, termed FG-BMK. This benchmark includes 1.01 million questions and 0.28 million images, providing a robust test bed for evaluating LVLMs. FG-BMK incorporates two evaluation paradigms: human-oriented and machine-oriented. The human-oriented evaluation employs dialogue-like interactions to assess a model's ability understand and respond fine-grained visual queries in conversational context. The machine-oriented evaluation focuses on two core fine-grained vision tasks tasks-image retrieval and recognition- directly measure the feature representation capabilities of LVLMs. Together, these evaluations enable a comprehensive assessment of LVLMs' fine tine-grained feature representation and semantic recognition abilities.
Through extensive evaluations across representative LVLMs and VLMs within FG-BMK, we derive several key insights:
The contrastive training paradigm in LVLMs proves more effective in enhancing the fine- tne-grained discriminability of visual features, whereas generative and reconstruction-bant training paradigms tend to yield weaker discriminability.
Aligning visual features with textual features in LVLMs can impair their fine-grained discrim- inability, particularly when there mismatch in granularity between the paired image-text data.
LVLMs and LVMs are more vulnerable to feature perturbations in fine-graing mne-grained tasks than in generic vision tasks.
LVLMs demonstrate relatively stronger capabilities in perceiving visual appearances but face chal- lenges in fine-grained category reasoning (which depends on the recognition of visual attributes).
Despite their advancements, LVLMs still lag behind fine-grained tailored models in handling fine-grained visual tasks.
These findings provide foundation for future research aimed at advancing LVLLM performance in fine-grained tasks and addressing the challenges uncovered by CGMMM.
2 RELATED WORK
Large Vision- D'sgua-Lnnguage Models Large Language Models (LLMs) like GPT-4 (OpenAl 2023) have made notable strides in text comprehension, reasoning, and generation. Building on this foundation, Large Vision-Language Models (LVLMs)) have emerged with impressive reasoning and perception abilities across diverse tasks. Enhancing LVLM performance has taken various directions: BLIP (Li (et al. 2022) uses noisy web data with bootstrapped captions; LLaVA (Liu et al., 2024a) employs G-n---generrted instruction-folloviin data; BLIP-2 (Li et al. 2023 integrates frozen image and text encoders; Qwen2.5-VL (Bai et al. 2023) leverages dynamic-
dynamic-resolutien resolution strategy for preserving native image resolution; Intern VL3 (Zhu et al. (025) unifies pre-training over multimodal data for improved effectiveness. While achieving success on many tasks, fine- grained visual challenges for these LVLMS remain underexplored. This work addresses this gap by assessing their performance in fine-grained domains, uncovering both their potential and limitations.
Large Vision-Language Model Benchmarks While LVLMs have shown impressive performance, various benchmarks have been developed to evaluate their capabilities in different domains. Spe- cialized benchmarks like ChartQA (Masry al. 2022) for chart understanding, GQA (Hudson & Manning 2019) for visual reasoning, and CAPability (Liu et al. 2025) for caption evaluation provide focused assessments. Additionally, evaluations in optical character recognition (Liu al.. 2024c) and adversarial robustness (Madry et al., 2018) offer insights into specialized aspects of LVL. performance. Holistic evaluations, such as LVLM-eHub (Xu al. (025)) and MMBench (Yuan et
2024), assess general multimodal perception and reasoning, while others like Vista (Lu et al. 2024 and MMMU (Yue et al., (024) include expert-level problems spanning multiple disciplines.
However, comprehensive evaluations of LVLMs in fine-grained visual tasks remain scarce. Existing efforts, such as (Geigle et al. 2024), primarily focus on fine fine-graine grained classification, and (Zhang et al.] 2024c) evaluates classification and reasoning with limited questions. To address it, propose FG-BMK, a benchmark designed to comprehensively evaluate LVLMs' fine-grained feature representation and semantic recognition capabilities across diverse visual tasks.
2

---

Published as a conference paper at ICLR 2026
Human-oriented Evaluation
Recguition
Knowledge
Granularity Recognition:
Q: eye une snower from followong
image!
Q₂ your ebsrrotion, specicx
Class the claes of the obyoct smas??
deg shown a Chilasha?? Yes
the object?
[blu,. black ... red, rufous]
Oy : Dees this dog beling to the species knowa japanese
[auklet, buntingg tesstes
Qr ls the eye color of the hird
this image black?
spenictt No
Specles: What
Type:
Do
Type
Type:
Machine-oriented Evaluation
Imuge Retrieval
Image Clavsification
Simileeit
Quary Set
Pradiieed Class Profeshiln
Gullioesun 0.75
Gullfeream
Gallery image
Gaffszeum 0.20
Guilerram
Laysar: Albatnoss 0.003
107542
Falcon 2000 0.01
Quary lmage
10.5978
GT: Guifstream
(Witii Mats-cled)
(Arroe Mett-cbeo)
Input: Image
Output: Imagee Similarity
Metrie: mAP
Input: Imager
Output: Class Probubility
Metric: Top-l Ace
Figure I: Our proposed benchmark: The human-oriented evaluation tests the model's ability to handle fine-grained visual queries (truc/false, multiple-choice, short-answer). while the machine-oriented evaluation directly assesses visual feature representation through image retrieval and classification tasks.
=true/false question, 5 =ts=pllpple-choice question, =short-answer question.
Fine-Grained Image Tasks Fine-grained visual tasks (Wii et al. 2022b
Xu et al. 2022 Wei et al. 2024b) Zhang et al. 2024b) pivotal in applications like biodiversity monitoring (Jing al. 2024b), object retrieval (Shen et al.
2024. Zhang et al. 2024a Jing et al. 2024a; Wang et al. 2024| Liu et al.
2022), and product recommendation (Wei et al. 2022a). While LV LLMs such as GPT-4, Intern VL, and Qwen excel in tasks like OCR, visual question answering, and image captioning, fine- sine-grained tasks remain a significant challenge. These tasks demand models to identify subtle, discriminative patterns in images (Song al. 2020) and leverage expert knowledge from LLMs for precise responses. To address this, we introduce comprehensive benchmark and conduct extensive experiments to evaluate LVLMs' performance fine-grained tasks. Our study systematically highlights their limitations and offers actionable insights for advancing model design and training.
THE EVALUATION BENCHMARK
3.1 EVALUATION TASKS AND METRICS
To comprehensively evaluate LVLMs, we adopt two evaluation paradigms: 1) human-oriented evaluation and 2) machine-oriented evaluation.
Specifically, since dialogue is the most direct way for humans to interact with LVLMs, the human- oriented evaluation assesses the model's ability to understand and respond to fine-grained visual queries in a conversational setting. We utilize three types of questions-ttrucffalsss multiple-choice, and short- (cf. Figure where response is considered correct if it includes the ground truth.
While, the machine-oriemae oriented evaluation encompasses two fundamental vision tasks (Wei al. 2022b): image retrieval and image classification. To evaluate the feature representation ability of LVLMs, we measure the accuracy of their visual features these tasks. Following the DINOv2 approach (Oquab) et al. 2023), we use mean Average Precision (mAP) for retrieval and Top-1 accuracy for classification.
The detailed aspects of both human-oriented and machine-oriented evaluations outlined below:
Human-oriented Evaluation:
Anribute Recognition: This includes true/false questions and multiple-choice questions, designed to evaluate the model's ability to identify visual attributes such as size, color, length, shape, pattern, and more.
3

---

Published as a conference paper at ICLR 2026
Knowledge Bias Estimation: This section consists exclusively of true/false questions, designed to uncover potential knowledge biases across different fine- ane-grained categories by evaluating LVLM's accuracy for each category.
Hierarchical Granularity Recognition: This section includes true/false questions, multiple-choice questions, and short-answer questions, designed to evaluate the LVLMs' ability to leverage domain- specific knowledge to identify object categories in images across different levels of granularity within hierarchical taxonomies.
Machine-oriented Evaluation:
Image Retrieval: retrieves images belonging multiple subordinate categories of meta-category by measuring the similarity of their visual features.
Image Classification: recognizes images into fine- one-grained categories, either within single meta- category (e.g., species of animals, models of cars) across multiple meta-categories, assessing the model's ability to handle diverse data sources and make accurate predictions.
More evaluation tasks details are presented in Appendix A.1
3.2
DATA CURATION
To ensure the quality and diversity of the data, we source images for the FG-BMP BMK benchmark from 12 well-established fine-grained datasets, which helps avoid issues inconsistent image quality and the risk of mislabeling often found in web-sourced images (Radford et al. (021).
Attribute Recognition: We design true/false and multiple-choice questions based on the fine-grained annotations of the images. For the multiple-chote questions, the options include all possible attribute candidates. In contrast, for the true/false questions, half negative samples, pairing the image with an incorrect attribute, and the other half are positive samples.
Knowledge Bias Estimation: In addition to positive samples, for each fine-grained category label, we pair it with images from other subcategories within the same super- super-category to generate negative samples for true/false questions.
Hierarchical Granularity Recognition: We design true/false, multiple-choice, and short-answer questions at various levels of granularity based on the hierarchical taxonomy labels of the images. For true/false questions, negative samples are created by pairing images with incorrect labels at the same hierarchical level (e.g., an image of Aves (birds) paired with Insecta (insects)). For multiple-choice questions, options
drawn from different categories within the same parent category of the hierarchical taxonomy (e.g., species-level options such as black-footed albatross and Laysan albatross, both within the genus Albatross). For short-answer questions, the model is prompted to generate its response directly.
Image Retrieval and Classification: We use the original labels directly from the datasets. For classification across meta-categories, we combine subcategories from different meta-categories into a unified training/testing set, where all fine-grained categories are mixed for training.
More specifically, for each task the human-oriented evaluation, we manually design several question templates to ensure both diversity and comprehensive coverage. More details of curated data can be found in Appendix A.2
OBSERVATIONS AND DISCUSSIONS
4.1 MODELS UNDER EVALUATION
Given the diverse landscape of existing LVLMs, we select nine widely-used open-source LVLMs, two closed-source models (GPT-40-1 120 OpenAl 2023) and 0ellah-2.0-flash (Gemini Team 2024) and one purely visual model with varying training strategies, as shown in Table 1 To enable clear comparisons and isolate the factors influencing performance, we use earlier versions of models from evaluation, where their distinct characteristics are evident
each family in machine-oriente and less affected by additional tricks or complex modifications. Further details about the evaluated models can be found in Appendix B
4

---

Published as a conference paper at ICLR 2026
Table I: Training strategies of the open-source evaluated models. "DINOv2" is purely visual model. "Con" denotes contrastive loss, "Gen" generative loss, "Mat" image-text matching loss, "Rec" reconstruction loss used in BEiT3, and "Dis" distillation loss used in DINOv2.
Loss Function
Training Data
Model
Vision Size
Con
Gen
Mat
Rec
Dis
< 0.IB
0.IB
1B
IB
InteraVL3--BB (Zhu et al. 2025)
VIT-L
InternVL-Chin (Chen et al. 2024
ViT-6B
LLaVA-L.5-7B (Luu ct al. 2024a
ViT-L
Owen2.5-VL-7B Ba et al 2025
VIT-600M
V
Qwen-VL--Cat (Bai et al. 2025
ViT-G
BLIP-2-XL [Li ct al. 2023)
ViT-G
Y
EVA-CLIP (Sun al. 2002
ViT-L
BEÎT3 [Wang aL 2025
VIT-L
CoCa et al. 2022)
VIT-L
DINOv2 (Oquab et al.
2023
VIT-L
4.2 HUMAN-ORIENTED EVALUATION
In the human-oriented evaluation, we assess the ability of LVLMs to understand visual content and utilize domain-specifii knowledge through conversational interactions. Specifically, we first evaluate the domain-specifie knowledge embedded in LVLMs by measuring their accuracy in identifying object categories across different levels of granularity, as illustrated in Figure 2 Then, we investigate whether LVLMs exhibit knowledge biases when recognizing different fine-graing categories by
ranking their accuracy in answering true/false questions for each category, cf. Figure [3] Additionally, we examine the capability of LVL.Ms to recognize fine-grained attributes, with results summarized in Table Lastly, Table Jable/ccompare compares the classification accuracy of LVLMs against state-of-the--a fine fine-grained tailored models on datasets spanning various domains. Based on these observations, we draw the following conclusions.
LVLMs struggle to distinguish excessively fine- grained categories. As shown Figure taking In- Ve3 (Zhu et 2025) as an example, its accuracy in answering true/false and multiple-choice questions declines the granularity becomes finer. At the class level (e.g.. "Is Insecta/Aves?"', the model achieves 99. 3,76% accuracy multiple-choice questions and 99.77% on true/false ques- tions However, as the granularity narrows the genus level, where negative categories are drawn from different genera within the same class (e.g., "Is the object in this image an albatross or gull?"), the model's 's accuracy on multiple-choice questions drops to 90.75%, a decrease of 9.01 %. When the granularity is further refined to the species level, where negative categories are from different species within the same genus (e.g., "Is the object in this image a black-footed albatross/Layson albu- tross""), the accuracy drops further to 62. 48% on true/false questions and 61.18% on multiple-choice questions. This demonstrates that the model struggles to distinguish between closely related species. The results for other LVLMs exhibit similar trends to those for Intern Additional examples of multiple-choice and true/false questions can be found in Appendix C.1
300
InueRaake multplechose
50
"Is the class of the object in this image an
20
Geros
Spedes
Figure 2: Results of Intern Zhuu et al. (025) on true/false and multiple-chon questions across different levels of granu- larity on the CUB-200-2011 (Wah et al.. (011) dataset. The x-axis denotes the granularity of the recognition questions.
The inconsistent recognition accuracy of LVLMS across fine-grained categories can be attributed to the characteristics of their training data and the underlying L.M base. To examine whether LVLMs exhibit knowledge bias in recognizing different fine- iieeggrineed categories, accuracy in answering true/false questions for each fine-grained category. As shown in Figure 3 using example, the model demonstrates highly inconsistent recognition abilities across categories, achieving nearly 90% accuracy for some while dropping to approximately 30% for others.
ranked their
LLaVA (Liu et al. 2024a)
'hen questions are relatively simple, L.I.Ss achieve very high accuracy. The higher accuracy multiple- choice questions compared true/false questions may result from randomness.
5

---

Published as a conference paper at ICLR 2026
Table 2: Attribute recognition accuracy of Intern Zhu al. 2025 on the CUB-200-2011
Wah
et al. 2011
dataset (values in parentheses represent the average accuracy for each attribute).
Color Attribute (47.40)
belly color
58.49
back color
34.98
bill color
51.31
breast color
54.25
crown color
55.30
eye color
84.59
forehead color
53.32
leg color
44.01
nape color
39.24
throat color
52.77
under tail color
34.69
underparts color
56.20
upper tail color
37.30
upperparts color
28.75
wing color
30.16
primary color
43.05
Pattern Attribute (50.13)
back pattern
40.94
belly pattern 68.13
breast pattern
65.12
head pattern
35.92
tail pattern
41.64
wing pattern
49.04
Shape Attribute (30.95)
bill shape
37.61
shape
52.37
tail shape
10.42
wing shape
23.39
Length Attribute (71.03)
Size Attribute (52.55)
bill length
71.03
size
52.55
We hypothesize that this inconsistency arises either from the uneven representation of fine-grained knowledge in the training data from the inherent difficulty LVLMs face in learning certain fine-grained categories. To test this hypothesis, we fine-tuned LVLMs on datasets with balanced occurrences of fine-grained categories and evalu- ated their performance. As indicated by the yellow dots in Figure the fine-tuned LLaVA shows consistently strong recogmition ability across all fine-grained categories. It suggests that the knowledge bias is primarily caused by the uneven representation of fine-grained knowledge training data, rather than the inherent difficulty of learning specific categories.
Accuraty of truntall Question
64
Orginnl ELaWA Fimeed und LLaMA
150
Eategorses
Figure 3: Comparison of the original (blue dots) and fine-tuned (yellow dots) LLaVA models fine- line-grained bird categories. True/false accuracy per category is ranked.
occurrence-balanee
To further validate this hypothesis, examined the occur- rence frequency of fine-grained categories in the LVLM's
training data. Interestingly, we found that such categories are almost absent from the training data. This indicates that the observed inconsistency in recognition ability is inherited from the language model underlying the LVLM, rather than being solely a consequence of the visual model itself. Additional results for other LVLMs exhibit similar trends and can be found in Appendix C.2
LVLMs exhibit significant room for improvement in recognizing fine-grained attributes. As shown in Table 2 of the paper and Table 14 in the appendix, LVLMs exhibit relatively stronger recognition abilities for attributes like pattern, size and length compared to others. For example, VL3 (Zhu al.) 2025) and Qwen2.5-VI. (Bai et al. 2025) achieve 50.13% and 45.12% average accuracy for pattern recognition, but only 30.95% % and 29.30% for shape recognition. Notably, only few attributes achieved accuracy above 70%, and several attributes scored low 10%, highlighting significant room for improvement LVLMs' fine-grained attribute recognition.
We also observe model-specific differences in attribute recognition. For instance, while Intern struggles more with pattern recognition compared to size, Gemini-2.0 1aamii-2.0-ffaa O-flash (Gemini Team 2024) shows the opposite trend. Additionally, find that the latest LVLMs show significant improvements in pattern and length recognition, while their advancements in color and shape recognition are more limited. Detailed results of the attribute recognition task provided in Appendix C.3
LVLMs do not outperform fine-grained tailored models in fine-grained tasks. Table 3
compare the recognition accuracy of LVLMs with that of fine- enn-igained tailored models. While LVLMS demonstrate notable performance fine-grained recognition, their accuracy accuracy- whether in short-answer questions or using the linear classifier method-falls short of that achieved by fine- grained tailored models. This disparity may stem from the fact that LVLMs are primarily optimized for general tasks, with less emphasis placed on fine-graine grainer domain capabilities (e.g., CAP (Behera et al. (021) uses context-aware attentional pooling to capture hierarchical contextual information from pixel to region to image level, leading to a 1.53% improvement in classification accuracy).

---

Published as a conference paper at ICLR 2026
Deps
Figure 4: Retrieval results of LVLM visual fea- tures on twelve fine-grained datasets. Different colors represent different models.
Classification results of LVLM visual features on twelve fine-grained datasets. Differ- ent colors represent different models.
Figure 5:
Intervi
DINb22
B2TT3
CUP
LLeVa
Inaami.
08000
BETT3
EVA CUP CoCs
CoCa
Figure 6: Nemenyi statistical test results for fine-grained retrieval. Black horizontal lines in- dicate the critical distance (CD), grouping mod- els with no significant performance differences.
Figure 7: Nemenyi statistical test results for fine- grained recognition. Black horizontal lines indi- cate the critical distance (CD), grouping models with significant performance differences.
Due to the architectural paradigm of LVLMs (typically VIT MLP + LLM), these components cannot be directly ap- plied. However, the core idea of attend- ing to hierarchical-Ievel details can still be incorporated into LVLMs (e.g., In- VL3 (Zhu et al. 2025) not only takes the full image as input, but also divides the image into parts to obtain more fine- grained visual representations). Enhanc-
Table 3: Comparison of LVLMs and fine- fine-grained tailored models on classification tasks. "SA" denotes LVLM re- sults fine-tuned fine-grained datasets for questions, "L." represents linear classifier results us- ing LVLM visual features, and "FG-Tailored" refers to state-of-the-art fine-grained tailored model results.
Datasets
SA
LC
FG-Tailored
CUB-200-2011
85.60
91.65
93.10
Diao
2022
Stanford Dog
86.49
90.50
97.30
BBera et al.
2022
ing LVLMs performance in specialized ar- cas (e.g., fine-grain ane-ggrined recognition) while preserving their strengths in general tasks presents an important and promising direction for future research.
Stanford Car
90.55
94.30
97.
TO Liu 2024
Food-101
95.25
95.67
98.60
(Behera al. 2021
FGVC Aircraft
66.19
78.88
95.40
Sıkdar
2024
4.3 MACHINE-ORIENTER EVALUATION
Compared human-oriented evaluation, machine-oriented evaluation takes a more direct approach assess the visual feature representations of LVLMs by employing two fundamental visual tasks: image retrieval and image recognition. This evaluation focuses on two key aspects: 1) Discriminability- the ability of visual features to distinguish fine-grained categories, and 2) Robustness maiinaaning accuracy under feature perturbations.
To evaluate discriminability, we start by analyzing the performance of visual features on retrieval and classification tasks, following the setting of DINOv2 (Oquab et al. 2023). Figures 4 and display retrieval and classification results across twelve fine-grained datasets, while Figures 6 and provide the Friedman test results for these tasks.
We then increase classification difficulty by merging fine-grained categories from different super- categories into a unified training and testing set. Figure presents classification accuracy within a single super-category and across multiple meta-categories. Visual features with strong discriminabiliit should maintain high accuracy even with diverse data sources. Additionally, we analyze how the vision encoder size in LVLMS impacts classification accuracy, shown in Figure

---

Published as a conference paper at ICLR 2026
Halook
Figure 8: Classification results of LVLM visual features fine-grained datasets. "Single" denotes accuracy from training single meta-category, while "Multiple" reflects accuracy from training on a unified dataset combining multiple ceta-categosies.
CUB-200-2911
Standied Doge
Teodlo:
DecePlatiion
StreCinn
Figure 9: Classification results with different vision encoder sizes. Bars filled with different patterns represent different models, with darker patterns indicating larger vision encoder sizes.
Furthermore, we examine how vision-language alignment-au critical component of LVLMs visual feature quality. Specifically, we compare classification accuracy between original visual features and those aligned textual features in fine-grained classification tasks, as shown in Table
To investigate robustness, we introduce perturbations to visual features using projected gradient descent (Madry et al. 2018) and analyze their impact on classification accuracy. Results of this experiment are presented in Table 5
The contrastive training paradigm in LVLMs effectively enhances the fine-grained discrim- inability of visual features. In Figure 4 and Figure vision encoders trained with contrastive paradigms (e.g., EVA-CLIP, InternVL, DINOv2) outperform those trained with reconstruction (BEIT3) and generative paradigms (Qwen) on fine-grained retrieval and classification tasks. As Nemenyi test results also indicate that Intern VL. EVA-CLIP, and DINOv2 perform significantly better than Qwen and BEiT3. In multi meta-category classification (cf. Figure 8). EVA-CLIP maintains high accuracy, with an average drop of just 1.96% compared to the single-category setting, while Qwen and BEiT3 show larger drops of 4.16% and 7.41%, respectively. This underscores the effectiveness of contrastive paradigms in fine-grained tasks.
shown in Figure 6 and Figure [
In experiments of vision encoder size, shown Figure ? we observe that DINOv2-B, with smaller vision encoder, achieves higher classification accuracy compared to BEiT3-L, with larger vision encoder, outperforming by 8.08% on CUB-200-20/1 and 9.49% on Stanford Dogs. We attribute these findings to the limitations of generative and reconstruction training paradigms, which fail to adequately distinguish between features different fine-grained categories, while maintaining similarity within the same. This limitation hinders their performance on fine-grained tasks.
The alignment strategy in LVLMs might impair the fine-grained discriminability of visual features. Alignment plays a critical role in LVLMs by bridging the gap between visual and textual features. To assess its impact on fine-grained visual fea- ture discriminability, we compare the clas- sification accuracy of LLaVA's (Liu et al.] 2024a) original visual features with those aligned to textual features on fine- mraigri classification tasks. As shown in the first two columns of Table 4 the original fea- tures demonstrate superior classification performance, outperforming the aligned ones by an average of 3.39%.
Table 4: Classification accuracy of LLaVA visual fea- tures before and after alignment. "Origin" denotes original features from the vision encoder. "Aligned" denotes features aligned text with inconsistent gran- ularity, while "Aligned-Foo -FG" denotes those aligned to fine- ene-grained text.
Datasets
Origin
Aligned
Aligned-Fr
CUB-200 200-2011
79.77
73.17
75.06
Stanford Dogs
81.24
78.14
80,69
Stanford Cars
87.57
83,90
85.63
Food-101
94.27
93.35
94.32
DeepFashion
69.94
67.30
67.75
This decline in performance be attributed to two key factors. First, aligning visual and textual features may introduce distortions due to inconsistencies between their respective feature spaces. Second, granularity inconsistencies in LVL.Ms' alignment fata-where fine-grained objects in images
8

---

Published as a conference paper at ICLR 2026
Table 5: Classification results of LVLMs' original and perturbed visual features on the fine-grained dataset CUB-200-208 / and the generic dataset CIFAR-100. "Origin" refers to results with original features, while "Perturbed" indicates results with perturbed features.
EVA-CLIP
CoCa
DINOv2
ViT
Datasets
Origin
Perturbed
Origin
Perturbed
Origin
Perturbed
Origin
Perturbed
CIFAR-100
93.05
50.76
86.94
52.23
93.38
42,39
89.81
72.15
CUB-200-2011
88.95
24.94
79.89
23.40
91.64
25.94
88.83
73.85
are paired with coarse coarse-grained textual descriptions (as demonstrated in our qualitative analysis in Appendix negatively affect the discriminability of the aligned visual features.
To validate the impact of granularity inconsistency, we construct a new aligmment dataset where the textual descriptions match the granularity of the fine-grained objects in images. We then retrain the alignment module in LLaVA using this dataset. As shown in Table [ the classification accuracy of visual features aligned with fine-grained textual content improves significantly, with increase of 2.55% on Stanford Dogs and 1.73% on Stanford Cars. While the aligned visual features still underperform compared to the original features, these results underscore the detrimental effects of granularity inconsistency on visual feature discriminability and highlight the importance of ensuring granularity alignment between image and textual content. Further experiments in Appendix [D.2 reveal that enhancing the fine-grained discriminability of visual features during the alignment stage can lead to improved LVLM performance on fine-grained tasks.
Larger vision encoder size or larger scale of web data provides limited improvement in the fine-grained discriminability of visual features. Regarding vision encoder size, as shown in Figure 1 increasing the size of DINOv2's 's vision encoder from DINOv2-B to DINOv2-L results in only 0.6% improvement in average classification accuracy, while further increasing from DINOv2-L. to DINOv2-G leads to just a 0.3% improvement. The classification accuracy of visual features in InternVL-6B is not superior to that of DINOv2-L, suggesting that merely enlarging the vision encoder has limited impact enhancing the fine-grained discriminability of visual features.
Regarding the amount of training data, as shown Figure 7 the vision encoder of EVA-CLIP, trained on over 2 billion data samples, does not outperform DINOv2, which trained on 142 million samples, in fine-grained classification and retrieval tasks. We attribute this difference to the quality of the training data, that DINOv2's dataset is carefully curated from large pool of data, whereas EVA-CLIP relies on crawled web data. A similar trend is observed when comparing DINOv2 with Intern VL (6B samples), suggesting that simply increasing the scale of training data, without considering its quality, offers limited improvement in fine-grained discriminability of visual features.
Visual features in LVLMs are more susceptible perturbations in fine-grained tasks. As shown in Table perturbing the visual features of EVA-CLIP causes a significant drop in classification accuracy on the fine- -ne-grained dataset CUB-200 200-2000 (from 88.95% to 24. 944... In contrast, on the generic dataset CIFAR-100 (Krizhevsky & Hinton (009)), the accuracy declines more modestly (from 93.05% to 50.765... 30.76%). Similar trends are observed for CoCa and DINOv2, indicating that fine-grained tasks are more susceptible to perturbations than generic tasks. We attribute this vulnerability to coarse- grained, noisy training data, which limits feature discriminability across fine- -g-ggriinedd categories, making it hard to distinguish them when visual features are perturbed.
In contrast, the Vision Transformer (Wightman) 2019) trained on the ImageNet (Deng et al. 2009) dataset with cross-entropy loss demonstrates superior robustness perturbations. It shows only minor declines in classification accuracy on both fine-grained and generic datasets. This suggests that adopting alternative training paradigms or incorporating high-quality, fine-gra mne-grained data (as seen in ImageNer) during training could help improve the robustness of visual features in LVLMs.
5 CONCLUDING REMARKS
We conducted a comprehensive evaluation to investigate the capabilities of LVLMs on fine- -nainnai visual tasks, leading to several key findings. First, we observed that the contrastive training paradigm significantly enhances the fine-gre -me-grrined discriminability of visual features, whereas generative and reconstruction-basee paradigms tend to underperform in this aspect. Second, aligning visual features

---

Published as a conference paper at ICLR 2026
with textual features can impair their fine-graines discriminability, particularly when there is mismatch in granularity between the paired image-text data. Third, our experiments revealed that LVLMs and LVMs are more susceptible to feature perturbations in fine-grainee tasks compared to generic vision tasks, highlighting a vulnerability that merits further attention. Fourth, while LVLMs demonstrate strong capabilities in visual appearance perception, they face notable challenges in fine- grained category reasoning, which depends heavily on recognizing subtle visual attributes. Finally, despite their advancements, LVLMs still lag behind specialized fine-grained models, underscoring the need for task-specific enhancements. These findings highlight key limitations of LVLMs, identifying critical areas for improvement, such as training paradigms, robustness perturbations, and reasoning capabilities. Our work lays the groundwork for future research to advance LVLM performance on fine-grained visual tasks.
Ethics Statement All data used in this study are collected from publicly available datasets that comply with standard research ethics. The benchmark tasks and annotations in FG-EMK con- structed through automatic or rule-based question generation without involving human subjects. Although the dataset SkinCon relate to medical images, it is fully anonymized and released under ereearch-permissiin licenses. Our use strictly follows its terms and involves no patient-identifying information, clinical decision-making, or sensitive personal data. We do not release any private or sensitive information, nor do our methods introduce additional safety, faimess, or legal concerns. Overall, FG-BMK provides a practical and ethically compliant testbed for evaluating LVLMs in fine-grained real-world scenarios.
Reproducibility Statement We ensure reproducibility by using only publicly available datasets and by clearly describing our benchmark construction process in Section 3 and Appendix [A] The evaluation pipeline, including prompt design, scoring methods, and model configuration, is detailed in Section 3] Appendix A and Appendix ( We provide extensive implementation details and dataset statistics the Appendix. All code, data splits, and evaluation scripts has be released.
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VL: A versatile rision-langual model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li. Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 12000...922 13923, 2025.
Yalong Bai, Yuxiang Chen, Wei Yu, Linfang Wang, and Wei Zhang. Products- 10K: A large-scale product recognition dataset. arXiv preprint arXiv:2008. 10544 2020.
Ardhendu Behera, Zachary Wharton, Pradeep RPG Hewage, and Asish Bera. Context-aware at- tentional pooling (cap) for fine-grained visual classification. In Proc. Conf. AAAI, pp. 929-937, 2021.
Asish Bera, Zachary Wharton, Yonghuai Liu, Nik Bessis, and Ardhendu Behera. SR-GNN: Spatial relation-aware graph neural network for fine-grained image categorization. IEEE Trans. Image Process., 31:6017-6031, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 101-mnngg discriminative compo- nents with random forests. In Proc. Eur. Conf. Comp. Vis., pp. 446-661, 2014.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Intern VL: Scaling up vision foundation models and aligning for generic essual-linguisic tasks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 24185-2418 2024.
10

---

Published as a conference paper at ICLR 2026
Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. SkinCon: A skin disease dataset densely annotated by domain experts for fine- grained debugging and analysis. In Advances in Neural Inf. Process. Syst., pp. 18157-18167, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. IEEE Conf. Comp. Vis. Pall. Recogn., pp. 248-255, 2009.
Qishuai Diao, Yi Jiang, Bin Wen, Jia Sun, and Zehuan Yuan. MetaFormer: unified meta framework for
recognition. arXiv preprint arXiv:2203.022 2022.
Gregor Geigle, Radu Timofte, and Goran Glavaš. African or european swallow? benchmarking large laiiuale models for fine- ame-grained object classification. arXiv preprint arXiv:2406. 11766.1499 14496, 2024.
Google Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 3iii:22033.053000 2024.
Saihui Hou, Yushan Feng, and Zilei Wang. VegFru: A domain dataset for fine-grained visual categorization. In Proc. IEEE Int. Conf. Comp. Vis., pp. 541-549, 2017.
Drew A Hudson and Christopher D Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 6709, 2019.
Dong Jing, Xiaolong He, Yutian Luo, Nanyi Fei, Guoxing Yang. Wei Wei, Huiwen Zhao, and Zhiwu Lu. FineCLIP: Self-distilled region-based clip for better fine-grained understanding, In Advances in Neural Inf. Process. Syst., pp. 27896-27918, 2024a.
Yinuo Jing, Ruxu Zhang, Kongming Liang, Yongxiang Li, Zhongjiang He, Zhanyu Ma, and Jun Guo. Animal-Beae Bench: Benchmarking multimodal video models for animal-centric video understanding. In Advances in Neural Inf. Process. Syst., pp. 23457-23469, 2024b.
A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel dataset for fine-grained image categorization. In CVPR Workshop on Fine- Fine-Grained Visual Categorization, pp. 806-813, 2011.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In Proc. IEEE Ins. Conf. Comp. Vis., pp. 554-561, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified linoualan understanding and generation. In Proc. Int. Conf. Mach. Learn., pp. 12888-12900 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proc. Int. Conf. Mach. Learn., pp. 19730-19742 2023.
Dichao Liu. Progressive multi-task anti-noise learning and distilling frameworks for fine-grained vehicle recognition. IEEE Trans. Intell. Transp. Syst., 25(9): 10677--766617777 2024. doi: 10. 109/ TITS 104.33002......5..
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proc. IEEE Conf. Comp. Vis. Patl. Recogn., pp. 26296-26306, 2024a.
Ling Liu, Maoxiang Chu, Rongfen Gong, Liming Liu, and Yonghui Yang. Weighted linear loss large margin distribution machine for pattern classification. Chinese J. Electron., 33(3):753-765, 2024b.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu- tu-Cheng Yin, Cheng- Lin Liu, Lianwen Jin, and Xiang Bai. OCRBench: On the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12), 2024c. ISSN 1869- 1919. doi: 1007/ sl 1432-024-4235-6 URL w /x doi org/ 1007/ 8 020-2-224443377

---

Published as a conference paper at ICLR 2026
Zhihang Liu, Chen-Wor Xie, Bin Wen, Feiwu Yu, Jixuan Chen, Pandeng Li, Boqiang Zhang, Nianzu Yang. Yinglu Li, Zuan Gao, et al. Capability: A comprchensive visual caption benchmark for evaluating both correctness and thoroughness. arXiv preprint arXiv:2502114914, 2025.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. DeepFashion: Powering robust clothes recognition and retrieval with rich annotations. In Proc. IEEE Conf. Comp. Vis. Patl. Recogn. pp. 1096-1100 1104, 2016.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Math Vista: Evaluating mathematical reasoning of foundation models in visual contexts. In Proc. Int. Conf. Learn. Representations, 2024.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. Proc. Int. Conf. Learn. Represen- tations, 2018
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rX.v:1306.51555 2013.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Proc. Conf. Association for Computational Linguistics, pp. 2263-2279, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doc A dataset for vqa on document images. In Proc. Winter Conf. Applications of Comp. Vis., pp. 2200-2209, 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proc. IEEE Int. Conf. Comp. Vis., pp. 722-729, 2008.
2023 OpenAl. GPT-4 technical report. arXiv preprint ar arXiv:2303.08774, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes. Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wor Wen Li. Wojeiech Galuba, Mike Rabbat. Mido Assran, Nico- las Ballas, Gabriel Synnaeve, Ishan Misra, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07198, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskeverothers. Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn., pp. 8748 8763, 2021.
Yang Shen, Xuhao Sun, Xiu-Shen Wei, Qing- Qung-Yuan Jiang, and Jian Yang. SEMICON: A learning- to-hash solution for large-scale fine-grained image retrieval. In Proc. Eur. Conf. Comp. Vis., pp. 531-548, 2022.
Arindam Sikdar, Yonghuai Liu, Siddhardha Kedarisetty, Yitian Zhao, Amr Ahmed, and Ardhendu Behera. Interweaving insights: High-order feature interaction for fine-grained visual recognition. In Proc. IEEE Int. Conf. Comp. Vis., pp. 1755-1779, 2024.
Kaitao Song, Xiu-Shen Wei, Xiangbo Shu, Ren-Jie Song, and Jianfeng Lu. Bi-modal progressive mask attention for fine-graine ane-ggrinedd recognition. IEEE Trans. Image Process., 29:7006-7018, 2020.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: Improved training techniques for clip at scale. arXiv preprint 2331v:2303.15389, 2023.
Tianchi. Bottled wine defect detection data set, 2021. URL htt:::://tianchhi www.sii.lllinccccc com/
?wd.aadaalaa 110147
Grant Van Horn Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proc. IEEE Conf.
Comp. Vis. Patt. Recogn., pp. 12884-12893, 2021.
12

---

Published as a conference paper at ICLR 2026
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD birds-200-2011 dataset. Technical report, California Institute of Technology. 2011.
Shanmin Wang, Hui Shuai, Lei Zhu, and Qingshan Liu. Expression complementary disentanglement network for facial expression recognition. Chinese J. Electron., Electron.,33())744.. 33(3):742-752, 2024.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: BEiT pretraining for vision and vision-language tasks. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pp. 19175-19186, 2023.
Xiu-Shen Wei, Quan Cui, Lei Yang, Peng Wang, Lingqiao Liu, and Jian Yang. RPC: A large-scale and fine-grained retail product checkout dataset. Science China. Information Sciences, 65(9): 197101, 2022a.
Xiu-Shen Wei, Yi-Zhe Song. Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang. and Serge Belongie. Fine-grained image analysis with deep learning: A survey. IEEE Trans. Patlern Anal. Mach. Intell., 44(12):897788848 2022b.
Xiu-Shen Wei, Tang-Tao Yu, Anqi Xu, Faen Zhang, and Yuxin Peng. MECOM: A meta-complction network for fine-grained recognition with incomplete multi-modalities. IEEE Trans. Image Process., 33:3456-3469, 2024.
Ross Wightman. Pytorch image models. Bytorch-inage-modul 2019.
https://qithub com/ rwi wowm/wwigttann
Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. LVLM-cHub: A comprehensive evaluation benchmark for large vision-language models. IEEE Trans. Pattern Anal. Mach. Intell., 47(3):1877-1893, 2025, doi: 10.1 10.1109/TPAMMI.224.55577
Shu-Lin Xu, Faen Zhang, Xiu-Shen Wei, and Jianhua Wang. Dual attention networks for few-shot fine tne-grained recognition. In Proc. Conf. AAAI, pp. 2911-2919, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
Liu Yuan, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. MMBench: Is your multi-modal model an all-around player? In Proc. Eur. Conf. Comp. Vis., pp. 216-233, 2024.
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun. Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pp. 9556-9567, 2024.
Ruru Zhang, Haihong E, and Meina Song. FSCIL-EACA: Few-Shot Class-Incrementee learning network based embedding augmentation and classifier adaptation for image classification. Chinese J. Electron., 11:9133911122 2024a.
Ruru Zhang, Haihong E, Lifei Yuan, Yanhui Wang. Lifei Wang, and Meina Song. FGM-SPCL: Open-set recognition network for medical images based on ine-gnained data mixture and spatial position constraint loss. Chinese J. Electron., 33(4): 1023- 2024b.
Yuhui Zhang, Alyssa Unell, Xiaohan Wang. Dhruba Ghosh, Yuchang Su. Ludwig Schmidt, and Serena Leung-Levy. Why are risually-rrunnde language models bad at image classification? arXiv preprint 2405 1.440058844555 18415, 2024c.
13

---

Published as a conference paper at ICLR 2026
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu. Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao. Yangzhou Liu, Xingguang Wei, Hongjee Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge. Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
A THE EVALUATION BENCHMARK
A.1 EVALUATION TASK DETAILS
In Section 3.1 we have described each evaluation task. Here, we provide further details. In the Knowledge Bias Estimation task, uncover potential knowledge biases across different fine-grained categories, we pair each image with its corresponding fine-grained label to generate positive samples for true/false questions. For constructing negative samples, each image paired with a single fine- grained label randomly selected from other subcategories within the same super-category. For each fine-grained category, we calculate the LVLM's accuracy on all coresponding true/false questions as measure of its understanding of that category's knowledge.
In the cross meta-class classification task, we follow the DINOv2 (Oquab al. 2023) method to train the model on unified training set where fine-grained categories from different datasets are combined. The model is then tested on each individual dataset to evaluate its performance.
A.2 DATA CURATION
Dataset We source images for the FG-BMK benchmark from 12 fine-grained datasets. These datasets cover a wide range meta-classes, with different categories and sample, providing comprehensive assessment of LVLMs capabilities fine-grained tasks across different domains. Table/Jjidicates their meta-classes, the amount samples, the number categories, For all datasets, construct human-oriented evaluation questions based on their test sets. We use the original labels directly from the datasets for the machine-oriented evaluation.
Table 6: Details of 12 fine-grained datasets sorted by their numbers of categories. "Meta-class" refers high-level categorization of the dataset. "Categories" refers to the number of fine-grained categories. "Samples" refers to the total number of samples in each dataset.
Datasets
Meta-class
Categories
Samples
Wine (Tianchi 2027)
Industrial
11
4,516
DeepFashion Luu el al. 2016)
Clothes
46
18,000
SkinCon Daneshjou et al. 2022
Dermatology
48
3,866
Flowers102 Nillsbck & Zisserman
2008
Flower
102
7,169
Food10T (Bossard al. 2014
Food
101
101,000
FGVC Aircraft (Maji et al.
2013
Aircraft
100
6,667
Stanford Dogs Khosla al.
2011
Dog
120
20,580
Stanford Cars (rruse et al.
2013
Car
196
16,185
CUB-200-2011 (Wah
2017
Bird
200
11,788
Veg (Hou et al, 2017] 7
Vegetable
292
146,131
Products- 10K (Bai et al.
2020
Retail
9,691
197,307
iNar2021
Van Horn et al.
2027
Plants
10,000
2,786,843
Human-oriented Question Templates When constructing true/false, multiple-choice short answer questions for each task in human-oriente evaluation, we manually design several question templates ensure both diversity and comprehensive coverage. Figure T0 illustrates the question templates we use for generating the tasks.
We also expanded the original template set to 10 diverse numan--ritter prompts and reconstructed the multiple-choice questions in the numan-oriented benchmark to examine the potential impact of linguistic diversity. As shown in Table 7 and Table g increasing the number of templates leads to only minor changes in accuracy, and the overall LVLM behavior and observed trends remain consistent. Therefore, as long as the template clearly states the question, the effect of the template quantity on the results is negligible.
15

---

Published as a conference paper at ICLR 2026
Question Template for Attribute Recognition Task
True/False Question:
wing color the bird {colory?
Is the breast pattern of the bird {pattemn??
Multiple-choice Question: What the wing color of the bird? Choose from the following list: {options).
Question Template for Knowledge Bias Estimation Task
True/False Question:
species the (meta class) {fine grained category) Answer with yes or no. Does this {meta class) belong to the species known as (fine grained category]? Answer with yes or no. Is the fmeta class) species in this photo (fine grained category]? Answer with or From your observation, is the species of the (meta class) shown {fine grained category)? Answer with yes or no
Question Template for Hierarchícal Granularity Recognition Task
True/False Question:
Is the species the [meta (meta_class) (fine lereeegraeed___e grained_category]) Answer with yes or no. Does this (meta class] belong to the species known siine_grained___ category)? Answer with or no. Is the {meta class} species in this photo a (fine grained category)? Answer with yes or From your observation, the species the (meta class) shown {fine grained category)? Answer with yes or no
Multiple-choice Question:
What the species of the (meta class; this image? Choose
from the following list: [options].
Answer the question using single word or phrase.
Can you identify the species of this (meta class}? Choose one answer from the following list: [options].
Answer the question using single word or phrase.
Which species does this {meta class) in the photo belong to? Choose
from the following list:
{options). Answer the question using single word or phrase.
Observing the (meta class) in the image, which of the following species it? Choose one answer from the following list: [options]. Answer the question using single word or phrase.
Short Answer question:
What species of the (meta_ [meta_class] class) is shown the image? Directly answer with species names.
Can you identify the species of this (meta class) from the image? Directly answer with species names.
Which species does the (meta class) in this photo belong to? Directly answer with species names.
Based on your observation, what species of the (meta class) depicted? Directly answer with species names.
Figure 10: Question templates for each task in huamn-oriented evaluation.
16

---

Published as a conference paper at ICLR 2026
Table Attribute recognition accuracy of Intern VL3 Zhu al. 2025) using original and extended
prompts on the CUB-200-2011 (Wah et al. (011)) dataset (values in parentheses represent the average accuracy for each attribute). Accuracy are shown in the format "original extended", with the left representing accuracy using the original prompt and the right using the extended prompt.
Color Attribute (47.40 47.45)
belly color breast color
58.49 / 60.04
back color
34.98 / 36.33
bill color eye color
51.31 49.64
54.25 / 55,91
crown color
55.30 54.01
84.59 / 82.96
forchead color
53.32/ 51.90
leg color
44.01 / 45.67
nape color
39.24 38.02
throat color
52.77 54.53
under tail color
34.69 / 35.80
underparts color
56,20 55.08
upper tail color
37.30 38.77
upperparts color
28.75 / 27.50
wing color
30.16/31.88
primary color
43.05 / 41,29
Pattern Attribute
(50.13 50.28)
back pattern
40.94 / 39.38
belly pattern
68.13 / 67.00
breast pattern
65.12 66.87
head pattern
35.92 / 34.66
tail pattern
41.64 / 42.93
wing pattern
49.04 00044/50. 50.84
Shape Attribute
(30.95 31.01)
bill shape
37.61 /3..41 23.39 / 24.98
shape
52.37 / 50.60
tail shape
10.42 12.04
wing shape
Length Attribute (71.03 69.71)
Size Attribute (52.55 / 54.21)
bill length
71.03 69.71
size
52.55 / 54.21
Table Results Intern VL3 using original and extended prompts on true/false (TF) and multiple- choice (MC) questions across different levels of granularity on the CUB-200-200 dataset. Results are shown the format "original extended".
TF
Class
98.79 /98.03
Genus
85,69 / 16.19
Species
61.88 62.34
MC
Class
99.42 / 99.558
Genus
88.13/87.66
Species
60.15/ 59,25 59.23
B EVALUATED MODELS
As shown in Table 9 we select nine widely-used open-source LVLMs, two closed-source models
(GPT-4o AO 2023) and Gemini-22.0flaat (Gemini Team 2024) and one purely visual model, each of which employs distinctive training recipes, including variations in vision encoder, language model, training losses and data.
Table Configurations of the evaluated models. "DINOv2-L" is purely visual model. "Con" stands for the contrastive loss, "Gen" for the generative loss, "Mat" for the image-text matching loss, "Rec" for the reconstruction loss as used BEiT3 (Wang et al. 2023), and "Dis" for the distillation loss as applied in DINOv2 (Oquab et al. 2023).
Component
Loss Function
Model
Vision Model
Language Model
Con
Gen
Mat
Rec
Dis
InternV3-7B InterVL-CChat V1.1
Intern VIT-L
Qwen2.5-7B
Intern ViT-6B
LLaMA2-13B
LLaVA-1.5-7B
CLIP-L
Vicuna-7B
Qwen2.5-VV
CLIP-600M
Qwen2.5-7B
V
v
Qwen-V DW
Openelip-G
Qwen-7B
BLIP-2-FLAN-TS, XL
EVA-CLIP-G
FlanT5-XL. XL
EVA02-CLIP-L
EVA02-L
CLIP-L
BETT--L-ITC
CLIP-L
CLIP-L
CoCa-L
CLIP-L
CLIP-L
DINOv2-L
CLIP-.
EVA-CLIP (Snn et al. 2023) aligns visual and textual features using contrastive loss, leveraging over billion web image-text pairs and advanced optimization techniques.
17

---

Published as a conference paper at ICLR 2026
InternVL3 (Zhu al. 2025) adopts unified pre-training approach over both multimodal and pure-text data, enhanced by variable visual position encoding (V2PE) and advanced post-training strategies for improved scalability and effectiieness.
InternVL. (Chen et al. 2024) leverages contrastive, matching, and generative losses in a multi-stage training process, with large-scale vision encoder and over 6 billion image-text pairs align visual and textual representation.
BLIP-2 (Li et al., 2023) bridges the modality gap between frozen image encoders and LLMs using lightweight Q-Former, leveraging contrastive, matching, and generative loss in two-stage pre-training process over 129 million data with fewer trainable parameters.
Qwen2.5VVL (Bai al. 2025) combines dynamic-resolution Vision Transformer with Window
Attention to reduce computational cost while preserving native image resolution.
Owen-VL. (Bai al., 2023) employs three-stage training process with generative loss, using VL. adapter to align visual and textual features while reducing computational cost over 1.4 billion image-text pairs.
CoCa (Yu et al., 2022) adopts task-specific attentional pooling to tailor visual representations for different training objectives, applying contrastive loss to train the first half of the decoder and generative loss to train the full decoder in an end-to-end manner over 5 billion image-text pairs.
BEIT3 (Wang et al. 2023) treats images as a foreign language, leveraging a mask-then- then-predict objective over 36 million image-text pairs to unify vision and language pretraining, and introduces a multiway transformer architecture for general-purposs modeling.
LLaVA (Liu et al. 2024a) aligns visual and textual features using a simple MLP with generative loss, leveraging 1.2 million GPT-4 (OpenAl 2023) generated multimodal instruction -oolloiinn data for training.
DINOv2 (Oquab et al. 2023 uses a self-supervised learning approach, leveraging knowledge distillation and a mask-ther teek-the--aee then-predict strategy over 142 million images to train the vision encoder.
For all our evaluated model, follow their official configurations to the inference. We set the temperature of all open-source models to O, while keeping the default for closed-source APIs.
C HUMAN-ORIENTED EVALUATIONS
C.1 RESULTS OF HIERARCHICAL GRANULARITY RECOGNITION
Figure shows VL33ss (Zhu al. 2025 accuracy in answering true/false and multiple-choice
questions within hierarchical granularity recognition task on CUB-200-201/ dataset. In Figure [T] we present additional results for GPT-4o OpenAl 2023). Gemini-2. 2-0ssh (Gemini Team 2024). Qwen2.5-VI. (Bai et al. 2025), LLaVA (Liu et al. 2024a) and Intern VL (Chen et al.) 2024) on CUB-200-200 (Wah et al. 201T) and iNar2021 (Vnn Horn al. 2021) datasets. As shown in the experiments, the accuracy of all models decreases as the granularity becomes finer. When the granularity level reaches the finest level, the models struggle to distinguish between closely related species.
C.2 RESULTS OF KNOWLEDGE BIAS ESTIMATION
we observe that LLaVA exhibit highly inconsistent recognition abilities across categories. We also conduct experiments with Qwen2.5-V. 5LL GPT-4o, and Gemini- 1ommini-2.0-fe -200-flash on fine- fine-grained datasets such as Aircraft (Maji et al.) 2013), Flowers 102 (Nilshack & Zisserman 2008) and Stanford Dogs (Khosla et al. 2011). As shown in Figure 12 and Figure 13 all LVLMs display similar trends, indicating inconsistent recognition abilities across fine-graine mne-grained categories. However, after fine-tuned on datasets with balanced occurrences of fine-grained categories, LVLMs demonstrate remarkable recognition abilities across all fine- ine-grained categories.
In Figure 3]
To construct datasets with balanced occurrences of fine- fine-grained categories, we select an equal number of images from each category. Then generate the same number of true/false questions for each -nn-ggraind category, thereby fine-tuning the LVLMs in a way that each category receives balanced representation.
18

---

Published as a conference paper at ICLR 2026
100
100
B true/talse
true/false
45
2 mutiple-chose
muliiple-hhsse
90
Accuncy
75
70
65
65
60
Kingsom Phylum
Class
Onder
Family
Genus
Species
Kingdom Phylu
Class
Order
Family
Genus
Species
(a) GPT-4o on iNar202/
(b) Genimi-2.0-fass on iNar2021
100
100
G trueffabe
true/false
90
2 chiipe
muliiple choie
Accuncy
50
50
40
30
20
Kingsom Phylum
Clas
Onkr
Family
Genus
Speciss
Kingdom Phylam
Class
Order
Family
Gans
Spocics
(c) LLaVA on iNar2021
(d) Intern VL iNar2021
100
100
Dttueffale
true/false
gimutplechhise
2 multiple-chokse
Acouncy
70
Accuracy
60
60
50
10
0
Class
Gerus
Stecles
Class
Genus
Species
(e) LLaVA CUB-200-2011
(l) Qwen2.5-VI. on CUB-200-2011
G true/false
G multiple chaise
60
Clus
Genus
Species
Inter VL on CUB-200-2011
Figure 11: Results of GPT-4o OpenAl 2023), Gemini-2.0 2.0-flah (Gemini Team 2024). Qwen2.5- VI. (Bai et al. 2025), LLaVA (Liu et al. 2024a) and Intern VL (Chen al. 2024 true/false and multiple-choice questions across different levels of granularity on CUB-200-201 (Wah et al.) 2011) and iNar202/ (Van Horn al. 202T) dataset. The x-axis denotes the granularity of the recognition questions.
19

---

Published as a conference paper at ICLR 2026
380
Accuracy of TruciFalse Question
70-
40
Gem in mlla2.
Index Fine Grained Categories
Index Fine-Grained Categories
(a) GPT-4o Flowers102
(b) Gemini-2.0-fian Stanford Dog
Figure 12: Knowledge bias estimation results of two closed- closed-source source models. True/false question accuracy for each category is ranked, with blue dots representing the original model.
230
Accuracy trueiralse Question
Original Qwen2.5-VL Frigi.2 Fine tured Qwen2.5 VL
Index Fine-Graines Categories
Figure 13: Comparison of the original and fine-tuned Qwen2.5-VL VL (Bii et al. 2025) models on eecurrence-balancee fine- fine-grained aircraft categories. True/false question accuracy for each category ranked, with blue dots representing the original model and yellow dots the fine-tuned model.
Table 10: Attribute recognition accuracy of LLaVA (Liu et al., 2024a on the CUB-200-2011
Wah
al. 2011 dataset (values parentheses represent the average accuracy for each attribute).
Color Attribute (44.34)
belly color
54.79
back color
41.90
bill color
41.44
breast color
49.56
crown color
48.71
eye color
69.27
forehead color
47.03
leg color
35.37
nape color
40.51
throat color
35.40
under tail color
38.88
underparts color
54.81
upper tail color
41.41
upperparts color
34.00
wing color
34.60
primary color
41.77
Pattern Attribute (23.69)
back pattern
27.27
belly pattern 26.41
breast pattern
24.24
head pattern
11.35
tail pattern
23.19
wing pattern
29.67
Shape A Attribute (14.05)
hill shape
1.39
shape
18.59
tail shape
9.89
wing shape
26.34
Length Attribute (15.71)
Size Attribute (49.47)
bill length
15.71
size
49.47
C.3 RESULTS OF ATTRIBUTE RECOGNITION
Table 2 and Table shows the attribute recognition accuracy of InternVLI and Qwen2.5-VL on the CUB-200-288 200-2011 dataset. The results of LLaVA, BLIP2, Intern and Gemini-2.0 2.0-flash are shown in Table 10 Table Table 12 and Table 13
20

---

Published as a conference paper at ICLR 2026
Table 1I: Attribute recognition accuracy of BLIP2 (Li al. 2023) on the CUB-200-2011
(Wah et
2011
dataset (values in parentheses represent the average accuracy for each attribute).
Color Attribute (37.94)
belly color
51.15
back color
39.64
bill color
23.42
breast color
50.17
crown color
42.59
eye color
23.59
forehead color
43.18
leg color
18.77
nape color
41.55
throat color
53.81
under color
37.98
underparts color
41.60
upper tail color
37.52
upperparts color
33.01
wing color
31.25
primary color
33.48
Pattern Attribute (11.34)
back pattern
14.66
belly pattern 7.82
breast pattern
9.48
head pattern
2.14
tail pattern
14.21
wing pattern
19.73
Shape Attribute (25.05)
bill shape
8.84
shape
34.69
tail shape
13.51
wing shape
43.19
Length Attribute (30.11)
Size Attribute (27.62)
bill length
30.11
sizc
27.62
Table
12:
Attribute recognition accuracy of InternVI. (Chen et al.
2024 the 178-200-2011
Wah
et al. 2011 dataset (values in parentheses represent the average accuracy for each attribute).
Color Attribute (35.78)
belly color
52.09
back color
33.89
bill color
26.59
breast color
46.58
crown color
39.91
eye color
23.68
forehead color
40.83
leg color
32.75
nape color
29.66
throat color
30.21
under tail color
32.31
underparts color
50.57
upper color
33.42
upperparts color
29.64
wing color
27.17
primary color
40.15
Pattern Attribute (34.71)
back pattemn
35.57
belly pattern 44.14
breast pattern
42.22
head pattern
11.81
tail pattern
35.86
wing pattern
37.31
Shape Attribute (23.03)
bill shape
12.16
shape
38.08
tail shape
15.49
wing shape
26.43
Length Attribute (29.31)
Size Attribute (47.70)
bill length
29.31
size
47.70
Table 13: Attribute recognition accuracy of Gemini-2.00 2a0mini-2.0-llsh 2.0-flash (Gemini Team 2024 the CUB-
200-2011 (Wah al. 201T) dataset (values in parentheses represent the average accuracy for each attribute).
Color Attribute (47.22)
belly color
62.09
back color
36.51
bill color
52.31
breast color
56.01
crown color
56.44
eye color
59.57
forehead color
53.55
leg color
40.66
nape color
40.40
throat color
60.23
under tail color
40.60
underparts color
59.65
upper tail color
39.99
upperparts color
29.66
wing color
29.21
primary color
38.69
Pattern Attribute (56.14)
back pattern
56.26
belly pattern
70.51
breast pattern
66.89
head pattern
39,56
tail pattern
52.33
wing pattern
51.26
Shape Attribute (48.75)
bill shape
61.62
shape
68.20
tail shape
32.13
wing shape
33.04
Length Attribute (71.82)
Size Attribute (52.72)
bill length
71.82
size
52.72
D MACHINE-ORIENTED EVALUATIONS
D.1 QUALITATIVE ANALYSIS OF GRANULARITY INCONSISTENCY LVLM ALIGNMENT
DATA
In the LVLM's alignment data, observe phenomenon of granularity inconsistency. where fine- grained objects in images are paired with coarse- -griinedd textual descriptions. Figure [4]shows some examples of granularity inconsistency, well constructed sample of properly aligned granularity.
In practice, ensuring fully consistent fine- ine-grained granularity across all image-text pairs often infeasible, especially when relying web-scale weakly labeled data. In retraining experiment
21

---

Published as a conference paper at ICLR 2026
Table 14: Attribute recognition accuracy of Qwen2.5-VL (Bai al. 2025) on the CUB-200-
201/ (Wah et al., 2011) dataset (values in parentheses represent the average accuracy for each attribute).
Color Attribute (40.39)
belly color
51.11
back color
32.89
bill color
46.50
breast color
44.84
crown color
46.54
eye color
54.85
forehcad color
44.57
leg color
37.79
nape color
36.49
throat color
40.74
under tail color
34.60
underparts color
50.20
upper tail color
34.92
upperparts color
27.20
wing color
26.03
primary color
36.96
Pattern Attribute (45.12)
back pattern
42.66
belly pattern 64.58
breast pattern
59.79
head pattern
14.57
tail pattern
45.04
wing pattern
44.11
Shape Attribute (29.30)
bill shape
15.30
shape
58.17
tail shape
5.63
wing shape
38.10
Length Attribute (63.20)
Size Attribute (52.56)
bill length
63.20
52.56
in Table we made efforts to construct more consistent alignment data, but some residual mismatch may still exist.
Qualitative Analysis of Granularity Inconsistencies in LVLMs' Alignment Data
Samplel:
Question: Render clcar and concise summary of the photo.
Answer: a cat looking back sitting on rock at the occan vacation.
Sample2:
NING
Question: Summarize the visual content image.
Answer: dog lying on the floor with text happy dog training can be cured by behavior experts.
Sample3:
Question: Deseribe the image concisely.
Answer: small brown bird drinking water from puddle.
Constructed Granularity Aligned Sample:
Question: What is the object speices?
Answer: The object species is great crested flyeatcher.
Qualitative analysis of granularity inconsistencies in LVLMs' alignment data and constructed sample of properly aligned granularity.
Figure 14:
D.2 IMPROVING THE FINE-GRAINED DISCRIMINABILITY VISUAL FEATURES DURING THE ALIGNMENT STAGE CAN ENHINNCE LVLM PERFORMANCE ON FINE-GRAINED TASKS.
In Table we can find that the alignment strategy might impair the fine-grained discriminabil- ity of visual features. We then conduct further analysis and find that improving the fine-graie fine-grained discriminability of visual features during the alignment stage can enhance LVLM performance on fine-grained tasks.
Specifically, we compare the two variants of LLaVA from Table on fine- one-grained short-answer questions: (1) Vanilla LLaVA, where the rision-languee alignment is trained on image-text pairs with granularity inconsistencies, (2) Retrained LLaVA, where the alignment module is trained on data with matched granularity.
22

---

Published as a conference paper at ICLR 2026
Table 15: Linear prob classification results LLaVA visual features and fine-tuned results of two variants of LLaVA on fine-graines short asnwer questions.
Linear Aligned
Fine-tuned
Datasets
Origin
Aligned-FG
Vanilla LLaVA
Retrained LLaVa
CUB-200-2011
79.77
73.17
75.06
85,60
86.32
Stanford Dogs
81.24
78.14
80.69
86,49
87.58
Stanford Cars
87.57
83.90
85.63
90.55
91.73
food-101
94.27
93.35
94.32
95.25
95.74
The results in Table 15 show that Retrained LLaVA consistently outperforms Vanilla LLaVA over all datasets, indicating that improving the fine-grained discriminability of visual features during the alignment stage can enhance LVLM performance on fine-graine tasks.
Building on this finding, we believe that incorporating contrastive learning objectives (e.g., patch- or region-level contrastive loss) during the alignment stage may further help preserve discriminative visual information.
D.3 RESULTS OF CLASSIFICATION ACROSS MULTI-CATEGORIES
In Figure have shown the classification accuracy both within single super-category and across multiple metu-categories in three datasets. Here, in Figure [3] we include more results on nine tne-grained datasets. As shown in the results, EVA-CLIP, trained with contrastive paradigm, maintains a higher score in classification across multiple meta-categories compared to Qwen and BEiT3, which are trained with generative and reconstruction paradigms.
E LLM USAGE DISCLOSURE
We acknowledge the use of Large Language Model (LLM) as general-purpose assistive tool during the preparation of this paper. Specifically, the LLM was employed in a limited capacity to refine the language, improve clarity, and polish the writing. The model did not play a significant role in research ideation, conceptualization, experimental design, or substantive content generation. All core ideas, analyses, and contributions are entirely the work of the authors.
23

---

Published as a conference paper at ICLR 2026
08-29320
Buched
1973
B111
Teins
RSC Soad
Nodec
P0
Thom Kme
Serralon
light
PUGLPP
80
Figure 15: Classification results of LVLM visual features on fine-grained datasets. "Single" denotes accuracy from training on single meta-category, while "Multiple" reflects accuracy from training on unified dataset combining multiple meta-categories.
24


Reviews:
[{"review_id": "enNzSc2Mpj", "reviewer": "Reviewer_Nmtf", "summary": "This paper presents FG-BMK, a large-scale benchmark containing 3.49 million questions and 3.32 million images that systematically evaluates VLMs on fine-grained visual tasks from both human-oriented dialogue and machine-oriented representation perspectives. Across twelve domain-specific datasets, the authors demonstrate that contrastively trained encoders achieve superior subordinate-level discriminability, while generative or reconstruction-based paradigms underperform; that vision–language alignment can degrade fine-grained accuracy when image–text granularity is mismatched; and that current VLMs remain more vulnerable to adversarial perturbations and less effective at attribute-based reasoning than specialised fine-grained models, thereby identifying critical directions for future multimodal model development.", "questions": "It's recommended to conduct experiment below to improve the limitations in the weakness session.\n\n1. Expert Relabelling: Commission ornithologists, botanists or other taxonomic specialists to manually verify every species-level true/false and multiple-choice pair, ensuring that labels reflect up-to-date consensus and that linguistic ambiguities are resolved.\n\n2. Expanded Metric Suite: Complement Top-1 accuracy with Top-5 and Top-10 scores, and integrate robustness indicators such as corruption-accuracy curves or fairness-aware metrics (e.g., worst-group accuracy) to provide a richer, more nuanced assessment of model behaviour.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduction of FG-BMK, the first million-scale benchmark dedicated to fine-grained image understanding, offering paired human-style Q&A and machine-centric retrieval/classification protocols for comprehensive VLM assessment.\n\n2. Extensive empirical evidence that training objectives, modality-alignment strategies, and encoder scaling choices exert measurable, task-specific effects on subordinate category recognition and robustness, with contrastive losses consistently outperforming generative or reconstruction losses.\n\n3.Systematic disclosure of knowledge bias, granularity sensitivity, and perturbation fragility in existing VLMs, coupled with quantitative gaps relative to fine-grained expert models, providing actionable guidance for data curation and architectural refinements toward stronger fine-grained visual reasoning.", "weaknesses": "1. The ground-truth answers of FG-BMK are directly inherited from the original coarse-grained annotations and are paired with template-generated questions; no domain expert relabelling was performed. Consequently, a non-negligible proportion of species-level items contain incorrect negative labels or ambiguous attribute descriptions. A model may therefore be penalized for a prediction that is in fact consistent with expert knowledge, undermining the statistical reliability of the reported accuracy rankings. (e.g., in the Spotlight study of Chihuahuas, the term \"Japanese spaniel\" is not common knowledge, and different participants gave different answers.)\n\n2. The machine-oriented protocol exclusively reports Top-1 accuracy and mAP, omitting recently proposed robustness and fairness measures. This conservative choice restricts the diagnostic value of the benchmark and may conceal performance disparities that become evident under more lenient or distributional metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eli74MNk7d", "reviewer": "Reviewer_Uk4d", "summary": "This paper introduces FG-BMK, a new, comprehensive benchmark for evaluating Large Vision-Language Models on fine-grained image tasks, an area the authors claim is currently underexplored. The benchmark consists of 1.01 million questions and 0.28 million images, which are sourced from 12 existing, well-established fine-grained datasets. The evaluation is structured into two paradigms, a human-oriented evaluation and a machine-oriented evaluation. Based on extensive experiments on 12 representative LVLMs, the paper reports several key findings. Critically, the authors find that contrastive training paradigms produce superior fine-grained features, and that the vision-language alignment process itself can impair fine-grained discriminability.", "questions": "Please respond to the weaknesses I mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper's greatest strength is the FG-BMK benchmark itself. The dual-paradigm (human-oriented and machine-oriented) evaluation is effective, allowing for a holistic assessment of both conversational understanding and raw feature quality.\n2. The paper delivers more than just a leaderboard. It uncovers several novel and important insights.\n3. The paper is exceptionally well-written, with clear figures and tables that make the complex results easy to digest.", "weaknesses": "1. The \"machine-oriented\" evaluation provides the paper's most novel and important insights. However, this evaluation requires access to internal model features, which is impossible for closed-source models. As a result, this entire, crucial part of the benchmark cannot be applied to SOTA models like GPT-4o and Gemini.\n2. The paper contains a clear factual error. The Ethics Statement explicitly claims the benchmark \"does not cover sensitive areas like medical imaging,\" but Table 6 in the appendix clearly lists the \"SkinCon\" dataset, which it explicitly labels as \"Dermatology\".\n3. The abstract's claim of \"comprising... 0.28 million images\"  could be mildly misleading, implying a new image collection. The core contribution is the 1.01 million questions and the evaluation framework, which are built on 12 existing public datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xzyZT53ymW", "reviewer": "Reviewer_xLqa", "summary": "This paper introduces FG-BMK, a large-scale benchmark designed to provide a fine-grained evaluation of Large Vision-Language Models (LVLMs) from both human-oriented and machine-oriented perspectives. The benchmark contains 1.01 million questions paired with 0.28 million images, covering 12 fine-grained visual datasets. The human-oriented evaluation focuses on assessing the model’s ability to understand and respond to fine-grained visual queries in conversational contexts. The machine-oriented evaluation measures model performance on two standard fine-grained vision tasks: image retrieval and image recognition.", "questions": "1. Novelty and differentiation: As mentioned in the first weakness, there are already numerous LVLM benchmarks covering fine-grained evaluation. Could you elaborate on the specific advantages or unique contributions of FG-BMK compared to existing benchmarks such as MERLIM (Villa et al., 2023), HallusionBench (Guan et al., 2023), MMVP (Tong et al., 2024), MMBench (Liu et al., 2024), and AMBER (Wang et al., 2023)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well structured and easy to follow.\n\n2. This paper attempts to address an important topic, the evaluation of Large Vision-Language Models (LVLMs) on fine-grained visual tasks. The authors propose a benchmark, FG-BMK, by curating questions over 12 existing fine-grained datasets.\n\n3. It provides a large set of pair of questions-images to comprehensely evaluate LVLMs in different relevant tasks, like Attribute Recognition, Hierarchical Granularity Recognition, etc.", "weaknesses": "1. Limited novelty: While the topic is relevant, the paper’s novelty is limited. Several recent benchmarks, such as MERLIM (Villa et al., 2023), HallusionBench (Guan et al., 2023), MMVP (Tong et al., 2024), MMBench (Liu et al., 2024), and AMBER (Wang et al., 2023) have already evaluated LVLMs across similar tasks. The paper does not clearly explain how FG-BMK differs from or improves upon these existing benchmarks.\n\n2. Limited model evaluation: The set of evaluated models is small and outdated. The study omits many recent (like LLaVA-OneVision, LLaVA More, EAGLE 2.5) and commercially competitive LVLMs (like, GPT-4o and Gemini).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIFa1rTLtH", "reviewer": "Reviewer_cSv5", "summary": "This paper introduces FG-BMK, a comprehensive benchmark designed to evaluate Large Vision-Language Models (LVLMs) on fine-grained image tasks, which remain underexplored compared to generic visual reasoning or captioning tasks. FG-BMK comprises 0.28 million images and 1.01 million automatically generated questions sourced from 12 established fine-grained datasets such as CUB, Stanford Cars, and FGVC Aircraft. The benchmark features two complementary paradigms:\n\n1. Human-oriented evaluation: measures LVLMs’ ability to answer fine-grained visual questions in conversational form (attribute recognition, hierarchical granularity reasoning, and knowledge bias estimation).  \n2. Machine-oriented evaluation: measures the discriminability and robustness of visual representations via retrieval and classification tasks.\n\nThe study evaluates 12 representative models (GPT-4o, Gemini-2.0-flash, Qwen2.5-VL, InternVL3, LLaVA, DINOv2, etc.) and provides systematic analyses. Main findings include:  \n- Contrastive training (e.g., CLIP, DINOv2) yields better fine-grained discriminability than generative or reconstruction paradigms.  \n- Text-image alignment can harm fine-grained distinctions, especially when text granularity mismatches image detail.  \n- LVLMs are more vulnerable to perturbations in fine-grained domains than generic vision tasks.  \n- Scaling model size or data volume brings limited gain compared to data quality and alignment design.  \n- LVLMs perform well on appearance description but lag behind domain-specific fine-grained models.", "questions": "See Weakness.\n\n- Additionally, can the authors provide more details on the PGD attack setup for feature perturbation? L∞ PGD requires specifying an upper bound for the perturbation. Do all encoders use the same upper bound?\n\nOverall, this paper delivers insightful analyses of LVLMs on fine-grained visual tasks, with solid and detailed evaluations from both semantic and feature-based perspectives. I recommend acceptance and would consider raising the score if the above concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Proposes a large, reproducible benchmark that systematically covers multiple aspects of fine-grained vision-language understanding.  \n- Provides clear empirical findings with actionable insights, notably about alignment granularity and contrastive pretraining.  \n- Includes both human-style and feature-based evaluations.\n- Reveals interesting findings for LVLM's behaviour in fine-grained tasks.\n- Gives insights for how to design and train vision representation encoders in the context of improving fine-grained perception of LVLM", "weaknesses": "- Primarily an evaluation paper; lacks methodological novelty or a new model proposal. I agree that a benchmarking and analysis paper can bring valuable insights to the field. However, since this paper already proposes several small fixes (fine-tuning on balanced fine-grained datasets, improved alignment strategies), could these be integrated into a more systematic approach that contributes to improving MLLMs?\n\n- Another concern is that automatic question generation may limit linguistic diversity and realism compared to human-written prompts. There are only a few chat templates shown in Appendix A.2. Could the use of more natural prompts affect performance?\n\n- Lack of a systematic comparison. Although this paper evaluates many mainstream models, it lacks a clear comparison of their results on FG-BMK (for example, an average of all sub-benchmarks on FG-BMK). A clear comparison would be important for readers to understand the performance and relative ranking of different LVLMs.\n\n- The benchmark is sourced from existing public datasets, which raises concerns that different LVLMs may have been trained on overlapping portions of these datasets. This could make fair performance comparison difficult. Can the authors provide an analysis of this potential train-test overlap issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)