Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models
Authors: Hulingxiao He
Geng Li
Zijun Geng
Jinglin Xu
Yuxin Peng
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Models
Fine-Grained Visual Recognition
Primary area: foundation or frontier models, including LLMs

Abstract: Multi-modal large language models (MLLMs) have shown remarkable abilities in various visual understanding tasks. However, MLLMs still struggle with fine-grained visual recognition (FGVR), which aims to identify subordinate-level categories from images. This can negatively impact more advanced capabilities of MLLMs, such as object-centric visual question answering and reasoning. In our study, we revisit three quintessential capabilities of MLLMs for FGVR, including object information extraction, category knowledge reserve, object-category alignment, and position of the root cause as a misalignment problem. To address this issue, we present Finedefics, an MLLM that enhances the model's FGVR capability by incorporating informative attribute descriptions of objects into the training phase. We employ contrastive learning on object-attribute pairs and attribute-category pairs simultaneously and use examples from similar but incorrect categories as hard negatives, naturally bringing representations of visual objects and category names closer. Extensive evaluations across multiple popular FGVR datasets demonstrate that Finedefics outperforms existing MLLMs of comparable parameter sizes, showcasing its remarkable efficacy. The code is available at [https://github.com/PKU-ICST-MIPL/Finedefics_ICLR2025](https://github.com/PKU-ICST-MIPL/Finedefics_ICLR2025).

Published as a conference paper at ICLR 2025
ANALYZING AND BOOSTING THE POWER OF FINE- GRAINED VISUAL RECOGNITION FOR MULTI-MODAL LARGE LANGUAGE MODELS
Hulingxiao He', Geng Li', Zijun Geng', Jinglin Xu', Yuxin Peng
"Wangxuan Institute of Computer Technology. Peking University School of Intelligence Science and Technology, University of Science and Technology Beijing Miehulingxiann ) stt..tuu.u...... pku. edu. en, gengzi jun20240160 jeng21jun2024@11 com sujinggiinooee com, pengyuxin@pku. edu.cn
ABSTRACT
Multi-modal large language models (MLLMs) have shown remarkable abilities in various visual understanding tasks. However, MLLMs still struggle with fine- grained visual recognition (FGVR), which aims to identify subordinate-leved cat- egories from images. This can negatively impact more advanced capabilities of MLLMs, such as object-centric visual question answering and reasoning. In our study, we revisit three quintessential capabilities of MLLMs for FGVR, includ- ing object information extraction, category knowledge reserve, object-categor, alignment, and position of the root cause as misalignment problem. To ad- dress this issue, we present Finedefics, an MLLM that enhances the model's FGVR capability by incorporating informative attribute descriptions of objects into the training phase. We employ contrastive learning on object-attribute pairs and attribute-calegory pairs simultaneously and use examples from similar but in- correct categories as hard negatives, naturally bringing representations of visual objects and category names closer. Extensive evaluations across multiple popular FGVR datasets demonstrate that Finedefics outperforms existing MLLMs of com- parable parameter sizes, showcasing its remarkable efficacy. The code available at htttts::// tww.bttttuuuggmmo htttpp:://itthbbbcco//////// www.://POISTTTTPPPPPCII c cs TCLR2025
INTRODUCTION
Multi-modal Large Language Models (MLLMs) Bai et af. 2023 Chen et al.] 2023) Zhang et al,, 2023b] Zhu et al., 2023 Dong al. 2024 Liu al.) 2024b Laurençon al.] 2024a b) have achieved remarkable advancements in understanding visual data, showcasing potential in advancing general artificial intelligence. These models enable users interact with images inputs, fostering seamless communication grounded in visual information. The impressive capabilities allow MLLMS excel various vision tasks while adeptly handling complex content comprehension and genera- tion. However, despite their versatility and linguistic proficiency, MI.LMs still face challenges in fundamental task of machine vision: fine-grained visual recognition (FGVR) (Zhang al.) 2024bb Geigle et al. 2024). which aims at identifying subordinate-level categories, such specific species of animals or plants (Wei et al. 2021). Poor FGVR performance of MLLMs hinders them from per- forming more advanced tasks like object-centric visual question answering and reasoning (Zhang ct al. 2024b). For example, in smart agriculture, poor FGVR performance of pests may lead to incorrect treatment strategies and large-scale reduction in food production.
Early works have investigated the phenomenon (Zhang et al.) 2024b) and attempted to improve the FGVR performance of MI.MMs by integrating open-set classification data into pre-training or fine-tuning stage (Geigle et al.) 2024 Zhang et al.; (0244)). However, fine-tuning solely on the classification task harms the general capability of the instruction following, while purely integrating clasiffication-fcussse data into the instruction tuning data brings limited improvement (Geigle et al., 2024]. making their direct utilization impractical. To understand why MLLMs underperform in
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Give a detailed deseription of object in this photo.
What breed of cat is photo?
Give detailed description of Bengal Car.
The car's coat tan and spotted
Rengal Cat's enat ton brown spotted
X Abyasinian.
pattern and a soft texture
pattern and a soft texture
Object Information Extraction
(b) Category Knowledge Reserve
Object-Catggory Alignment
Three quintessential capabilities of MLLMs for fine-grained visual recognition. Current MLLMs possess acceptable capabilities in image information extraction and category knowledge reserve but struggle with aligning objects to their corresponding subordinate-level categories.
Figure I:
FGVR, we revisit and evaluate three quintessential capabilities of MLLMs as shown in Figure (a) Object information extraction.
D
essential accurately and fully extract the necessary information for distinguishing objects. (b) Category knowledge reserve. MLLMs should reserve sufficient knowledge of subordinate-level categories. (c) Object-category alignment. With rich visual information extracted and sufficient category knowledge reserved, visual objects and category names should be aligned in the representation space to enhance classification performance.
We analyze the representation space of MLLMs and their corresponding visual language models (VLMs) like CLIP (Radford et al. [021], revealing that: (1) Object information lost exists between VLMs and MLLMs but is not the bottleneck. During the propagation of object features output from the vision encoder in modality connector and language model layers, the necessary visual informa- tion for distinguishing objects is almost preserved. Our observation is consistent with object hallu- cination in MILMs (Zhou et al., (023), which finds that the modality connector tends to decrease the representation diseriminability, while LLM has significant impact. (2) Category knowledge is relatively sufficient, but category names cannot fully capture the semantics. Existing LLMs utilized in MLLMs can output detailed and distinguishing descriptions about subordinate-level categories, but category names are not discriminative in the representation space of LLMs. (3) Misalignment between the visual object and category name leads to underperformance Despite a slight reduction in both object and category representation discriminability, the current learned modality connector insufficient for effectively matching visual object representations to subordinate-level category names, as most VLMSs do.
Motivated by the aforementioned analysis, we propose Finedefics, an MLLM designed to enhance the model's ability to identify subordinate-level visual object categories. Our framework builds upon Idefics2 (Laurençon et al. (2024)) and is specifically tailored to boost the power of FGVR. To facilitate alignment between visual objects and category names, descriptions summarized from information visual attributes are utilized as the intermediate point to bind them in the representation space of L.Mss. Concretely, we separately feed token sequences of visual objects, attribute de- scriptions, and category names into MLLMS to obtain global representations from the last layer of LLMs respectively. Contrastive learning is performed on global representations of abject-attribute pairs and attribute-category pairs simultancously, with additional hard negatives from similar but incorrect categories, bringing representations of visual objects and category names closer. Subse- quently, being trained solely on both open-set and closed-set FGVR data through instruction tuning, Finedefics demonstrates exceptional FGVR performance gains. Benefiting from attribute augmented alignment, Finedefics outperforms established counterparts across six popular FGVR datasets and notably surpassing Idefics2 (Laurençon et al. 2024b) and Qwen-VL-CCat (Bai al. 2023) by an average of +10.89% and +9.43%, respectively.
In summary, our contributions are as follows: (i) We revisit the quintessential capabilities MLLMS for FGVR and investigate the root cause of underperformance in FGVR: misalignment between vi- sual objects and category names. (ii) We propose Finedefics for enhancing the model's FGVR ac- curacy, which uses informative attribute descriptions to effectively align visual objects and category names in the representation space of LLMs. (ii) With extensive experiments on six popular FGVR datasets, we demonstrate the superiority of Finedefics.
2

---

Published as a conference paper at ICLR 2025
Dipss
dag
(a) Object dist. of SigLIP.
(b) Category dist. of SigLIP.
(c) Object-category dist. of SigLIP
Jeat Campgry
Cateprr
Josrace spase
34
(d) Object dist. of Idefics2.
(c) Category dist, of Idefics2.
(f) dist. of Idefics2.
Figure ObjeciCCateerryObiicccaaagog representation visualization of SigLIP and Idefics2.
2 WHY Do MULTI-MODAL LARGE LANGUAGE MODELS UNDERPERFORM IN FINE-GRAINED VISUAL RECOGNITION?
This section scrutinizes the root causes of underperformance in FGVR via comprehensive empiri- cal analyses on three quintessential capabilities of MLLMs: object information extraction, category knowledge reserve, and object-category alignment by comparing the representation space with cor- responding VLMs (Idefics2 (Laurençon et al. 2024a and SigLIP (Zhai et al.) in our experiments).
Assuming image containing an object O, is processed by the vision encoder V. and learnable modality connector F, 3 to be transformed into visual object token sequence of length m: S: = - oz. I Input category name in textual modality C, is passed through an embedding layer Eg of the LLM obtain the category embedding sequence of length n: St [c] c2, ci..... c. Subsequently, the object embedding sequence St and category embedding sequence St are individ- ually passed through the LLM layers Lo to obtain the output from the last layer:
Notations.
H (S))
(la)
H (S)),
(lb)
where Hi ( of,...,, J. and H!(1) [24, .., 2. Afterward, select two ways to represent the global semantics of output sequence following (Zhang et al. 2024b): 1) last token embedding o' et and (b) average of the token embedding sequence o² (E om/m, ट
(Ei Gn For VLMs, the projected [CLS] embedding outputs from last layer of vision encoder V. and text encoder T., are taken represent the global semantics of O, and C. denoted ot and ccss respectively.
2.1 OBJECT INFORMATION EXTRACTION
In the task of FGVR, a model that excels at object information extraction is required to have dis- criminative representations, i.e., large inter-class distance and small intra-class variance. To com-
3

---

Published as a conference paper at ICLR 2025
Table 1: Feature probing on Idefics2 and SigLIP with features of objects and category descriptions.
(a) Object features.
(b) Category description features.
Model
Feature Type
Acc.
Model
Feature Type
Acc.
Last
94.99
Last Avg.
92.51
Idefics2
Idefics2
Avg.
90.24
90.41
CLS
95,28
CLS
84.70
SigLIP
SigLIP
Avg.
94.44
Avg.
87.78
pare the object representation space of SigLIP and Idefics2, we select three subordinate-level cate- gories "Chi huahua", " "Japanese spaniel", "Maltese dog") "I from Stanford Dog-120 (Krause et al. 2013), and randomly sample 100 examples per category for t-SNE (Van der Maaten & Hinton 2008 visualization. As shown in Figure 2al since the vision encoder V normally frozen throughout training (Liu et al. 2024b a), the output object token sequence S preserves discrimina- tive information for classification. We then hypothesize that the information is lost after propagating through the modality connector 3 and L.M layers Lg. However, various objects belonging to the same subordinate-level categories can still cluster together and distance from each other, illus- trated in Figure 2d] To quantitatively compare the representation discriminability, we use feature probing experiments (Zhang et al., (024b) test the hypothesis. Concretely, on top of the last token embedding o'n or average of the token embedding sequence o', we train linear classifier on the training set of Oxford-IIIT Pet-37 (Parkhi et al.) 2012) and evaluate on the test set. Table Ia we observe that although information is lost, the impact on the performance is limited.
2.2 CATEGORY KNOWLEDGE RESERVE
Trained on enormous internet-scale corpora, LLMs are known for encoding the expert knowledge for general categories in their weights, but ask ourselves, is the expert knowledge quintessen- tial for FGVR already contained in MILMs? We hypothesize that MLLMs' underperformance in FGVR tasks stems from the inadequate knowledge of subordinate-level categories. To test the hy- pothesis, we investigate whether LLMs utilized in MLLMs can distinguish different categories by generating discriminative descriptions. Specifically, we probe the knowledge Idefics2 via us- ing the prompt ["Give a brief description of distinguishii (CLASS NAME]"). For each subordinate-level category in Oxford-IIIT Pet-37 (Parkhi et al., 2012), we set the number of return descriptions to 200 and then equally divided them into the train set and test set. The class names in returned descriptions are replaced with demonstrative pronouns to avoid leakage of classification labels. Similarly, we conduct linear probing experiments top of é" and e² As shown Table Ib Idefics2 exhibits better classification performance than the text encoder of SigLIP, demonstrating its superiority in reserving category knowledge. Despite the rich semantics the generated category description, the category names have lower discriminability in the representation space of 1defics2 than the text model of SigLIP, illustrated in Figure 2b and 2e
features of
2.3 OBJECT-CATEGORY ALIGNMENT
Since our empirical study shows that Idefics2 has an acceptable capability of object information extraction and adequate knowledge of subordinate-level categories, we hypothesize that the mis- alignment between the visual object and category name is the root cause. We randomly sample 120 object-category pairs from Stanford Dog- Dog-120 (Krause et al., 2013), and visualize the distributions of the last token embedding of the object or and the category name ét in the same representation space. As shown in Figures 2c and 21, object and category representations have significant semantic gaps. Since category names may not fully represent the semanties of the visual data (Lyu et al. 2024]. the object cannot match the ground-truie -ruth category in the representation space and thus fails to decode into the correct category name.
4

---

Published as a conference paper at ICLR 2025
Useful Attribute Discevery
Stage I: Attribate Augmented Contractive Learning
Stage Classifcatiinn
Centored Instraction Toning
Atbbtt Description Generation
ObjectAttribtte
Attribute C Categary
distingiihing specitle pizto
Centraative Learsing
Coatradive Learnite narninn
Open-ect Closad-szt
Cheeiforoo
coot cout, gattent
LLM
LLM
Categeyy Calizeey Contrastive Leariing
Label with value
Visual Actribure Extraction
VB
Label with valler
VI
What the coat colarreent patteral for resture of the cot in this photo?
No Label
[B0S]
[BOS]
[EOS)]
[8.08]
[ROS]
O
lan and locress spatiil putlicn sostare
VQA Madel
Large Language Model (LLM)
Attribute Description Summorication
Suromarzo the inforzation from the ytu Prate descriptio: [coat coler: tan and browa]
Modality Connector
u.. cats coot spotted textare [BOSS
brown
cat's ftufly pattern, [B05]
Rengal [eO8]
[EOS]
Vision Encoder
oafe is taa and
brewn wilh a spatted poltern and soft Testure...
Poeilve
Hond Negetio Anriuat Description
Proiive Congory Name
[and Negrive
LLM
Vieua Otjeet
Atribune Description
Carpgory Narse
(4) Attribute Deseription Construction
(b) Attribute Augmented Aligament
An illustration of framework to build Finedefics. (a) Attribute Description Construction, which aims obtain informative attribute descriptions of objects. (b) Attribute Augmented Align- ment, which aims to use constructed attribute descriptions to bind visual objects and category names, thus enhancing the model's FGVR capability via a two-stage training paradigm.
Figure 3:
METHOD
After thoroughly investigating the root cause of the underperformance in FGVR, this section for- mally introduces Finedefics, which enhances the model's FGVR performance by better aligning visual objects and category names. The framework to build Finedefics is illustrated in Figure com- posed of two key components: (1) Attribute Description Construction for extracting useful attribute information that can distinguish different categories.
(2) Attribute Augmented Alignment dedi- cated to using constructed attribute descriptions as the intermediate point to bind visual objects and category names in the representation space of LLMs, thus boosting the subsequent Classification- Centered Instruction Tuning.
3.1 ATTRIBUTE DESCRIPTION CONSTRUCTION
Although it has been demonstrated in (Liu 2022 El Banani al. 2023) that language is
a powerful tool for capturing semantic relationships. dependency on category names exclusively align with extracted visual embeddings is unreliable. As discussed in Section 2.2 leveraging adequate knowledge of subordinate- categories, LLMs can describe distinguishing features that better capture category semantics than category names. Inspired by (Liu al.) (024c) that exploits cascade of foundation models to translate useful visual information from visual to textual modality, we propose constructing sample-win wise attribute descriptions for each FGVR training set.
Specifically, the construction comprises three steps: 1) Useful Attribute Discovery by LLMs, such as GPT-4 (Achiam et al., (202)) and LLaMA TTovrron et al. 2023). These attribute names are employed as keys to instruct Visual Question Answering (VQA) models (such as BLIP-2 (Li et al. 2023) and
LLaVA (Liu et al.] (024b)) for extracting useful attribute values. 2) Visual Attribute Extraction by VQA models. These attribute values enrich the information for distinguishing subordinate-les
categories. 3) Attribute Description Summarization by LLMs. These descriptions help alleviate the gaps between visual objects and category names in the training phase of Finedefics.
5

---

Published as a conference paper at ICLR 2025
Userful Attribute Discovery. Given the super-category of each FGVR dataset, such as aircraft for FGVC-Aircraft (Maji et al. 2013), we first identify a useful set of attributes that set apart the subordinate-level categories. For example, the wing shape attribute can help distinguish various aircraft models. To discover such key visual cues, we tap into the expert knowledge of L.Mss, which is otherwise only restricted experts. Specifically,, ask LLMs: ["Your task is to tell me what are the useful attributes for dist Eistinguishing (SUPERCLASS) {CLASSUNIT} in a photo of [SUPERCLASS] Formally, LLM takes super-category Csup as input and outputs list of useful attributes:
NC = (Plis (Csup)
(2)
where NC = {N Now) are the generated attribute keys for the category Cup- Lo are LLM layers, and pdis is the How-to LLM-prropp in (Liu et al. 2024c).
Visual Attribute Extraction. With the discovered attribute names New, leverage VQA models that excel at identifying general visual attributes (e.g.. shape, color) of objects to extract each attribute value per sample. For example, if attribute is wi wing shape, VQA models are prompted to give a brief description of the wing shape, which is a much easier task than recognizing many subordinate-level categories. Following (Liu et al. 2024c). we add gen- eral attribute name No = ("General description of the image"] and its prompt Poxt ("Questions: Describe this image in details. Answer: "]. For- mally, VQA model takes as input an image It its super-category C'up and the attribute names N the output visual attributes given as:
V Q (I.. PeulPe Pea((NC)))
(3)
where V; (Vi...., V denotes the extracted set of visual attributes for image 7; Q. is the VQA model, and Pex the Identify VQA-prompt in (Liu et al. 2024c).
Attribute Description Construction. After obtaining the structured set of attribute key- value pairs, we further ask LLMs: ("Summarize the information ion you get about the (SUPERCLASS) from the general description and attribute description with five sentences. "]. The summarized attribute description contains richer semantics of subordinate-level categories, making it much easier for LLM to understand.
Formally, given the set of attribute names N and attribute values V,, LLM outputs a summarized attribute description for image I;:
A, = Lo (P (Non, vi)).
(4)
where A; the attribute description constructed for image 1;, and Poon the revised Reason LLM- prompt in (Liu et al., 2024c) for summarization task only. Expanding upon our newly built attribute descriptions, we transfer traditional (object, category) pairs FGVR datasets to (object, attribute, category) triples. Without specification, the category refers to the subordinate-level category instead of the super-category in subsequent sections.
3.2 ATTRIBUTE AUGMENTED ALIGNMENT
With the constructed informative attribute descriptions, we introduce new training paradigm named Attribute Augmented Alignment to build our Finedefics. It comprises two stages: (I) Attribute Aug- mented Contrastive Learning for aligning visual objects and category names in the representation space of L.Ms.. (II) Classification-Centeee Instruction Tuning for enhancing the model's ability to follow the FGVR task instruction.
Stage I: Attribute Augmented Contrastive Learning. For each object-attribute-caan
triple (O., As, Ci), we utilize the vision encoder V. and the learnable modality connector Fa trans- fer O into an object embedding sequence of length S Joi of of..... .f, with length m. To better capture the glohal representations, we follow (Jaag et al.] 2024) to pass an [FOS] to- ken through embedding layer E, of LLMM to obtain the vector representation and append it to the visual embedding sequence St Therefore, obtain the newly built object embedding quence $3 [ot, o2, ..,,,,, on oEos). Similarity, obtain the attribute embedding sequence $

---

Published as a conference paper at ICLR 2025
[ai, al...., ap. 4sss) with length (p+1), and category embedding sequence S: [c),cl, .... en cos with length (n + 1). Then, 5% St. a: St. are individually fed into LLM layers Lo. and the embeddings of the last predicted token Ofos ooos: Fos are utilized as the global representations of O₁, Ai, Ci, respectively. Without specified, we use 0' ooss a' = =fas= e = Eos for simplicity.
To improve the effectiveness of contrastive learning. we then mine difficult incorrect category names for each example object Oj used in the FGVR dataset. To do this,
CLIP model (Radford ct al. 202T) for mining hard negative samples: for every example image, we select three images along with their attribute descriptions from the three most similar but incorrect categories. At- tribute descriptions and category names from these hard negative samples are subsequently treated additional negatives. Thus, the formulation of Object-Atrrbute Contrastive (OAC) loss with the inclusion of hard negatives can be described as follows:
Sin (am (0*,at)
Co
Σ
log Σ
exp
(5a)
exp
exp Sim(l
+
Σ
exp Sim(
(8)
WE
8 E
Sim Sim(""'a")
LAO
Σ
log
exp
(5b)
Σ
exp Sim (ak,
(B
c.m Cocc = (Cob + Lao)/2,
(5c)
where Ain denotes the attribute representation set of hard negatives for the object O, Sim(:,. measures the cosine similarity in a semantic space.
Similarly, Attribue-Categor Contrastive (ACC) loss with the inclusion of hard negatives formu- lated as follows:
exp Sim(a" ,ea)
Chu =
Σ
llog
(6a)
Σ
Simla 00) + Σ
exp expSim
Sim(at,
(0),ar,e))
BEB
exp Sim(( (amm(("''))) .2t)
CCA
log
(6b)
Σ exp Sim(ar,
E expmim(a=,ea)
(ō) (
EB
Acc (C'x + Cloa)/2
(6c)
where Ci., denotes the category representation set of hard negatives for the object O₁.
As discussed in Section 2.2 it is hard to differentiate between category names in the representa- tion space of LLMs. Inspired by the intra-modal contrastive loss to promote the model's ability to differentiate between hard nagative captions (Zhang et al. 2024a), we additionally define Category- Category Contrastive (CCC) loss as follows:
Lcco =
Σ
log Σ
expSsm(ee,et eak"
(8
To maintain the generative power of the model, we use the attribute descriptions as LLM-augmented captiona formulate the attribute description generation task. Therefore, the optimization object of the first stage can be defined as follows:
O 3,0 = arg min Cat + (Chrnc + Circe + Lccc)/2,
(8)
3,8
where Call denotes the attribute description generation loss.
Stage II: Classification-Cenle Centered Instruction Tuning. In the second stage, we formulate the FGVR dataset as two kinds of instruction tuning data: open-set QA data and closed-set multiple- choice data. Then we fine-tune the model using this classfiication-centeee Consequently, the optimization object of the second stage can be formulated as:
instruction tuning data.
O!l 3.0
arg min ccls
99)
R.0
where Lale denotes the generation loss of clasiffication-centeeed instruction tuning data.

---

Published as a conference paper at ICLR 2025
Table Comparison with leading methods on six FGVR datasets. #P denotes parameters count.
Model
#P
Dog-120
Bird-200
Aircraft-102
Flower-102
Pet-37
Car-196
Avg.
LLaVA 1.5
7B
38. 38.96
35.24
34.71
51.37
52.25
46.92
43.24
LLaVA-Next (Mistral)
34.88 88
32.49
53.72
42.22
VL. v2
7B
39.92
33.90
35.01
54.89
53.69
46.29
43.95
InstructBLIP Vicuna
7B
41.60
32.78
31.68
30.90
54.92
48.25
43,36
InstructBLIP Flan-TS-XL.
47.10
32.15
29.19
62.29
64.58
49.22
Phi-3-Vision
4B
39.80
37.63
42.33
51.59
56.36
$4.50
47.04
BLIP2 Flan-TS-Xe Ta-XT
4B
46.17
33.70
32.94
64.32
65.00
67.68
51.64
InternLM XComposer 2
41.47
40.53
63.23
53.89
48.47
Pali-Gemma
51.68
36.62
39.87
69.64
75.42
64.64
56.31
Ideficsl
9B
39.74
36.50
34.62
51.70
48.51
29.42
40.08
Ideficx2
57.90
47.17
56.23
72.78
81.28
80.25
65.95
Qwen-VL-Chat
10B
66.18
52.30
45.96
75.95
87.82
76.23
67.41
Finedefics (ours)
72.86
57.61
63.82
89.88
92.18
84.67
76.84
(+6.68)
(+5.31)
(+7.59)
(+13,93)
(+4.36)
(+4.42)
(+9.43)
EXPERIMENTS
In this section, we evaluate the performance of Finedefics aiming to answer the following questions: (1) Can Finedefics effectively improve FGVR accuracy in MLLMs? (2) Does each core design of Finedefics benefit the accuracy improvement? (3) Is Finedefies effective in aligning visual objects and category names in the representation space of L.MMs?
4.1 IMPLEMENTATION DETAILS
Datasets. We conduct experiments on several popular FGVR datasets that include CaltechUCSD
Bird-200 (Wah al.) 2011), Stanford Car-196 (Krause et al. 2013), Stanford Dog- 120 (Krause
et al. 2013), Flower- 02 (Nilsback & Zissermann 2008), Oxford-IIIT Pet-37 (Parkhi et al.) 2012). and FGVC-Aircraft (Maji et al. 2013). Following (Geigle al. 2024). leverage the test sets as resources for annotated data and frame FGVR as multiple-choice task with well-defined answer candidates. To facilitate Finedefics, tion, build open-set QA and closed-set multiple-choice instructing tuning data, ensuring that these images are different from the ones used in testing.
select the training sets of them to construct attribute descrip-
Evaluated MLLMs. We build Finedefics upon Idefics2 [Laurençon et al.)
2024b) for its open-
source accessibility and leading zero-shot performance. Several recent MIL.Ms of comparable parameter sizes are evaluated, including LLavV 1.5 (Liu et al., (024b). I.La VA-Next (Liuu et al.) 2024a), MobileVI.M
2024), InstructBI.IP Flan-T5-XL (Dai al.) 2024). Pision (Abdin et al.] 2024). BLIP2 Flan-T5-XL (Li et al.
(Chu et al., 2024). InstructBi.IP Vicuna (Dai et al.
Idefics (Laurençon et al, 2024a). Idefies2 [Laurençon et al. 2024b)), and Qwen- Owen-VL-Char VL-Chat (Bai et al. 2023).
2023 InternLM XComposer (Dong al. 2024). Pali-Gemma
Training Settings. All seeds are fixed across the training procedures for fairness. We train Finedefics using the QLoRa technique (Dettmers et al., (024). updating adapters in the LLM and modality connector including perceiver resampler with NVIDIA A6000 GPUs with 48G of mem- ory. We 4-bit quantization, with y 8 and 8 for LoRa, and learning rate of 2e-4. For both stages, the model is trained for one epoch with the warming steps of 60. The accumulated batch size set to 64 and 128 for stage and stage respectively.
4.2 MAIN RESULTS
compare Finedefics with previous leading approaches six popular FGVR datasets. Finedetics exhibits a significantly enhanced FGVR capability compared to a wide range of MLLMs. Notably, Finedefies shows superior performance than Idefics2 by average of +10.89% and Qwen- VL-Chat of +9.43% across all datasets. Note that Finedefics is built upon Idefics2 (Laurençon) 20245). high-performnggng model various vision-language and vision-centrie tasks, and the
In Table
8

---

Published as a conference paper at ICLR 2025
Table 3: Analysis of Finedefics.
"Original" represents the zero-shot performance of Idefics2.
(a) FT methods.
(b) Effectiveness of attributes.
(c) Training paradigm.
Method
Avg.
Method
Avg.
Method
Avg.
Original
65.95
Original
65.
Original
65.95
Finelune
0.03
CL (obj-cat.)
72.72
One stage
25.42
Finedefics (ours)
76.84
CL (obj-att.-cat.)
76.84
Two stages
76.84
Diee Catagoy
Dise Calagor
Died
Catagry
1100
4150
4/50
(a) Finetune
(b) CL (object-category)
(c) Finedefics (ours)
Figure 4: Representation visualization of Finetune, CL (object-category) and Finedefics.
enhanced performance on FGVR makes it valuable foundation to benefit more advanced tasks with finer granularity.
4.3 ANALYSIS OF FINEDEFICS
Does Fine-tuning Solely on Additional Open-set FGVR Data Bring Performance Gains? In Table 3a we fine-tuning Idefics2 solely on additional open-set FGVR data. We observe that it deteriorates the instruction following capability for answering multiple-choice questions in our test settings. Finedefics outperforms the fine-tuned model, which indicates that Finedefics effectively boosts FGVR accuracy by integrating attribute augmented alignment into the training paradigm rather than solety fine-tuning on additional data.
Does Attribute Descriptions Contribute Performance Gains of Contrastive Learning? To demonstrate the impact of augmenting contrastive learning with attribute descriptions, conduct experiments and report the results in Table 3b In the ablation experiments, we employ contrastive learning object-category pairs without utilizing attribute descriptions. The results show that the attribute description benefits the alignment between visual objects and category names.
Is Training in Two Stages Necessary for Building Finedetics? We further analyze the necessity of training in two stages, i.c., representation alignment before instruction tuning. Specifically, we fine-tune Idefics2 with a combined loss of classiiiation-centtre instruction tuning and attribute augmented contrastive learning. The results are reported in Table 3c We observe that fine-tuning in one stage is prone struggle with optimization and leads to degraded performance, which indicates the effectiveness of training in two stages.
Visualization Does Finedefics Effectively Align Visual Objects and Category Names? To substantiate our objective of enhancing the alignment between visual objects and category names with the auxiliary visual attributes, we randomly selected 100 data from Oxford-IIIT Pet-37 (Parkhi et al. 2012) for visualization. As illustrated in Figure 4a substantial gap between the object and category is observable in the data distribution when fine-tuning without contrastive learning. In Figure 4b] contrastive learning object-category pairs without attribute descriptions involved fails to decrease the gap. In Figure 4c with the usage contrastive learning on object- bottect-tttriutteeeee -attbut triples, the gap decreases significantly, thus boosting FGVR accuracy.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Multi-modal Large Language Models. Multimodal Large Language Models (MLLMs) aim to enhance machines' ability to understand and process complex information by integrating multiple data modalities such as vision, text, and audio. In recent years, MLLMs have achieved significant progress three key areas. First was large-scale pre-training and fine-tuning, as seen models like BLIP-2 (Li et al. 2023), LLaVA (Liu et al.] 2024b), MiniGPT-4 (Zhu et al., 2023), PaI.M-F (Driess) ct al. 2023). Kosmos-2 (Peng et al. 2023) and Visual ChatGPT (Wu et al. 2023), which used pre- training on vast multimodal datasets and were then fine-tuned for specific tasks, greatly improving the models' generalization ability and task performance. The second area was cross-modal consis- tency, focusing
ensuring information consistency across different modalities through techniques like contrastive learning. Models such as Shikra (Chen et al.) 2023), FROMAGe (Koh et al.) 2003), DLP (Tian al. 2024), BuboGPT (Zhao et al. (023b), ChatSpot (Zhao et al. 2023a), and Qwen- VL [Bai 2023) enhanced performance in multimodal tasks by strengthening the alignment between modalities. The third area was interpretability and transparency. Models like ViperGPT (Surís al., 2023). GPT-4 (Achiam et al. (023)) PandaGPT (Su et al., (023). Video-LLaMA (Zhang et al. 2023a), and Video-ChatGPT (Mazz et al.] 2023) enhanced the explainability of model decision- making by incorporating attention mechanisms and natural language feedback, enabling users to understand better and trust the model's output. Despite these achievements, MLLMs still face chal- lenges, such the inability to extract informative visual features, insufficient understanding of subordinate-level categories, and misalignment between visual objects and category names.
Fine-Grained Visual Recognition. FGVR (Welinder et al. 2010 Maji et al. 2013) Wei et al.) 2021) aims to classify visually similar subordinate categories under a broader super-category often requiring expert- prrvidddd auxiliary annotations (Krause et al.) 2013 Zhang et al.) 2014] Vedaldi et al. 2014 He & Peng 2017) due to the subtle differences between objects. FGVR methods can be divided into three types: (i) attention- based methods enhance the model's ability to recognize sub-
tle differences by focusing on the most critical areas of the image. (ii) hierarchical representation methods effectively handle the subtle differences between categories by constructing hierarchical feature representations that allow the model to refine image recognition progressively. (iii) metric learning methods improve the model's discriminative power in fine- mme-grained classification tasks by learning metric space where samples of the same class are closer and those of different classes are further apart. Moreover, TransHP (Wang al., 2023) integrated laaguae models, making FGVR less reliant on annotations and adaptable across various tasks. HI2R (Chen et al.) introduced a hypergraph-guidee guided approach that captures intra-class and inter-class relationships. enhancing the model's ability to discern subtle distinctions between fine-grained categories. CLEVER (Choud- hury et al. 2024] extracted non-expert descriptions from images and trained a fine line-grained textual similarity model to match image descriptions with Wikipedia document sentences accurately. Re- cent advancements like FineR (Liu et al., 2024c) employed large language models to translate visual attributes into text, enabling category identification without expert-defined labels.
6 CONCLUSION
In this paper, our objective is to analyze and boost the power of FGVR for MLLMs. We investi- gate the root cause of underperformance from three quintessential capabilities: object information extraction, category knowledge reserve, object-categon alignment, we position the problem as the misalignment between visual objects and category names. To address the challenge, we propose Attribute Augmented Alignment, designed to use attribute descriptions as an intermediate point to bind them. Based on the aligned representation space, we build Finedefics, a new MILM adept at identifying the subordinate-level category of the visual object. Our experiments, conducted on six popular FGVR datasets, demonstrate the remarkable performance of Finedefics. The validity of our methodology is substantiated through rigorous empirical studies.
Future Works. While Finedefics attains remarkable results across various FGVR datasets, it would encounter challenges in effectively learning new subordinate-level categories, and thus developing fine-tuning methods that can boost the continual FGVR capability for MLLMs is promising future research direction.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the grants from the National Natural Science Foundation of China (61925201, 62132001, 62432001, 62373043) and Beijing Natural Science Foundation (L.247006, 4252020).
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2XXiv:2303.08774 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 0e233008.29960 /2966, 2023.
Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.1511 2023.
Lu Chen, Qiangchang Wang, Zhaohui Li, and Yilong Yin. Hypergraph-guided intra-and inter- category relation modeling for fine-grained visual recognition. In ACM Multimedia 2024.
Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. The curious layperson: Fine-grained image recognition without expert labels. International Journal of Computer Vision, 132(2):537-535 2024.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint 4xXh:2240..03766, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint arXiv:2402.00776,, 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internin-xcomposere:: Mastering free- form text-image composition and comprehension in vision-language large model. arXiv preprint 17X1v:2401.16420, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint 2.033:2303.003 2023.
Mohamed Banani, Karan Desai, and Justin Johnson. Learning visual representations via language-guided sampling. In CVPR, pp. 19208 19208-19220, 2023.
Gregor Geigle, Radu Timofte, and Goran Glavaš. African or european swallow? benchmarking large vision-language models for fine ime-grrined object classification. arXiv preprint arXiv:2406. 14496, 2024.
Xiangteng He and Yuxin Peng. Fine-grained image classification via combining vision and lan- guage. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5994-6002, 2017.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Bootstrapping vision-language learning with decoupled language pre-training. Advances in Neural Information Processing Systems, 36, 2024.
MiaoyJJaang Haiyang Xu, Mengfan Dong. Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 77036-27044 2024.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. International Conference on Machine Learning, pp. 17283- 17300, PMLR, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCVW, pp. 554 554-561, 2013.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024a.
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint 1.Xiv:2405.02246,, 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. /CML, pp. 19730-19742. PMLR, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- 2024a.
fion, pp.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Democratiz- ing fine-grained visual recognition with large language models. arXiv preprint 47X.v.2401.13837, 2024c.
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lx, Zhiyuan Liu, and Ge Yu. Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval. arXiv preprint arXiv:2209.00 00179, 2022.
Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, and Lin Wang. Unibind: LIm- unified and balanced representation space to bind them all. In CVPR, pp. 26752-26762, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint (.XXv:2306605422, 2023.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint Xixx:1306.5155 2013.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pp. 729-729. IEEE, 2008
OpenAI. Chatgpt. https://oppnt wwtp.//..ccc/cccccc hht.bgtt..chhhttpp// 2023.
12

---

Published as a conference paper at ICLR 2025
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pp. 3498-3505. IEEE, 2012.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. pp. 8748 8763. PMLR, 2021.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305. 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11888- 1898, 2023.
H Touvron, T Lavril, G Izacard, X Martinet, MA Lachaux, T Lacroix, B Rozière, N Goyal, E Ham- bro, Azhar, al. Open and efficient foundation language models. Preprint at arXiv. https://oii. 2302, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008.
Andrea Vedaldi, Siddharth Mahendran, Stavros Tsogkas, Subhransu Maji, Ross Girshick, Juho Kan- nala, Esa Rahtu, lasonas Kokkinos, Matthew B Blaschko, David Weiss, et al. Understanding ob- jects in detail with fine-grained attributes. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3622-3629, 2014.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. NeurIPS, pp. (0506-10518, 2019.
Wenhao Wang, Yifan Sun, Wei Li, and Yi Yang. Transhp: Image classification with hierarchical prompting. Advances in Neural Information Processing Systems, 36:28187-282000 2023.
Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang. and Serge Belongie. Fine-grained image analysis with deep learning: A survey, TPAMI, 44(12); 8927-8948, 2021.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023a.
Le Zhang, Rabiul Awal, and Aishwarya Agrawal. Contrasting intra-modal and ranking cross-modal hard negatives enhance visio-linguistic compositional understanding. In CVPR, pp. 13774- 13784, 2024a.
Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for fine- tiningrined cat- egory detection. In Computer Vision-ECC ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proceedings, Part 13, pp. 834-849. Springer, 2014.
13

---

Published as a conference paper at ICLR 2025
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang. Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internim- xcomposer. A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 15/ 12, 2023b.
Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are risualy-grrundder language models bad at image classification? arXiv preprint 18811 2024b.
Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, et al. Chatspot: Bootstrapping multimodal Ilms via precise referring instruction tuning. arXiv preprint arXiv:2307.0947. 09444, 2023a.
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal Ilms. arXiv preprint arXiv:2307.0858// 2023b.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng. Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00544 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint orXiv:2304.10500 2023.
A APPENDIX
A.1 MORE ABLATION STUDIES
Attribute augmented alignment on other MLLMs. We build Finedefics upon Idefics2 (Laurençon) et al. 2 To confirm the general applicability of Finedefics, we conduct attribute augmented alignment (A on another typical MILM: LLaVA 1.5 (Liu al., 2024b). As shown Table 4 after employing our proposed method, LLaVA 1.5 gains an accuracy improvement by 13.97% on average, demonstrating the effectiveness and generalizability.
Effects of attribute types. We analyze the effects of specific attribute types in FGVR tasks. Specif- ically, we selectively remove typical attribute types from [color, shape, texture, size] to evaluate the contribution to performance improvement. As shown in Table [ all four types of attributes play crucial role in distinguishing subordinate-level categories, but the contribution varies with the dataset. For example, color and texture more critical for specific datasets, like flowers and birds.
Effects of hard negatives. We compare using hard negatives and simple negatives for contrastive learning. Specifically, we replace hard negatives with randomly sampled simple negatives, meaning that the negatives used for contrastive learning are less visually similar to positives and easier to distinguish from them. As illustrated in Table 6a] after applying contrastive learning with simple negatives, the improvement is limited. With the utilization of hard negatives, the modality gap decreases further, and the model harvests a significant accuracy improvement.
Effects of two-stage training stages. We analyze the effects of two-stage training by evaluating Finedefics by selectively removing specific training processes within each stage. As shown in Table 6b pretraining solely fails to follow the task instruction, while instruction tuning (I.T.) solely has limited performance gain. Instead, pretraining and instruction tuning are complementary further boost the accuracy, confirming the effectiveness of our two-stage training paradigm.
Effects of description quality. We first design an empirical study to evaluate the description quality, i.e how reliable the attribute descriptions we built. Similar to the probing experiments in Section 2.2 we test the representation discriminability of our constructed attribute descriptions on the training set of Oxford-IIIT Pet-37 Parkhi al. 2012) with splitting ratio of 1:1. The accuracy is 68.27% 27%. showing that the attribute descriptions can be well distinguished from each other though there exist subtle visual differences that difficult describe in words. Furthermore, to evaluate Finedefics's 's sensitivity to the description quality, we use three different quality levels of descriptions: (1) com- plete descriptions, (2) noisy descriptions (i.e., replacing some attribute descriptions with incorrect
14

---

Published as a conference paper at ICLR 2025
Table 4: Effects of employing attribute augmented alignment (A*) on LLaVA 1.5.
Model
Dog- 120
Bird-200
Aircraft-102
Flower-102
Pet-37
Car-196
Avg.
LLaVA 1.5
38,96
35.24 43.44
34.71 44.49
51.37
52.25
46.92
43.24
LLaVA 1.5
57.10
53.26
78.50
66.47
57.21
Table 5: Effects of attribute types.
Color
Shape
Texture
Size
Dog-120
Bird-200
Aircraft-102
Flower-102
Pet-37
196
Avg.
71.15
57.40
61.63
88.19
91.36
84.22
75.66
72.49
57.75
62.02
90.19
90.73
85.26
76.41
71.98
55.47
60.82
88.81
89.48
80.70
74.54
70.65
57.84
59.74
91.49
90.30
82.94
75.49
72.86
57.61
63.82
89.88
92.18
84.67
76.84
Table 6: More ablation studies. "Original" represents the zero-shot performance of Idefics2.
(a) Negative types.
(b) Training stages.
(c) Description quality.
Method
Avg.
Pretrain
I.T. Avg.
Method
Avg.
Original
65.95
0.00
None
72.72
Simple Neg.
74.26
76.13
Noisy
75.62
Hard Neg. (ours)
76.84
76.84
Complete (ours)
76.84
(d) Levels of detailed descriptions.
(c) Description construction.
Method
Avg.
Method
Avg.
None
72.72
Upper-hound
52.52
Short
76.11
Tag-based baseline
50.48
Long (ours)
76.84
Finedefics (ours)
51.12
ones by prompting ChatGPT OpenAl (2023), and (3) no descriptions (i.e., object-category align- ment w/o attribute). Results in Table 6c demonstrate Finedefics's 's robustness description noise.
analyze the impact of description length and detail on alignment effectiveness, we compare three different levels of detailed descriptions: (1) long descriptions, (2) short descriptions (i.c., generating short descriptions input images with BLIP-2
Effects levels detailed descriptions.
Li et al. (2023), and (3) descriptions (i.e., object-category alignment w/o attribute). Results in Table 6d demonstrate that rich and informative category information expression plays a crucial role in boosting the alignment between visual objects and category names.
Comparison with FineR. We compare the clustering accuracy (cAce) used in FineR Liu et al. (2024c) with the classification accuracy used in Finedefics. FineR be considered improv- ing the FGVR performance by using the attributes as a zero-shot manner. Note that classification accuracy can be considered as the perfect case of Hungarian matching used to obtain clustering ac- curacy. Results in Table 7 show the superiority of building an attribute-aware model compared to multi-agent system using attributes zero-shot manner.
Effects of attribute description construction. We evaluate the crucial rule of our proposed con- struction method acquiring per-sample attribute descriptions. To this end, we compare with an upper-bound and a bascline trained solely on Bird-200 dataset: (1) Upper-bound: we use Bird-200 (Wah al., 2011Y's 's per-sample ground-truth attributes, annotated by humans, for pretraining and instruction tuning. (2) Tag-based baseline: Given the category name, we prompt ChatGPT (Ope- nAl 2023) to acquire per-class attribute descriptions for each attribute tag used in our construction process, without using actual image samples. Then, these per-class sets of attribute-description pairs are assigned to each sample belonging to the class. Since for the same super category, the attribute values can differ significantly, this class-wise attribute description construction process introduces some noise. As shown in Table (6c) Finedefics surpasses the baged baseline, demonstrating the superiority of building detailed per-sample attribute descriptions. Moreover, observe that the
15

---

Published as a conference paper at ICLR 2025
Table Comparison with FineR. Clustering accuracy is used in FineR while classification accuracy is used in Finedefics.
Model
Dog-123
Bird-200
Flower- 102
Pet-37
Car-196
Avg.
FineR
48.10 10
51.10
63.80 89.88
72.90 92.18
49.20
57.00
Finedefics (ours)
72.86
57.61
84.67
79.44
Table 8: Objett-category alignment quality for Idefics2 and Finedefics.
Metric
Dog-120
Bird-200
Aireraft-102
Flower-
Pet-37
Car-196
Avg.
Idefics2
0.14
0.12
0.12
0.09
0.15
0.18
0.13
Finedefies (ours)
0.28
0,17
0.28
0.32
0.27
accuracy of Finedefics is close to the upper-bound, showing the potential of using LLMs and VQA models to obtain large-scale attribute data for alignment without human annotations.
Performance on common object recognition. We investigate how Finedefics performs on com- mon object recognition after pretraining and instruction tuning on FGVR datasets. As shown in Table we test on andeesat- (INaavversarial) (Hendrycks et al. 2021b), ImageNet- rendition (IN-rendition) (Hendrycks al.) 2021a), and ImageNet-sketch (IN (IN-sketc)) (Wang et al.) 2019). Results show that training solely on FGVR tasks has minor impact common object recognition accuracy. Therefore, training should performed on combined dataset comprising both coarse-graid graiied and fine lne-ggained recognition tasks. Developing fine-tuning methods that pre- vent such catastrophic forgetting to enhance FGVR capability without impacting common object recognition is promising future research direction.
A.2 MORE EVALUATION METRICS
Confusion matrix analysis. Besides the classification accuracy, we conduct confusion matrix anal- ysis on Oxford-IIIT Pet-37 (Parkhi al., 2012) identify where the model struggles across cat- egories. As shown in Figure Finedefies can recognize most of the categories correctly, while struggles in identifying a small portion of categories, such as Wheaten Terrior (52%), Staffordshire Bull Terrier (58%), Ragdoll (71%) American Bulldog (74%), and Birman (78%).
Alignment quality evaluation. We provide quantitative analysis of the object-category alignment quality for Idefies2 and Finedefics in Table The object-category alignment quality is calculated as the mean cosine similarity between embeddings of visual objects and their corresponding cat- egory names of each class. We observe that Finedefics significantly increases the object-category alignment quality, which further demonstrates the effectiveness in boosting alignment.
A.3 DISCRIMINABILITY COMPARISON BETWEEN DIFFERENT DATASETS
For each dataset, we provide a quantified analysis inter-class distance and intra-class variance for visual objects, well as inter-class distance for category names, respectively. As shown in Table 10 Dog-120 and Aircraft-100 have the smallest inter-class distance for visual objects, meaning that the subordinate-level categories of dogs and aircrafts are typically more visually similar and difficult to distinguish. Moreover, Pet-37 and Dog-120 have the largest intra-class distance for visual objects, meaning that attribute values differ significantly for the of pets. Most importantly, category names have lower discriminability than visual objects in the representation space despite the super category. For example, though Flower-102 has large inter- class distance and small intra-class variance for visual objects, the inter-class distance for category names sigmificantly different from other datasets.
subordinate-levv category
A.4 QUALITATIVE COMPARISON
We visualize and analyze the predictions of Finedefics and Idefics2 in Figure 6 Finedefics cessfully captures the nuance of the object features, setting them apart from visually similar sub- ordinate categories. This confirms that Finedefics effectively captures fine-grained visual details
16

---

Published as a conference paper at ICLR 2025
Table 9: Comparison on common object recognition datasets.
Model
IN-adversarial
IN-rendition
IN-sketch
Avg.
Idefics2
79.84
93.23
68.21
80.43
Finedefies (ours)
75.96
92.43
66.77
78.39
Table 10: Discriminability comparison between different datasets. O and C denote visual object and category name, respectively.
Metric
Dog- 120
Bird-200
Aircraft- 102
Flower- 102
Pet-37
Car-196
Inter-class Dist. for O (t)
0.36
0.37
0.31
0.41
0.43
0.49
Intra-class Var. for O (J)
0.27
0.20
0.12
0.07
0.28
0,19
Inter-class Dist. for C (f)
0.95
0.66
0.75
0,90
0.99
0,48
100
Abysa un
American Bul Tener-
Dasket ibund
Beagle--
Bengal -
Bombay
80
Brissh Shorthair
Chiuuauu
Epypt
English Cocker Spani
Germen Snorchaired
60
hovanese- 0
Tous Labea
Leo uenterger-
Maine
Minuture Wisccher-
Persian
Pormenn
Pug
Ragda1
Russian
Sairt Bernard
Scottion temer-
Stamese-
Sphynx-
Wheaten Tenner...
Areeven baloo
arrer can Bull lerer
bwoee hount
Burge
simat-
Bomber
tooe
Brish Braah
Epyoian Way
Engrin Salar-
Corman Cermun Sterdar
re-
Groat Grost Pylness-
wpnntr Did
Loobeegee
Mahe Cont-
Mnatun frouter
nowioundent-
Pamba
rume hug
kapdi
Rusan the
samoyr
scotin Neler
to tn-
Sammse
splyec-
Netorrsii sull B Keter-
staffrraii Whote peter
Cocker
Prodcted Label
Figure 5: Confusion matrix of Oxford-IIIT Pet-37.
from images, connects them with category names in the representation space, and then generates precise, fine-grained predictions. Furthermore, diets incorrect labels. We randomly pick three examples from the incorrect label and observe that the ground-truth and predicted label share most of the attributes, with only few exhibiting subtle differences. Concretely, as shown in the first row, Ragdolls are characterized by their fluffy, medium- to-long coats with distinct dark markings on the ears and around the eyes, whereas Siamese cats have short coats with darker color on their faces. Similarly, in the second row, pink primroses feature light pink color with fellow-green centers, while tree mallows exhibit vibrant pink color with dark purple to black centers.
two examples where Finedefics pre-
17

---

Published as a conference paper at ICLR 2025
Options: Common Raven
Options: Pomcranian
Groove billed
kecshond Samoyed
American
Shiny Cowbird
Eskimo dog
Idefies2: American Crow
Idefics2: Eskimo dog
Finedefies (ours): Groove billed
Finedefies (ours): Samoyed
Bird-200
(b) Dog- 120
Options: Audi TT Hatchback 2011
Options: 767-200
Coupe 2012
767-300
Audi TTS Coupe 2012
Cootlaewtal
777-300
Audi S5 Coupe 2012
767-400
Idefies2: Audi AS Coupe 2012
Idetics2: 767-300
Finedefies (ours):
Coupe 2012
Finedefies (ours): 767-400
(c) Car-196
(d) Aiveraf--102
Options: tiger lily fire lily
Options: Ragdoll
sword lily
Maine Coon
peruvian lily
Sinmese
Idefies2: fire lily
Idefies2: Birman
Finedefics (ours): peruvian lily
Finedefics (ours): Ragdoll
(c) Flower-100 102
(f) Pct-37
Figure 6: Qualitative comparison on FGVR datasets, where green indicates correct predictions and red indicates incorrect ones.
Ragdoll Finedefies (ours):
Examples Siamese
(a) Pet-37
Ground-trttl: pink primrose (ours): tree
Examples of free mallow
(b) Flowcr-102
Figure 7: Error analysis examples. The left column shows the image for prediction, while the right column shows examples of the incorrect label.
18


Reviews:
[{"review_id": "rtkEOxmDmQ", "reviewer": "Reviewer_qkG7", "summary": "This research paper investigates the challenges faced by multi-modal large language models (MLLMs) in fine-grained visual recognition (FGVR), a task involving identifying specific sub-categories within broader categories, such as distinguishing different types of birds. The authors pinpoint the root cause of this underperformance as a misalignment between visual object representations and corresponding category names within the model's representation space. To address this, they propose Finedefics, an MLLM architecture that uses informative attribute descriptions of objects as an intermediate point to bridge this gap. This approach effectively aligns visual objects and category names during training, leading to significant performance gains in FGVR tasks compared to existing models.", "questions": "- In the experiment in Section 2.1, what prompts did you use? Also, to clarify, was linear probing performed after the connector or the LLM?\n- Could you explain Figure 2 (c+f) in more detail?\n\n- [1] H. Laurençon, L. Tronchon, M. Cord, and V. Sanh. *What matters when building vision-language models?*\n- [2] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou. *When and Why Vision-Language Models Behave like Bags-of-Words, and What to Do About It?*", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Section 2 provides a thorough and engaging explanation of the problem.\n- Problem formulation is straightforward and easy to understand.\n- The paper demonstrates significant improvements across all tested datasets.", "weaknesses": "- Despite the strong results, the proposed methods lack substantial novelty compared to prior works ([1], [2]) that also leverage foundation models for data augmentation and model fine-tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UKYNIR85Oy", "reviewer": "Reviewer_1pZP", "summary": "The paper proposes a model called Finedefics to improve FGVR in MLLMs, which typically struggle with distinguishing subordinate-level visual categories. The authors identify misalignment between visual objects and category names as a key challenge, proposing a two-stage training approach that uses attribute-augmented contrastive learning to align objects and category descriptions more effectively. Evaluations across several FGVR datasets show that Finedefics significantly improves recognition accuracy over existing MLLMs.", "questions": "In addition to the issues mentioned in Weaknesses, there is another point  about generalization  that needs to be clarified, as follows:\nEach MLLM has different levels of perceptual sensitivity to fine-grained details. For example, some models excel in low-level visual feature recognition, while others are better suited for high-level semantic understanding. The proposed approach relies heavily on the specific architecture of Idefics2, particularly the model’s vision encoder, modality connector, and alignment layers. Other MLLMs might not have identical modules or configurations, which could make direct application of the proposed training paradigm challenging.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper does an excellent job identifying the specific challenges of FGVR within MLLMs. It is worth mentioning that this paper provides valuable insights into the object-category misalignment issues in existing MLLMs. \n2. The proposed Finedefics framework introduces a creative method for addressing FGVR challenges by employing attribute-augmented alignment.\n3. The paper’s methodology is thorough, with a well-delineated two-stage training process.", "weaknesses": "In general, the motivation for this submission is easy to understand and insight is interesting, but the experimental analysis is limited. In addition, there are still several weaknesses, as follows:  \n1. The proposed Finedefics method was constructed and validated solely based on the Idefics2 model, with no tests conducted on other prominent MLLMs. Different MLLMs, like BLIP-2, LLaVA, or Qwen-VL-Chat, vary significantly in their abilities to process visual features, generate textual descriptions, and perform contrastive learning. Without experimental validation across these models, it’s challenging to confirm the general applicability of Finedefics. \n2. The paper contrasts \"object-category\" and \"object-attribute-category\" pairs, but it does not analyze the effects of specific attribute types (e.g., color, shape, texture). An ablation study could selectively remove particular attribute types to evaluate their contribution to performance improvement. This would help clarify which attributes are most important in fine-grained recognition tasks, particularly if certain attributes are more critical for specific datasets, like flowers or birds.  \n3. While standard contrastive learning ablation is provided, the study lacks a comparison between hard negatives and simple negatives. Additional experiments could assess performance when hard negatives are removed, highlighting whether they significantly enhance alignment and fine-grained recognition capability.  \n4. The current study compares one-stage and two-stage training, but more granular analysis could be performed. For example, selectively adding or removing specific training processes within each stage could reveal the individual effects of the alignment and instruction-tuning stages. This would help clarify the extent to which each stage contributes and whether they are complementary under certain conditions.  \n5. The methodology relies on attribute extraction from pre-trained models like GPT-4 and LLaVA. However, there’s no empirical analysis of how reliable or accurate these attributes are for all instances in the datasets, especially for categories with subtle visual differences. Since the attribute descriptions are generated by pre-trained LLM and VQA models, it would be valuable to conduct ablation studies using different quality levels of descriptions (e.g., complete descriptions, noisy descriptions, or no descriptions) to test the model’s robustness and dependency. \n6. The paper mainly uses accuracy as an evaluation metric, which may not capture the nuanced performance differences in FGVR tasks. Metrics like confusion matrix analysis could provide more insight, especially for identifying where the model struggles across categories. Furthermore, the visualization section could include quantitative measures to assess the object-category alignment quality, rather than relying solely on visual interpretation.  \n7. The paper suggests that category names alone lack discriminability within MLLMs. A test could be designed where category names are replaced with varying levels of detailed descriptions (e.g., short versus long descriptions) to analyze the impact of description length and detail on alignment effectiveness. This would verify whether the form of category information expression significantly affects model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pbufDNX0Mc", "reviewer": "Reviewer_VD1i", "summary": "This paper studies the task of Fine-grained Visual Recognition (FGVR). The authors point out the underperformance of modern MLLMs on FGVR and thoroughly explore the cause. The findings suggest that the misalignment between visual objects and category names is the main reason that hinders MLLMs' performance on FGVR. To this end, a novel instruction-tuning framework named Finedeficsfor is proposed to enhance the performance of MLLMs on FGVR by leveraging attribute descriptions. Extensive comparisons, experiments, and ablation studies are conducted across six FGVR benchmarks to validate the effectiveness of the proposed method (model) and its components.", "questions": "(1) In the current paper, pet (dogs and cats) images are largely used as experimental subjects in the analysis of MLLMs in FGVR in Section 2. Pets, such as dogs and cats, have a unique characteristic: for the same pet category (or breed), the attribute values (e.g., “Fur Color: Brown”, “Fur Pattern: Dots”) can differ significantly (the so-called large intra-class variance in FGVR). This characteristic also applies to cars (StanfordCar196). However, I wonder: what would be the observation for birds (CUB200) or flowers (Oxford-flower)? For the same bird or flower species, the attribute values are rather fixed—i.e., the same bird species will always have the same body color pattern. In this case, would the observations and conclusions change? Would the category names still have lower discriminability in the representation space?\n  \n(2) Can the authors provide: 1) qualitative results; 2) failure case analysis? It is necessary to show how the model behaves towards FGVR QA since this is the way it will be used in deployment.\n\n(3) One open question: Why is instruction-tuning MLLMs for FGVR useful and valuable? If the goal is to achieve high FGVR performance, supervising a strong pre-trained model on FGVR datasets can achieve higher performance. Since both paradigms require training, why do we use MLLMs for FGVR?\n\n**Further Suggestions:**\n- For the mining of hard negatives for each category, a suggestion is to explore using negatives that are highly similar to the target class but dissimilar in diverse attribute aspects. The current method relies on overall CLIP similarity for decision-making. Note that this is just a suggestion—I understand that it takes time to implement and train the model. The authors are not required to show this during the rebuttal period, but it would be interesting to see.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Before listing the strengths of this paper point by point, I would like to first acknowledge and highlight the research approach employed by this work.**\n```\nThe way this paper conducts its research on improving MLLMs’ capabilities for FGVR demonstrates the authors' rigorous and meticulous scientific approach: identifying the problem (MLLM underperformance in FGVR) → understanding the cause (misalignment between objects and categories) → proposing a solution (leveraging informative attribute descriptions to effectively align visual objects and category names in the representation space of LLMs) → achieving improved performance (surpassing Idefics2 and Qwen-VL-Chat by an average of +10.89% and +9.43%, respectively).\n\nI greatly appreciate this work's research methodology. The carefully designed analysis of MLLMs in FGVR presented in Section 2 will provide valuable insights and guidance for the community and future researchers. These insights may be even more valuable than the proposed method itself and its performance improvements, as they will significantly aid further research in this area. Thank you to the authors for the contribution!\n```\n\n**Strengths Point-by-Point:**\n- The preliminary exploration of revisiting MLLMs’ capabilities for FGVR shown in Figure 1 is really interesting and intuitive. **I really appreciate this experimental design**. It highlights the core issue—Object-Category Alignment—that hinders MLLMs' FGVR performance.\n- The proposed instruction tuning method, consisting of Attribute-Category Contrastive, Object-Attribute Contrastive, and Category-Category Contrastive, is novel and well-designed. The design is both technically innovative and conceptually intuitive.\n- The proposed method boosts MLLM performance by a significant margin and consistently outperforms other MLLMs at similar parameter scales.\n- Comprehensive ablation studies are conducted to assess the effectiveness of the proposed components.", "weaknesses": "(1) Since the proposed method relies on the attributes obtained via FineR [1] cross-modal chain-of-thoughts prompting,  FineR's FGVR performance should be compared as a baseline in the main experiments. FineR can be consider as improving VLM's FGVR performance by using the attributes as a zero-shot manner. The clustering accuracy used in FineR can be compared with classification accuracy (classification accuracy can be consiered as the perfect case of Hungarian matching used to obtain clustering accuracy). Although an apple-to-apple comparison is hard since FineR consists of multiple agents, this comparison can serve as a good reference to the readers.\n\n(2) In the study of Finedefics in Section 4.3, the influence of construction of Attribute Descriptions should also be examined since it is a crucial component for acquiring per-sample, per-class attribute descriptions. To this end, one experimental design I can suggest is comparing the current construction method with an upper-bound and a baseline on a dataset, such as CUB-200:  \n  i) **Upper-bound:** The authors can use CUB-200’s ground-truth attributes, annotated by humans, for instruction tuning.  \n  ii) **Tag-based baseline:** Given the class names (e.g., Blue-throated Blue Warbler), the authors can directly prompt an LLM to acquire both useful attributes (e.g., body color pattern) as tags and possible attribute descriptions (e.g., dark blue) for each class’s attribute tags, without using actual image samples. Then, these per-class sets of attribute-description pairs can be assigned to each sample belonging to the class. Since this acquisition process is not conditioned on the training images, it might be noisier. It would be interesting to see how sensitive the model performance is to this upper-bound and baseline.\n  \n(3) The proposed method requires instruction tuning of MLLMs. Although the main goal is to improve MLLM performance for FGVR, it is still important to investigate whether this FGVR-centric tuning hampers the performance of MLLMs on general QA or common object recognition.\n\n**Minor:**\n- At P4#L193: Similarity —> Similarly\n\n**P.S.:** I understand that the rebuttal period provides limited time to address all the concerns and questions raised. For me, the most critical issues are Q1 (no training required) and Q2 outlined above. **If time constraints prevent the authors from addressing all my concerns, please prioritize responding to Q(1) and Q(2), and allocate time to address the other reviewers' major concerns.** It is fine for me if the authors could not address all my concerns and questions (which might take time to train) within the limited time during the rebuttal period. No worries. But it is recommended to include them in the final revision.\n\n[1] Liu, M., Roy, S., Li, W., Zhong, Z., Sebe, N., & Ricci, E. (2024). Democratizing fine-grained visual recognition with large language models.  In ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)