Why are Visually-Grounded Language Models Bad at Image Classification?
Authors: Yuhui Zhang
Alyssa Unell
Xiaohan Wang
Dhruba Ghosh
Yuchang Su
Ludwig Schmidt
Serena Yeung-Levy
Venue: NeurIPS 2024 poster (2024)
Keywords: vision-language models
image classification
Primary area: interpretability_and_explainability
TL;DR: We explain why visually-grounded language models are bad at classification and propose a simple data intervention method to fix that.

Abstract: Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.

Why are Visually-Grounded Language Models Bad at Image Classification?
Yuhui Zhang
Alyssa Unell Xiaohan Wang'
Dhruba Ghosh? Yuchang Su
Ludwig Schmidt 1.2
Serena Yeung-Levy't
Stanford University
Universityy of Washington TTinnhhua University
T[yuhuiz, wawpiiis,udwigsc.. sydwiggc,,eyyuungeeee
Abstract
Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using Visually- grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data- related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMS can match the accuracy of state-of-the-art classification models. Based these findings, enhance VLM by integrating cillffccationffoussee datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers its general capabilities, resulting in an improvement of 11.8% on the newly collected Image WikiQA dataset
Introduction
The ability to recognize objects within images is fundamental capability of machine vision. Over the past 15 years, the field has experienced significant breakthroughs due deep learning and large-scale datasets [1] (88. For instance, the renowned ImageNet dataset, designed to classify images into 1,000 categories, the error rate has dramatically decreased from 47.1 % 2009 to 9.1% % in 2024, representing a f-lold reduction 30 12). Consequently, these classification models have superseded most human labelers.
Nowadays, the community has focused on more sophisticated and nuanced capabilities in the quest for visual intelligence. signals from vision encoders with large language models, have recently emerged as promising VLMs like GPT-4V (36), Gemini- [44], or Claude-3 [3] have demonstrated advanced visual understanding abilities, such as answering math questions from table images or generating HTML code from design sketches.
language models (VLMs), which integrate visual
paradigm 2 36 32
In this work, we revisit the fundamental task of image classification using VLMs. Surprisingly, we find that various public and proprietary VLMs struggle with image classification in both open-world settings, where the class list is unknown, and closed-world settings, where class names are provided in the context ($2). Despite having many more parameters, there is significant gap between the
Project page: https://yuhui: zh15 http:://yuuaa--ttttgtttbbbbbba.. wit.uua...o/AAAlcsssssseeereee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

VLMs are bad image classifiers
Why are VLMs bad?
How to improve VLMs?
guinea
What is the image?
What the notive region this object?
Answer: guinea
000
VLM
CLIP
LM
A, South America
B. Africa Australia
The image shows amall rabbit
80
What in the image? (D) Choose one from...
LLoVA trained on original dota:
image shows
Proprietory VLMs
The native region could vary depending the species.
Prompt Variation?
VLMs
Inference?
Space?
40
Inference Strategy?
LLoVA trained originolellassification:
The animal guineo pig. which native to South America.
20
Training?
Information Lost?
Training Objective?
therafore, the answer
BLIP LLaVA Gemini Clauue
Data?
Enough Data?
Classification foundational!
language models (VLMs) underperform CLIP in classification by large margin, though they often use CLIP as a vision encoder. (Middle) We investigate several hypotheses about why VLMs are bad classifiers and find that the main reason is data. Critical information for image classification is encoded in the VLM's latent space but can only be decoded with enough data during VLM training. (Right) Based on our analysis, we improve VLM by integrating classification data into its training, and find that the improved classification capabilities serve as foundations for more advanced capabilities such as visual question answering.
Figure Overview. (Left) Different
performance of VLMs and their commonly used vision encoder CLIP [37 Our evaluation protocol involves feeding each image and a list of class names (in the closed-world setting) the VLM as context and asking what is in the image; success is defined by whether the generated output contains the ground- -trut class name.
To understand why VLMs underperform in classification settings, we investigate several hypotheses regarding VLMs inference (such prompt variations, label set size, inference strategy; $3.1), training (such as information lost, training objective; $3.2), and data (such as data-performance correlation; {3.3). Our extensive analyses suggest that the primary reason for the observed gap is data. We find that the information necessary for classification is encoded in the VLM's latent space but can only be decoded with proper training data. Specifically, there is strong correlation between class presence during VLM training and performance in those classes. Furthermore, training VLMs on classification datasets achieves the same performance level state-of-the-arr classification models.
Motivated by our analysis, we propose a simple method to enhance VLMs' general capabilities by integrating traditional blssification-focusee datasets into VLM training ($4). We believe that classification is the foundation for more complex, advanced visual capabilities; for example, recog- nizing object prerequisite for answering complex questions about it. To verify this, created Image ImageWikiQ WikiQA, which contains complex real-world questions about ImageNet objects. On ImageWik, iQA, find that VLMs fine-tuned on the ImageNet classification dataset achieve substantially higher accuracy in recognizing these objects and provide more accurate answers to these non-classification questions, outperforming pre-trained VLMs by 11.8% This suggests that classical classification data can be beneficially reused in the VLM training process to enhance VLM performance.
In summary, contributions are threefold:
Evaluating VLM classification weaknesses: Our evaluations using ten VLMs across four benchmarks show that VLMs significantly lag behind CLIP in classification, uncovering a gap unaddressed by previous research.
Analyzing causes of poor classification performance: Testing various hypotheses, we find that the lack of alignment data, rather than information lost or training objective, is the primary reason for VLMs' underperformance in classification.
Enhancing VLMs with classification data: By adding classification data, we improve VLMs' accuracy object recognition and overall performance, supporting that classification capability foundational for complex object-related reasoning.

---

Model
Open-World World Setting
Closed-Woo Setting
Public VLM
BLIP2-2.77 27
25.3
27.0
0.0
46.9
N/A
14.2
2.7
22.3
IBLIP-7B [0
14.6
1.9
0.0
36.5
N/A
26.8
N/A
58.4
IBLIP-13B [10
14.7
2.4
0.0
36.4
N/A
20.0
N/A
59.5
LLaVA1.5- R2
22.8
5.9
0.0
47.1
N/A
10.2
0.0
62.1
LLaVANeXT-V77
29.4
12.8
0.0
52.5
N/A
8.5
0.0
66.6
LLaVA1.5- 11141.5-13B [32]
24.3
5.3
0.0
49.9
N/A
7.2
0.1
70.9
LLaVANeXT-NNe 32
32.3
17.7
0.0
54.2
N/A
16.1
3.6
77.3
Proprietary VLM
Claude3 3
53.6
51.2
0.3
68.6
51.1
58.3
45.1
90.9
GeminiPro
39.2
10.5
0.1
60.1
56.0
62.0
66.6
91.6
GPT4 [36
48.5
51.0
0.1
61.0
60.6
79.9
58.2
94.2
CLIP
CLIP-L 37
N/A
N/A
N/A
N/A
74.8
76.0
77.5
95.8
EVA-G
N/A
N/A
79.2
90.2
97.9
Table 1: Evaluations of VLMs and CLIPs on standard image classification benchmarks.
VLMS exhibit poor performance image classification, significantly lagging behind CLIP models.
ImageNet
[1].
=FFlowers10
35
-StanfordCars
21. CCaltech101 13..
VLMs are Bad at Image Classification
We begin by evaluating state-of-the-art
language models (VLMs) using standard image classification benchmarks. Our findings reveal that these VLMs significantly underperform compared to state-of-the-arr classification models, such as CLIP.
2.1 Models
VLMs. We selected widely-used state-of-the-art VLMs, covering different archi- tectures, training methods, and data. These VLMs include three proprietary ones, GPT-4-Turbo (shortened as GPT4, same below) (36), Gemini-Pro-Visie and Claude-3-0pus (Claude3) (3) and seven public ones, LLaVA1 5-Vi ALaVA1.5-Vi c-Va7/ /13BB (LLaVA1. (LLAVA))--7/1BB BLIP2-OPT2. 7B (BLIP2-2 7B) (27] and wtw.ruccttLI--iiccaaar.. Details of these models are provided Appendix $A.1
(GeminiPro) [4].
(LLAVAMeXT-M7B/VTT)
31]
(IBLIP-7/13)
10
CLIPs.
For comparison, used two state-of-the tte-th--n image classifiers, CLIP- CLIP-VIT-L/14-333
(shortened as CLIP-L, same below) (37] and WVA-ViT-G/14 (EVA-G) 43 Notably, CLIP-L and
EVA-G utilized by the LLaVA series 332 and the BLIP series vision encoders (27] respectively.
Therefore, the VLMs should theoretically have the same classification capacity as these vision models. Details are listed in Appendix $A.1
Data
We evaluated the aforementioned models four widely-used image classification benchmarks: ImageNet (shortened Caltech101 [3], which contain 50,000, 6, 149, 8,041, and 4,331 test images from 1,000, 102, 196, and 101 classes, respectively. ImageNet and Caltech cover Flowers and Cars cover more fine-grained objects. Further details are provided in Appendix A.2
same below) 11. Flowers
351 StanfordCars
and
coarse-grained objects, while
2.3 Evaluation Protocol
VLMss. We performed image classification in two settings: an open-world setting where the label set is not provided and closed-world setting where classes concatenated in the prompt. We

---

feed the image and the prompt to the VLM and let the VLM complete the rest of the tokens. One closed-world example is: "<image> What type of object is in this photo? Choose one from <class name A>, <class name B>, ... We define success on single example as whether the ground-truth label included the VIM generation. We report the success rate of all test examples.
CLIPs. CLIP can only be used in a closed-world setting where the label set is known. Following Radford et al. [37]. we used the prompt "a photo of a <class>" generate the text feature. For each image, we selected the class with the highest cosine similarity to the image feature. We did not include prompt ensembling to fairly compare with VLM. We report the accuracy of all examples.
2.4 Results
Table 1 reports the performance of different VLMs and CLIP models on these classification datasets.
We find that VLMs exhibit poor performance in image classification, significantly lagging behind CLIP models. For instance, on the ImageNet dataset, the best proprietary VLM, GPT4, only achieves an accuracy of 60.6% in the glosed-worlde setting, whereas the best CLIP, EVA-G, attains an accuracy of 79.2%. In the open-world setting, the best public VLM, LLaVANeXT-MPI achieves just 32.3% accuracy. The performance disparity is even more pronounced in fine-gran ain-ggrined classification datasets like Flowers 102 and StanfordCars. Notably, all LLa models use CLIP-L as the vision encoder, and although the total parameter count of these models is at least 20 times greater than that of the vision encoder, they significantly underperform compared to it.
Moreover, we find that worled-world setting often outperforms open-world setting. This is expected as the provided label set narrows the prediction space. However, since closed-wort settings require including all class names in the context, they can result in an extremely long context that leads to high costs or even exceeds the VLM's 's context limit. For example, most public VLMs only support 4K context length, which cannot feed 1K ImageNet classes. We also find that larger and better LMs slightly improve VLM performance. For instance, LLaVA1. 5-B outperforms LLaVA1. 7B and LLaVANeXT-M7B outperforms LLaVANeXT-V75
3 Why are VLMs Bad Image Classifier??
Given that visually-grounde language models (VLMs) underperform CLIPs at classification by large margin, reported in $2 we seek understand the reasons behind that. We investigate several hypotheses concerning major differences between VLMs and CLIPs, which can be generally categorized into inference ($ (...) training ($3.2. and data ($ (3.3)
We start with inference-related questions. For example, does prompt variation, such as chain of thought, affect final performance? Does reducing the label set size in context narrow the gap between VLMs and CLIPs? Does performing probabilistic inference to force the generation into the label set help? We find none of these factors can fully close the gap between VLMs and CLIPs.
2. Therefore, we switch to training-relate the vision encoder still preserved the VLM's latent space? Is the text generation objective as effective cross-entropy loss for learning classification? Surprisingly, the results show that the information is preserved, and the text generation objective is adequate for learning classification.
questions. For example, is the visual information from
Finally, we investigate data-related questions. For example, does the VLM training data include enough classification data and cover enough classes? We find a strong correlation between class exposure in training and model performance. Moreover, VLMs can achieve the same level of performance as CLIPs when trained with enough data. These results suggest that data is the primary cause and effective solution for the poor classification performance of VLMs.
3.1 Inference
In this section, we investigate three questions related to VLM's inference, including prompt variation, label set size, and inference strategy.
Prompt variation. well known that language models (L.Ms) are sensitive to prompts 16 To understand the effect of prompts on classification performance, tested three semantically

---

Method
LLaVA1 -5-7
BLIP2-2.7B
G
Prompt Variation
Base Prompt
22.8
5.9
0.0
47.1
25.3
27.0
0.0
46.9
w/ Prompt Alternative 1
19.7
3.5
0.0
45.4
27.6
38.8
0.2
47.8
w/ Prompt Alternative 2
21.6
6,6
0.0
48.1
24.3
30.6
0.1
47.9
w/ Label (Fixed Order)
N/A
6.1
0.1
70.5
N/A
28.0
2.1
53.8
w/ Label (Random Order)
N/A
10.2
0.0
62.1
N/A
14.2
2.7
22.3
w/ Label (Random) + CoT
N/A
18.1
0,4
64.5
N/A
8.5
0.0
16.1
Inference Strategy
Direct Generation
22.8
5.9
0.0
47.1
25.3
27.0
0.0
46.9
Prob Inferenee (Sum Tokens)
34.8
14.5
26.7
77.8
21.0
34.8
48.8
36.8
Prob Inference (Avg Tokens)
35.3
16.5
18.2
65.6
5.1
19.9
1.2
12.3
Prob Inference (Avg) w/ CFG
47.6
26.8
48.8
85.6
38.7
54.1
69.2
70.3
Table 2: Analysis of VLMs from the inference perspective. (Top) We explore prompt variation such wording, label order, thain-of-thougee and find it has limited impact on the performance. (Bottom) We leverage the probabilistic inference strategy, which improves the performance but still fails to close the gap between VLMs and CLIPs. Results are from the official validation set.
ImageNet
Flowers
Cars
Caltech
100
100
100
100
80
80
60
100
20
100
LLaVA1.5-78
CUPL
Figure 2: Analysis of the label set size. For each image, randomly sample 100, 20, candidate classes from all the classes. The performance gap between VLMs and CLIPs becomes smaller when the number of classes is reduced. X-axis: number of classes; Y-axis: accuracy (%).
similar but differently worded prompts, compared feeding the label in dataset order or random order within the context, and leveraged the zero-shot chain-of-thought (CoT) prompting technique by adding "let's s think step by step" at the end of the prompt [47 20
We find that prompt variation has a limited impact the performance. From Table 2 can see that changing the wording of prompts results in performance variation within 3% for both LLaVA1. 5- and BLIP2-2.0 7B the ImageNet dataset. Different label orderings impact LLaVA1 5-7B less than BLIP2-2 2. 7B. ofain-of-thought prompting consistently improves perfor- mance for instruction-tunet -unteedn-tnnedddeeee model LLaVA1. 5- but not for BLIP2-2.7B.
Label set size. The label set size can be very large in practice (e.g.. 1000 for ImageNet, 196 for StanfordCars), which results in an extremely long context when we concatenate all the class names. Since LMs often struggle with long contexts [3), explore reducing the number of classes. For each image, we randomly select K = 2, 5, 20, 100 classes, always including the ground-trut label, and re-evaluate the VLM and CLIP performance.
We find that the performance gap between VLMs and CLIPs narrows with reduced label size, but the gap always exists. As shown in Figure 2 when evaluating LLaVA1.55 5577B ..-.BB and CLIP-L on ImageNet, the gap decreases from 46% with 100 classes 6% with 2 classes. However, the relative gap becomes larger, evidenced by 23.9x error rate with 2 classes compared to 7.3x error rate with 100 classes. The performance gap between VLMs and CLIPs always exists in all the settings.
Inference algorithm. The default inference algorithm for classification with VLMs directly generating the class name given prompt. As the generation is open-ended, even when provided with list of candidate choices, the generation may not match one of the pre-defined classes. mitigate this problem, employ probabilistic inference techniques for VLMs. Specifically, for each class name,
5

---

Model
Trainable
Model
Feature
Fine-tuning
Probing
CLIP-L
Linear
85.2
98.6
91.5
97.6
LLaVA1.5 5-
Last Tok
76.9
94.5
81.0
96.7
LLaVA1 MBBB
Proj
85.7
97.6
90.4
97.5
LLaVA1. .577
Avg Tok
77.1
96.2
82.8
97.3
LLaVA1. 5-7B
Proj+LM
NaN
97.9
90.7
97.8
BLIP2-2.7B
Last Tok 80.3
98.8
91.0
98.0
EVA-G
Linear
86.5
99.2
94.3
98.5
BLIP2-2.7B
Avg Tok 81.4 98.9
92.6
98.0
BLIP2-2.78 7B
Proj
88.0
99.0
93.9
98.8
Table 3: Analysis of VLMs from the training perspective. (Left) We conduct feature probing experiments on the VLM's last layer and find that the information required for classification is mostly preserved in the VLM's latent space. (Right) We fine-tune VLMS on the classification datasets using the text generation objective and find that the text generation training objective is as effective as the traditional cross-entropy for learning classification, which eliminates the VI.M-CIIP performance gap, with VLMs now being the statc-of-the-an classifier. Results are from the official validation set.
we compute prob(class name image, prompt) and select the class name with the highest probability as the prediction. Since class names consist of multiple tokens (e.g.. "guinea pig" consists of two tokens), we either average the probabilities of all tokens or sum up all the tokens 77]. We also explore classifier-free guidance (CFG) techniques by ranking * prob (class name image, prompt) (1 - t) * prob(class name prompt) with varying guidance coefficients (0).
We find that the probabilistic inference method improves the performance, but the gap persists. As shown in Table 2 LLaVA1. 5- achieves 35.3% accuracy on ImageNet using probabilistic inference compared to 22.7% using the direct generation method. Adding classifier-free guidance further improves the performance, where LLaVA1.5-78 performance boosts to 47.6% on ImageNet. However, it still leaves around a 30% performance gap between VLMs and CLIPs, as its vision encoder CLIP-L achieves 74.8% on ImageNet. Moreover, the probabilistic inference approach is computationally expensive in practice because we need to compute the probability of each class,
3.2 Training
Since inference-basee modifications fail to close the performance gap between VLMs and CLIPs, here we investigate two questions regarding to the training of VLMs. We study whether the visual information is lost in the VLM and whether the text generation objective is suitable for learning the classification task.
Visual information lost in the VLM. VLMs process images using an image encoder, such as CLIP, which has strong classification capabilities. We hypothesize that, during the propagation of image features output from the vision encoder in language model layers, the necessary information for classification is lost. To test this hypothesis, we conduct feature probing experiments. Specifically, the features from the VLM's last layer have shape corresponding to the length of the inputs (image tokens plus text tokens using the open-world prompt). We take the average of these token features or the last token feature and train a simple linear classifier on top of these frozen feature representations. The linear classifier is trained the training set and evaluated the validation set. Training details are provided in Appendix B.4 A higher accuracy indicates that less information is lost.
Surprisingly, we find that the information necessary for classification is largely preserved in the VLM's latent space; however, it cannot be effectively decoded. In Table 3 we show that the probing accuracy of LLaVA1... 5- on ImageNet is 77.1% %2 which is close to the 85.2% probing accuracy of CLIP-L, the vision encoder used by LLaVA1. 5-BB. The same conclusion holds for BLIP2-2. 7B and its vision encoder EVA ViT-G/14, demonstrating that most information is preserved during the VLM computational process. However, this information cannot effectively decoded, as shoos that the best accuracy LLaVA1 1.AVA1.5-7B 5B can achieve on ImageNet is 47.6%.
Training objective. Since information is encoded in VLMs, we wonder whether we can train them to decode the information. VLMs can only be trained to perform classification by auto-regressively generating text-form labels. We hypothesize that this generative objective may be more difficult and less effective in learning classification compared to the traditional cross-entropy loss. To understand the efffcctiveness of this training objective, we explore fine-tuning VLMs on classification datasets. We

---

100
Imagelec Accuracy
40
CUP
Finetuned LLaVA
101 (438
[20. [30, (85 class) dass)
[40,
130, 160,
170,
[80. 190. [100.
1KK 100K)
Class Frequency
Figure 3: Analysis of VLMs from the data perspective. We study the relation between the ImageNet class frequency in the VLM training data and the VLM's classification performance on those classes. A strong correlation is observed, indicating that data determines VLM classification performance.
convert classification datasets into the text generation format using the template (e.g., "<image> What type of object in this photo? <class name>"). We fine-tune two model architectures (LLaVA1 .-BB and BLIP2-2.77 7B) across different datasets and compare their accuracy to that of fine-tuned CLIP models. Additionally, we investigate fine-tuning different parts of the models, such as only fine-tuning the projector between vision encoder and language models or fine-tuning the projector along with the language models using LoRA [7]. Training details are provided in Appendix [35]]
We find that the text generation training objective is as effective as the traditional cross- entropy for learning classification tasks, which eliminates the performance gap between VLMs and CLIPs, with VLMs now being the state-of-the-art classifier. From Table 3, we find that LLaVA1. 5-7B achieves 85.7% accuracy ImageNet, same 85.2% accuracy when fine-tuning its vision encoder CLIP-L. The same findings apply to all the VLMS on all the datasets.
Moreover, find that fine-tuning only the projector is sufficient and has better numerical stability. From Table 3 we can see the same level of accuracy achieved by fine-tuning the projector and fine-tuning projector and LLM using LoRA on the three datasets. We also find that fine- tuning LLM with LoRA often leads to numerical instabilities, such as spikes in loss. While the loss sometimes returns normal, other times it does not. For example, despite trying various hyperparameters for ImageNet, instability persisted (see Appendix şB.5). In contrast, fine-tuning only the projector always results in steadily decreasing loss. Notably, the projector is often much smaller (e.g., a 2-layer MIP in LLaVA) compared to LLMs, suggesting the potential for parameter-efficient prefix tuning for VLMS 28
3.3 Data
As observed, the visual information is preserved in the VLM's latent space, and the VLM's training objective is sufficient for learning classification tasks. We hypothesize that the poor classification per- formance of VL.Ms is due to their training data. For example, there might be insufficient classification data or a lack of diverse classes. To investigate this, we analyze the LLaVA1. 5-7B training data (32]. the only fully publicly available VLM training dataset. We examined the relationship between the frequency of class occurrence and the VLM's classification performance on those classe
Our findings indicate that data determines VLM classification performance. As shown in Figure 3 there is a strong correlation between the presence of class labels the VLM training data and the VLM's classification accuracy for those classes. The LLaVA1 55 model achieves 82.7% zero-shot accuracy on ImageNet classes with more than 10,000 occurrences in its training data, but only 3.0% accuracy on classes with fewer than 10 occurrences. The Spearman correlation between class frequency and class performance is very high (0.76 on ImageNet; see Appendix (B.6). In contrast, there is correlation between its vision encoder CLIP-L'S performance those classes with the same VLM training data, and after fine-tuning LLaVA1.5- 5-7B on the ImageNet classification data, the strong correlation disappears. Combining all of results, we conclude that data plays a critical role in determining the VLM's classification performance.
Given that data availability is critical for VLM classification performance, now examine whether the data type specifically impacts results. For instance, to train a VLM to classify a class like "guinea pig." should it rely on cesifficaatioffocuusee data (c.g.. "<image> What type of object is in this photo?
LLLaVA training contains stages. Here combine the data from the two stages. More in Appendix B.6
7

---

Model
Fine-tuning Data Type
LLaVA1. 5-7B Zero-shot
5.9
0.0
47.1
CLIP-L Fine-tuned
Classification Data
98.6
91.5
97.6
LLaVA1 5-7B Fine-tured
Classification Data
97.6
90.4
97.5
LLaVA1 5-7B Fine-tuned Captioning Data
92.0
85.4
95.7
Table 4: Analysis of data types. We fine-tune the VLM on the caption-focused data generated by GPT4 using the same experimental settings as Table 3 and find that data is the main determining factor for VLM performance, and the data type does not matter much.
Guinea pig") or can it use broader types, such as caption-focused data (e.g., "<image> Generate a caption for the image. A guinea pig playing in the grass") or VQA-focused data (e.g.. "<image> What is the native region of this guinea pig? South America")
In previous experiments (see (3.2), we fine-tuned the VLM with a classification-fousse template. Here, we fine-tune the VLM on caption-focused data generated by prompting GPT4 to produce captions incorporating each class's ground-truth name. Specifically, we used the prompt: "Generate short caption for the image. The caption must include the ground-truth label of the image, which is <class name>." This approach produces diverse captions per class, such "A guinea pig playing in the grass" or "A close-up view of guinea pig in an indoor environment." " We then fine-tune the VLM on this caption- fooused data using identical experimental settings and evaluate it against the model data. Success is determined by whether the generated caption for
trained with classification-focuut validation image includes the ground-trute truth class name.
We find that data presence is the primary factor for VLM classification performance, with data type playing minimal role. From Table we observe that models fine-tuned on both classification- and caption-focusee focused data perform similarly on Flowers 102, StanfordCars, and Caltech 101, achieving accuracy gains of at least 48.6% over the con-finnetunnet model. This suggests that enabling VLM to classify a specific class does not require caasificationnsseciff data; any data containing the class name suffices.
These findings emphasize a critical data-centric view of VLM training. A common question for VLM training is whether alignment between vision encoder outputs and text generation necessitates multi-modal data with specific class labels, despite both vision and language components seeing these classes in uni-modal pre-training. Previous studies have suggested that this alignment stage may be data-efficient or even unnecessary [32 18. likely due to their evaluation settings not requiring vision- centric fine-grained recognition. Our work, however, shows that 1) multi-modal data is essential for alignment and 2) increasing data quantity linearly boosts performance. Concurrent work from DeepMind supports our findings, showing that most VLM tasks benefit from pre-training larger datasets 6
Improving VLM with Classification Data
Based the analysis presented in $ in this section, discuss the enhancement of visually- grounded language models ( (VLMs) by integrating cuasiffication-fccusse data into their training. We demonstrate that this data intervention not only boosts the VLM's classification performance but also enhances its general capabilities.
4.1 Motivation
Classification is fundamental to enabling more advanced capabilities of VLMs, such as visual question answering and reasoning. For example, suppose a virtual assistant is helping visually impaired individuals prepare mushrooms as food. In that case, the model must correctly identify the mushroom species to answer questions like "Is this mushroom poisonous?" The poor classification performance of VLMs lays weak foundation for their advanced capabilities.
we identify the primary cause of poor classification performance the lack of data. Therefore, we propose a straightforward solution: integrating classification- focused data into the VLM training process. We hope that incorporating classification data not only enhances classification accuracy but also improves general capabilities.
From 3

---

Model
Model
Acc
Naive Baselines
Public VLM
Random
25.0
BLIP2-2.
21.7
Max Freq
25.9
IBLIP-7B [T]]
36.3
GPT4 w/ GT Class
100.0
LLaVANeXT-V7B
37.0
GPT4 w/o Image
0.0
IBLIP- 10]
37.5
Human w/ GT Class+Wiki
96.5
LLaVA1.5-12 B2
37.8
(LaVA1.5-773 w/ GT Class
55.9
LLaVA1 .-7B [32
38.0
LLaVANeXT-MMT [3]
41.9
Proprietary VLM
GeminiPro [44)
49.1
Finetuned VLM
Claude3 3
54.3
LLaVA1.5-78 Finetuned ImageNet
30.6
GPT4 [36
61.2
LLaVA1. 5-7B Finetuned on ImageNett+LLaA
49.8
Table 5: Evaluations of VLMs on
ImageWikiQu is multiple-choice question- answering dataset collected by feeding the Wikipedia pages of ImageNet classes to GPT-4. We find that current VLMs perform poorly in answering these questions, suggesting that their poor classification performance is fundamental limitation for more advanced capabilities. Integrating classification data into VLM training enhances both their classification and overall capabilities.
4.2 ImageWikiQA
To verify our hypothesis, need dataset that evaluates both the classification and advanced capabilities of VLMs. However, current visual question answering benchmarks, such as VQAv2 (15), MM-Vet [51], and MMMU [52], primarily focus on advanced capabilities, such reasoning and knowledge grounding, rather than classification. The objects in these datasets are relatively simple, and questions often be answered by identifying only the general category of an image, such as "mushroom" or "flower", rather than specific types of mushrooms or flowers. contrast, classification datasets have no questions related to more advanced capabilities.
To bridge the gap between classification and advanced capabilities, we introduce ImageWikiQA, an object-centric, knowledge-intensivg question answering dataset that combines both worlds. Each question in Image WikiQA is a multiple-choiis (c.g.. "<image showing guinea pig> What is the native region of this object? A. South America; B. Africa; C. Asia; D. Australia"). Although each question does not directly ask for the category of the object within the image, the question only be accurately answered if the class of the object correctly identified. Example questions from ImageWinn WikiQA can be found in Figure T (right) and Appendix $
question with four options and one correct answer
ImageWikiQU) is created by generating questions based on Wikipedia pages of ImageNet classes using GPT4. Specifically, for each class in ImageNet, we parsed the Wikipedia content of the class following Bujwid et [8] and then provided this content along with prompt to GPT4 to generate five questions per class. We instructed GPT4 replace the class name with "this object" in the questions so that the ground-trum class name not provided. We retained all questions that GPT4 could answer correctly with the ground-trun -trnt class name and could not guess the answer without the class name. To ensure the question quality generated by GPT4, four human annotators attempted to answer the questions with the ground-truth class name and Wikipedia page and achieved an accuracy of 96.5%. Afterward, we randomly sampled at most 3 ImageNet images for each question, rebalanced the choice distribution, and composed the final Wikie dataset. In total, there 2000 multiple-choice questions, each with an image, question, four candidate choices, and a reference to Wikipedia sentences, with random guess accuracy of 25.0% and frequency accuracy of 25.9%.
4.3 Results
Table presents the performance of various VLMs on the ImageWikiQA dataset.
We find that current state-of-the-art VLMs perform poorly on answering these questions given images. For example, GPT4 achieves 100% accuracy when the ground-truth class name is pro- vided, but only achieves 61.2% accuracy with images. Similarly, Claude3 and GeminiPro only achieve 54.3% and 49. 1% accuracy, respectively. These results indicate that the poor classification performance of VLMs is a fundamental limitation for more advanced capabilities.

---

Furthermore, we find that integrating classification data into VLM training improves its classifi- cation and general capabilities. We fine-tuned LLaVA1. 7- on the ImageNet 1.28M classification data and original 665K LLaVA instruction-tuning data (training detail in $C222. which is able to achieve 84.4% accuracy on ImageNet classification compared to 22.8% for non-fine-tuned models. This improvement in classification translates to an 1188 accuracy boost on ImageWikiQA, demon- strating classification is indeed a foundation for VLM's advanced capabilities. However, it is worth noting that fine-tuning solely on the classification task harms general capabilities. When fine-tuning LLaVA1. 5-7B on ImageNet only, their accuracy on ImageWikiQA drops to 30.6%. Therefore, fine- combined dataset. Developing fine-tuning methods that prevent
tuning should be performed such catastrophic forgetting is a promising future research direction.
Related Work
Visually
i'sually-gounnee grounded language models. Visually-grounded language models (VLMs) refer
large family of models that integrate visual signals into language models by modeling p(y W9 2),
where y; a text token and x a visual input such as an image or video. Recently, many powerful
VLMs have been developed, including proprietary ones like GPT-4V (3). Gemini [44] Claude [3]
Flamingo 2 and public ones like LLaVA [32 31) BLIP 27 TO, OpenFlamingo
Otter [26].
5 The typical architecture of VLMS comprises three components: a visual encoder (often CLIP (37), language model, and projector that bridges the visual outputs and language model inputs. The projector can be simple as linear layer or MLP (e.g., LLaVA (62)
Fuyu D. QwenVL. VL.
or a complex architecture such as Transformer with cross-modal attention (e.g., BLIP (27)) VLMs are usually trained on image-text captioning data 40 39] and carefully designed instruction-tuning
data [32] with both the vision encoder and language model initialized from pre-trained versions. In this work, we evaluate widely used VLMs in classification settings and analyze two prominent architectures, LLaVA 32] and BLIP 27
Analysis of gisuullygrruunded language models. While VLMs have demonstrated impressive performance, many questions remain unanswered. Recent works have explored their limitations from various perspectives, including architectures (34), training recipes [8 24), data [4 46). vision encoders 45), language models [2] 44, input resolution [41], and proposed solutions for these
limitations. For instance, Tong et al. 45 discovered that CLIP vision encoders, employed by most VLMs, often fail to distinguish between certain image pairs despite their apparent visual differences. They suggested utilizing alternative vision encoders, such DINO, to address this problem. Our work aligns with these studies in understanding VLM limitations and proposing solutions. We find that current VLMs struggle with image classification, and we thoroughly investigate the reasons behind this, proposing a simple solution to integrate classification focused data into VLM training.
Image classification. Image classification is one of the most fundamental capabilities of machine intelligence. Starting in the 1990s, researchers collected datasets like MNIST (25) for digit classi- fication and CIFAR (22) for object classification, using various machine learning algorithms such as SVMs and MLPs to tackle the problem. A significant milestone was the ImageNet [1], which elevated the scale and quality of datasets to new level, driving advancements in deep learning. With the rise of deep supervised and self-supervised learning 23 9. performance on these datasets soon began to saturate. As a result, classification models have superseded most human labelers, leading researchers to focus on more advanced visual intelligence tasks like visual reasoning. In this work, we revisit this simple yet fundamental task, discovering that current VLMs struggle with it. We demonstrate that classification remains essential, as it forms the foundation for more advanced capabilities, and enhancing VLMs' classification capabilities improves their overall performance.
6 Conclusion
In this work, we explored the use of soually-grounder language models (VLMs) as image classifiers. We found that their performance limited across various datasets. We then analyzed the reasons behind these limitations and, based on our findings, trained a VLM with enhanced general capabilities.
10

---

Acknowledgements
We thank Irena Gao, Jeffrey Gu, Weixin Liang, Alejandro Lozano, Shiori Sagawa, Ruocheng Wang. Yunzhi Zhang, Orr Zohar for providing valuable feedback to this project. This work is in part supported by the NSF Al Institute for Foundations of Machine Learning (IFML), Open Philanthropy. and the Allen Institute for AL. Serena Yeung-Levy is Chan Zuckerberg Biohub - San Francisco Investigator.
References
[1] Adept. Fuyu-8b: A multimodal architecture for agents, 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[3] Anthropic. Introducing the next generation of claude, 2024.
[4] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01333 2023.
[5] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
[6] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, al. Paligemma: versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726, 2024.
[7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
[8] Sebastian Bujwid and Josephine Sullivan. Large-scale zero-shot image classification from rich and diverse textual descriptions. In EACL Workshop, 2021.
[9] Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICMI., 2020.
[10] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purper lision-language models with instruction tuning. In NeurlPS, 2023.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[13] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop, 2004.
[14] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. In NeurlPS, 2023.
[15] Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017.
11

---

[16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2020.
[17] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
[18) Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07755 2024.
[19] James Kirkpatrick. Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnicszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 2017.
[20] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurlPS, 2022.
[21] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshop, 2013.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[24] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246, 2024.
[25] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdd/nmistl/ 1998.
[26] Bo Li, Yuanhan Zhang, Liangyu Chen, Jínghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03776,, 2023.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 20115 2023.
[28] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[29] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP, 2023.
[30] Yuanqing Lin, Fengjun Lv, Shenghuo Zhu, Ming Yang, Timothee Cour, Kai Yu, Liangliang Cao, and Thomas Huang. Large-scale image classification: Fast feature extraction and svm training. In CVPR, 2011.
[31] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[32] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[33] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. TACL, 2023.
[34] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403.00 022443309611, 1, 2024.
[35] M-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008
12

---

[36] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.0877, 2023.
[37]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
[38] Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammana- manchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint arXiv:2306. 178006 2023.
[39] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. L0ion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 2021.
[40] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
[41] Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang. and Trevor Darrell. When do we not need larger vision models? arXiv preprint arXiv:2403. 13043, 2024.
[42] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
[43] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprini arXiv:2303. 1..3303.1388) 15389, 2023.
[44] Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean- bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 1xXx:23312.1800 2023,
[45] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint 24ii:2401.002000 2024.
[46] Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining con- cept frequency determines multimodal model performance. arXiv preprint arXiv:2404.0444 04125, 2024.
[47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
[48] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
[49] Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang, and Jingdong Wang. Gpt4vis: what can gpt-4 do for zero-shot visual recognition? arXiv preprint arXiv:2311.15732. 2023.
[50] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfci Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. TMLR, 2022.
[51] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 024900 2023.
[52] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
[53) Zihao Zhao Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML, 2021.
13

---

Broader Impacts and Ethics Statement
In this work, we identify core limitation of visually-groundee language models (VLMs) and propose simple solution to address this issue. We find that VLMs underperform in image classification. By integrating classification data into VLM training, we have demonstrated that this intervention not only enhances classification accuracy but also improves the general capabilities of VLMs. This advancement holds promise for real-world applications, such as virtual assistants for visually impaired individuals. However, there potential risks associated with incorporating race-based or gender- based classification data, which could amplify discriminatory biases VLMs. We strongly emphasize the importance of ethical and responsible use of our approach to ensure it contributes positively to our digital ecosystem.
Reproducibility Statement
We provide an open-source implementation of our work and have released the ImageWikiQA dataset at https://g:l thub. com/yuhui- 2h15/ 2a115/LMMcaas/// assifferr This will enable researchers reproduce all experiments described this paper and conduct their own analyses on additional VLMs and datasets.
Limitations
Due computational constraints, we cannot evaluate all possible VLMs and datasets. We select two representative ones, LL and BLIP, and conduct analyses on four datasets. Additionally, as find that VL.Ms have information in their latent space, conduct large-scale training decode them. Developing a zero-shot method to decode this information without extensive training could be promising direction for future research.
Compute Resources
We use four NVIDIA L40S GPUs for all experiments. Most inference experiments take 5-20 hours on single L40S GPU. The most computationally intensive step is fine-tuning. Fine-tuning LLaVA1. 5-7B 1.28M ImageNet data takes 1.4 days for one epoch using four GPUs. Fine-tuning LLaVA1 5-7B on 1.28M ImageNet data combined with 665K original LLa VA instruction-tunigg data takes 3.5 days for one epoch using four GPUs. For the 13B model, the cost approximately 40% higher.
Summary of Appendix
We provide additional details for each section in the main paper.
In A
we provide details of models and data for 2
. In B
we provide details of experimental settings and results for
In we provide details of our collected ImageWikiQA dataset for
In
we summarize the contributions of our work.
A Supplementary for Section 2
A.1
Model
We include model details in Table
A.2 Data
We include data details in Table 7
14

---

Model
Link
Property
Version
GPT4 35
https://llatormm. www.emooonnaiiiiio..io.. howpa///..ooooooodudu/////iioonn
Proprictary gpt-4-turoo-2224-00
GeniniPro 44
https//ai.googll.ddey.
Proprictary gemini-L.0-pro-vsilee
Claude3 [3]
https anthropic com/
Proprictary -laude33-ppus2222-44
LLaVA1.5-78 32
https//hugginggien https//haggiiggfeec.couuuatttttu wow.iuaatttiiaaalaaaa.........
Public
LLaVA1.5-138 332
htpp://huggingaaan https///hnggiigiaacccc//iaaaaaaa wot.///aaacciii/l/aaaaa.........
Public
LLaVANeXT--V7 31
Public
LLLVANNeTT-M78 31
hilp://huggingfiee http:///wgnngieeee..c//uuuuu//a/ www.uuuooiiiiiia/aaa///.aaaaaaaa
Public
1.TPP2-2.BB 2
Public
IBLIP-7B 10
hhips://uuggingfacc.
Public
IBLIP-13B 10
Public
CLIP-L 37
https:/gittub..oo/opnnai///
Public
EVA-G
https://gthbb..ommbaaiiaaanog
Public
Table 6: Model details.
Dataset
Link
Training
Validation
Classes
ImageNet
I
http://www.inage-nee.
1.28M
50K
1000
Flowers102 35
flowers102
2.0K
6.1K
102
StanfordCars
http://www.eeorrroooooyoaaaaasas 196
8.0K
Caltech101 13
4.3K
4.3K
101
Table 7: Data details.
Model
Accuracy w/o Synonyms
Accuracy w/ Synonyms
Improvement (A)
BLIP2-2.78
27.8
IBLIP-7B
14.6
16.5
1.9
IBLIP-13H
14.7
16.6
ShaVA1.5-77
22.8
24.6
1.8
LLaVANeXT-V73
29.4
32.2
2.8
LLaVA1.5-1
24.3
26.0
1.7
LLaVANeXT-MM7
32.3
35.1
2.8
Claude3
53.6
56.3
2.7
GeminiPro
39.2
42.5
3.3
GPT4
48.5
51.1
2.6
Table 8: Model accuracy with and without label synonyms on ImageNet, and corresponding improvement.
A.3 Synonymous Text Labels in ImageNet Evaluation
In ImageNet, many classes have multiple valid textual labels. For instance, the class identifier n0 n01496331 corresponds to both "electric ray" and "crampfish". To address this, we incorporated label synonym in our evaluation. We treat the classification as correct if any textual label is included in the VIM generation. Including synonyms improves accuracy by only 1%-3% (Table 8 still leaving notable performance gap when compared CL.P.
B Supplementary for Section 3
B.1 Prompt Variation Detail
Prompt details. We use three different prompts with varying wordings. Base Prompt: "What type of object in this photo?" Prompt Alternative 1: "Identify the object in this image." Prompt Alternative 2: "What is the main object depicted in this photograph?"
Label order details. For the fixed order, we use the default dataset label order for each image.
15

---

ImageNet
Flowers
Cars
Caltech
10
25
20
2
5 20 100
2
100
20
100
2
5
100
Relative
LLaVA1.5-78
Figure 4: Analysis of the label set size. For each image, we randomly sample 100, 20, 5, and candidate classes from the entire set of classes. While the absolute performance gap between VLMs and CLIPs decreases as the number of classes is reduced, the relative performance gap increases. The X-axis represents the number of classes, and the Y-axis represents the relative error rate between LLaVA1.5- 5- and CLIP-L.
Model
# Classes
LLaVA1.5 7B
2
94.3
67.1
78.5
96.4
5
92.3
50.5
67.9
96.6
20
82.0
29.6
22.0
91.8
100
46.6
10.0
1.0
62.1
BLIP2-2.70 7B
14.3
17.4
29.6
5
11.9
16.9
1.3
26.7
12.2
1.5
20.9
100
5.5
11.2
3.9
22.4
CLIP-L
2
99.8
96.9
99.8
99.9
5
99.2
92.8
99.1
99.8
20
97.5
85.7
96.1
98.9
100
92.7
76.3
85.4
95.7
Table Analysis of the label set size. This is the table version of Figure
B.2 Label Set Size Details
Performance details. The table version of Figure 2 provided in Table By reducing the number of labels for classification, we can narrow the gap between VIM and CLIP, but gap remains across all label set sizes, with just two labels (two-way classification).
Relative performance gaps. We find that while the absolute gap between VLMs and CLIPs narrows with reduced label size, the relative gap increases. For example, in two-way classification on ImageNet, VLMs have 5.7% eиrоr rate, while CLIP has a 0.2% error rate, resulting in 28.5x gap; for 20 classes, VLMs have an 18.0% error rate, while CLIP has a 2.5% error rate, resulting 7.2x gap. This trend is indicated by the error rates between VLMs and CLIP shown in Figure
Larger gap on fine-grained classification datasets. The absolute gap on Flowers and Cars is larger than ImageNet and Caltech when reducing the label set size. This may be because Flowers and Cars are more fine-grained classification datasets, while ImageNet and Caltech are more coarse-grained. VLMs are weaker in fine- fine-grained classification compared to CLIP, resulting in larger gap.
B.3 Inference Strategy Details
Experimental details. Probabilistic inference methods require separate computations for each class, which introduces significant computational costs. For example, on ImageNet with 1000 classes, each example takes 1000 times longer to process than with direct generation pipeline. Therefore, randomly sampled 1000 examples from each dataset's 's validation set for evaluation.
16

---

Position
100
200
300
500
600
-2
-1 (Last)
Average
Accuracy 0.7
17.1
18.1
25.0
25.8
23.7
52.9
84.0
85.8
94.5
96.2
Table 10: Probing the last token or the average token results in much better performance
than probing other token positions. Experiments are done using LLaVA1. 5- on the Flowers102 dataset.
B.4 Information Loss Details
CLIP global vs. local features. The global feature of CLIP is commonly used in traditional classification settings, which has token each image. However, VLMs utilize local patch features for classification, which has 576 tokens for each image using CLIP-L. One might hypothesize that local features contain less information than global features. In practice, VLMs use the second-to-last layer of CLIP local features rather than the last layer. Since the last layer's global feature is the weighted average the second-to-last layer's features through self-attention computation, we can theoretically conclude that local features have at least the same level of information CLIP global features.
Feature extraction details. We extract features from VLM's last layer by feeding the LLaVA default prompt "USER: <576 Image Tokens> What type object in this photo? ASSISTANT:" into the LLaVA, BLIP default prompt "<Image Tokens> Question: What type of object is in this photo? Answer:" into the BLIP. We use either the last token feature (i.c., the token ":") or the average feature over all the tokens.
Linear probing details. We train the linear layer on the training set with a batch size of 512, a learning rate of le-3 using the Adam optimizer, and for 500 epochs. After training, report the best performance on the validation set.
Probing position. In practice, find that probing the last token the average token results in much better performance than probing other token positions, as shown in Table TO We leave this as an interesting question for future research.
B.5 Training Objective Detail
LLa VA fine-tuning details. We convert each image and class label into the text format using the LLaVA default template "USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT: <Class Name>." " We conduct two settings for fine-tuning LLaVA. the first setting, only fine-tune the MLP projector between CLIP and the language model (LM). The projector is trained on the training set with batch size 64 and a learning rate of 2e-5 using the AdamW optimizer for 50 epochs (1 epoch for ImageNet), with warmup ratio of 0.03. In the second setting, we fine-tune both the MLP projector and the LM using LoRA. The projector and LM are trained on the training set with batch size of 64, learning rate of 2e-5 for the projector and 2e-4 for the LM, using the Adam W optimizer for 50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03, a LoRA rank of 128, and LoRA alpha of 256. For both settings, we report the best performance on the validation set after training. Fine-tuning on ImageNet for epoch requires 130 L40 GPU hours.
BLIP fine-tuning details. We convert each image and class label into the format using the BLIP default template "<Image Tokens> Question: What type of object is in this photo? Answer: <Class Name>." We train the Q-former projector between CLIP and the LM on the training set with batch size of 64, a learning rate of 2e-5, and weight decay of 0.05 using the Adam optimizer for 100 epochs (10 epochs for ImageNet), with 1000 warmup steps. After training, we report the best performance on the validation set. Fine-tuning on ImageNet for 10 epochs requires 120 L40 GPU hours.
CLIP fine-tuning details. We initialize linear layer on top of the CLIP image encoder. The CLIP model is frozen, and only the linear layer is trained. The linear layer is trained on the training set with a batch size of 512, a learning rate of le-3, and no weight decay using the Adam optimizer for 300 epochs (40 epochs for ImageNet). After training, we report the best performance on the validation set.
17

---

train/loss
pragene-loo--tte firetnellavatb imagenet finetunellavatt homprealoooommooggeeoooooonnn imagenet-oon
firetunellvaalb prgp://aaatooooggeeoooooos
14
12
6
2
/slobal
10k
15k
train/lost
finetunellava To-cers-lora
Enetunellava llvoaab-looooeesorr aeteeuellllaa celbech pht--allcc----ppectto
wowwrssersss-sspp.ooo projectoor
train/plohal step.
Figure 5: Fine-tuning only the projector improves numerical stability. (Top) Fine-tuning LLMs with LoRA often results in numerical instabilities, manifesting as spikes in loss (purple, green, brown, orange curves). In contrast, fine-tuning only the projector leads to a consistently steady decrease in loss (teal curve). Despite experimenting with various hyperparameters for ImageNet, the instability remained. (Bottom) Occasionally, the spikes normalize with continued training. Here, present an example using the StanfordCars dataset (pink curve).
Spcarman Correlation
Pearson Correlation
C
CLIP-L
-0.13
0.30
0.04
-0.03
-0.08
0.08
0.00
LLaVA1.5-77 Fine-tuned Imageleet
-0.02
-0.03
0.04
0.30
-0.05
0.08
0.04
0.18
5-VVA1.5-7B
0.76
0.40
N/A
0.69
0.35
0.64
N/A
0.27
Table 11: Correlation between class frequency and model's accuracy on the class. This supple-
ments Figure 3
Numerical instability. When fine-tuning both the projector and the LM with LoRA for LLaVA, we experience significant numerical instability. ImageNet, despite adjusting three different hyperparameters such as the learning rate and batch size, the loss consistently peaks during training and does not recover. On the StanfordCars dataset, numerical instability is also observed, but the loss returns to normal during training. This can be seen in Figure
18

---

Method
LLaVA1 5-7B
G
Prompt Variation
Base Prompt
22.8
5.9
0.0
47.1
48.5
51.0
0.1
61.0
w/ Label (Fixed Order)
N/A
6.1
0.1
70.5
61.2
79.0
57.1
91.2
w/ Label (Random Order)
N/A
10.2
0.0
62.1
60.6
79.9
58.2
94.2
w/ Label (Random Order) + CoT
N/A
18.1
0.4
64.5
62.2
79.0
59.8
94.0
Table 12: Analysis of VLMs from the inference perspective. Compared to Table 2 added the results for the proprietary GPT4 model. We explore prompt variation such as label order and chain- of-thought (CoT) and find it has limited impact the performance, which applies to proprietary
VLMSs such as GPT4.
ImageNet
Flowers
Cars
Caltech
100
100
100
80
80
90
60
40
40
50
20
100
20
100
LLaVA1 5-78
CLIPAL
GPT4
Figure 6: Analysis of the label set size. Compared to Figure 2 added the line for the proprietary GPT4 model. LLaVA1.5-78 uses CLIP-L as its vision encoder, while GPT4 presumably employs superior vision encoder. Therefore, comparing GPT4 to CLIP-L not entirely fair. However, observe that GPT4 performs worse than CLIP-L on ImageNet even when the number of classes is reduced to 2.
B.6 Data Detail
Data processing details. The training of LLaVA comprises two stages: pre-training and instruction tuning. The pre-training stage utilizes noisy image captioning data, whereas the instruction tuning stage employs meticulously curated multi-turn conversation data. We tokenize all sentences within the data, compute the frequency for each class, and then calculate the accuracy for each class. This results in 1000 (frequency, accuracy) pairs for ImageNet's 1000 classes. This process is similarly applicd to all other datasets.
Correlation results. We compute the Spearman and Pearson correlations between frequency and accuracy of zero-shot LLaVA1.5- 7B. The results are presented in Table II For comparison, conducted the same analysis for CLIP-L and fine-tuned ALaVA1155-75
Correlation for two stages. Given that LLaVA training involves two stages, we examine whether pre-training instruction-tuning data has a greater impact on classification performance. We compute the Spearman and Pearson correlations separately for pre-training and instruction-tune tuning data. Our findings indicate that the correlation similar for both stages. For instance, on ImageNet, the Spearman and Pearson correlations are 0.73 and 0.34 for pre- pre-training, and 0.76 and 0.33 for instruction tuning. These values are close the combined data correlations, which are 0.76 and 0.35 for Spearman and Pearson correlations, respectively, as reported in Table 11
B.7 Analysis of Proprietary VLMs
In the main paper, we analyzed two public VLMs about why they are bad at image classification. For proprietary VLMs, they haven't released the training details and data usage, so it is impossible to directly address training-related
telatee hypotheses.
For inference-related hypotheses, we conduct additional experiments with GPT4 (Table 12 Figure 6 and the conclusion is the same as the public VLMs. For example, GPT-4 achieves 60.6% accuracy on ImageNet without COT. With COT, GPT-4 achieves 62.2% accuracy, demonstrating that CoT has limited impact image classification, even for large VLMs.
19

---

For related and data-related hypotheses, recent work PaliGemma [6] from DeepMind shows that most VLM tasks benefit significantly from longer pre-training with more data (in its Figure and Appendix K). This aligns with our main conclusion that data is the critical factor in improving VL.M performance.
B.8
Open-world -lored-world Setting
3 $4 all use "open-world" settings (not providing the label set in the prompt), except for prompt variation analysis and label set size analysis in $3.1 using "closed-world" settings (providing the label set in the prompt).
Naturally, the "elosed-worid" setting narrows the model's generation space and increases accuracy. When the model can easily predict the class in the label set by modifying the inference space (probabilistic inference in $ and probing VLM in $3.2) or training VLMs with more classification data ($4), the advantage of the "closed-world" setting doesn't exist. Thus, for these experiments, we use the "open- world" setting.
B.9 Number Comparison of Table
1 vs. Table 2
Table "Base Prompt w/ Label (Random Order)" is equivalent to Table "Closed- World Setting". In this setting, we concatenate label candidates in a random order for each question: "What's in the image? Choose one from random shuffle([cat, dog, pig])". The accuracy for these two settings is identical between Table 2 and Table I
Table 2 "Base Prompt w/ Label (Fixed Order)" concatenates label candidates in a fixed order for each question: "What's s in the image? Choose one from [eat, dog, pigl". This setting used rule out the possibility that the order of labels might affect model performance.
B.10 Why Do VLMs Have Information but Cannot Decode?
In 3 we find that VLMs have the essential information for classification. If the VLMs have the information, then why are they not able to decode it?
One possible reason is that the VLM's decoding space is too large and not aligned with the visual features. Specifically, VLM decoding performed through next-word prediction, which usually involves vocabulary of over 10,000 words. The output text embedding is not aligned with the visual features due to insufficient data to align these spaces.
In general, having information in a model does not necessarily mean the model can express that information. This phenomenon is also observed in other research areas. For example, after training ResNets or ViTs with self-supervised learning methods like SimCLR, the model acquires discrinina- tive information for different classes. However, this information can only be decoded by adding and training a linear layer on new dataset.
Similarly, demonstrate that VLMs possess classification information, but it is not readily express- ible. Adding dassification-relatee data helps bridge the gap between "information possession" and "information expression" (refer to Table 3).
B.11 Why Does Fine-tuning Only on ImageNet Get Worse Performance on Image WikiQA?
In Ta the "Finetuned ImageNet" model performance (30.6) is worse than the non-finetuned model performance (38.0). This is because fine-tuning often improves in-distribution performance but does not necessarily enhance out-of-distribttion or general capabilities.
Table presents the results of models fine-tuned on ImageNet and evaluated on Image WikiQA, which are vastly different datasets. ImageNet questions only require classification, such "Q: <image> What is in the image? A: dog," while ImageWikiQU WikiQA questions demand both classification and knowledgeereasooingg such "Q: <image> What is the native region of this object? A: South America."
Fine-tuning solely on ImageNet trains the model to classify, but it can lead to a loss of general capabilities like reasoning and knowledge, resulting in lower performance on ImageWikiQA (30.6 fine-tuned vs. 38.0 pre-trained). This phenomenon known as "catastrophic forgetting" [19].
20

---

100
Imageler Accuracy
40
LLaVA
Rinetuned LL
Unner Probed LLaVA
[20. (438 class) dass)
[30.
[30,
130, [60, 170, class)
[80. [100. 1KK
122
Class Frequency
Figure 7: Analysis of linear probed VLM from the data perspective. Compared to Figure 3 added the line for the linear probed LLaVA model, which has zero correlation. The zero correlation is expected because probing on the ImageNet dataset equalizes class frequencies, ImageNet classes are evenly distributed.
However, when we fine-tune on both ImageNet and instruction tuning data, the model learns classifi- cation while retaining its original capabilities, leading to significant performance improvement on ImageWikiQA (49.8 fine-tuned 38.0 pre-trained).
B.12 Analysis of Linear Probed VLM from Data Perspective
Figure 7 plots results for the linear probed LLaVA model, which aligns closely with the fine-tuned LLaVA. This alignment shows zero correlation with class frequency since fine-tuning probing on the balanced ImageNet dataset equalizes class frequencies, where each class evenly represented.
C Supplementary for Section 4
C.1
Details
ImageWikiQA only contains test set, providing valuable benchmark for assessing both fine-grained classification and knowledge-based reasoning VL.Ms.
Data collection details.
The ImageWikiQA dataset was constructed in two stages. In the first stage, we generated questions the class level for each of the 1,000 ImageNet classes. For instance, questions like "What the native region of <class name>?" were created to capture high-level class information. In the second stage, we refined these questions at the image level. For each ImageNet class, we randomly sampled 3 images from the 50 available images from the ImageNet validation set and formulated specific questions like "What is the native region of <image i>?". Each question in ImageWikiQA is paired with single image, ensuring only one image per question.
The ImageWikiQA dataset contains 2,000 unique questions. question like "What the native region of <image i>?" is counted multiple times if it is asked for different images classes. For example, if the question is asked for 2 cat images and dog images, it would count questions in total. To avoid over-representing any particular question, question like "What is the native region of <image i>?" is limited to a maximum
occurrences across the 2,000 questions.
Data generation prompt. We provide the prompt used to generate ImageWikiQU in Figure 8 and the prompt used to filter ImageWikiQA in Figure
More examples.
We provide more examples of the ImageWikiQA dataset in Table 14
C.2 Results Details
LLaVA fine-tuning details. For ImageNet, we convert each image and class label into the text format using the LLaVA default template "USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT: <Class Name>." Then, we concatenate the original LLaVA 665K instruction tuning dataset with the 1.28M ImageNet classification dataset. Due to numerical stability concerns, we train only the projector layer of L.L.V. The projector is trained on the combined dataset with batch size of 64, learning rate of 2e-5, using the AdamW optimizer, for epoch, with a warmup ratio of 0.03. The training process takes approximately 340 hours on single L40 GPU.
21

---

Come with five multiple choice questions for a {classname) to examine the knowledge of expert.
The questions should come from the following Wikipedia articles on 4 {classname):
frikipedia page) <cr
** Instruccion:::
Each question should have four choices, of which is the correct answer.
Note that the Wikipedia articles will not be accessible to the
+ test-takers, so please do not reference apecific details from + the articles in the questions.
Use "this object" rather than "(classname)" to give test-takers an image this object, not the word 4 itself.
the questions. We
Each question should have four fields: "question" (str), "choices" (list [strl), "ansver" (int, starting from 0)), and "reference" (str, original sentences from Wikipedia articles).
Output in JSON format.
Figure 8 Prompt used to generate ImageWikiQA using GPT4.
Answer the multiple choice question below. {question with ground truth class name}
A. {choice A}
B. {choice B}
c. {choice C}
D. {choice D}
Output only one character A/B/C/D.
Answer the multiple-choice question below. (question without ground ttruh class name}
A. {choice A}
B. {choice B}
C. {choice C)
D.
{choice D}
The question mentions "this object". However, we don't know what the
4 object is. Try your best to guess the ansver without knoving
+ the object.
Output only one character A/B/C/D.
Figure 9: Prompt used filter ImageWikiQA using GPT4,
C.3 Performance of Fine-tuned VLM on Other Datasets
To understand whether incorporating the ImageNet 1. 1.28M classification data into the original LLaVA instruction-tuning data will harm the model's general capability, we further evaluated the performance of our fine-tuned esuualy-grounder language model (VLM) additional benchmarks to ensure its robustness across a variety of tasks. In particular, we tested the model on TextVQA [42], POPE [9], and MMVet [5T). The results, presented in Table 13, show that the fine-tuned LLaVA1. 5-7B maintains almost identical performance compared to the original version.
This result aligns with expectations since our fine-tuning process incorporated all of the original LLaVA training data during instruction tuning. For instance, on TextVQA, the fine-tuned model achieved an accuracy of 58.0%, closely matching the original model's 58.2% 2%. Additionally, the fine- tuned model slightly outperformed the original on POPE Popular and POPE Adverse benchmarks, showing small improvements of 0.2% and 0.3%, respectively. Performance MMVet remained
22

---

Model
TexcVQA
POPE Popular POPE Adverse MMVet
LLaVA1.5 5- (Official Released)
58.2
86.1
84.2
31.1
L 7- (Further Finetuned)
58.0
86.3
84.5
31.1
Table 13: Performance of LLaVA1.5-7B before and after fine-tuning on TextVQA, POPE, and MMVet datasets. Fine-tuning resulted in consistent performance across all benchmarks.
unchanged at 31.1%. 1%. These findings demonstrate that the fine-tuning process preserves the model's strengths across these varied benchmarks.
C.4 Low Accuracy on WikiQA with Ground-truth Class Names
Table 5 presents the accuracy on Image WikiQA, which includes questions derived from Wikipedia requiring extensive world knowledge (e.g.. "What the native region of the guinea pig?").
The modest accuracy of 55.9% by LLaVA1. 5-7B with provided ground-truth class names reflects its limited world knowledge for ImageNet classes. This limitation aligns with findings that smaller models, like Vicuna-7B, lack world knowledge compared to larger models like GPT-4.
C.5 Other Fine- Fine-Tuning and Inference Strategies
Other fine-tuning inference modifications, such as applying linear probing loss in the VLM output space during training or using k-nearest neighbors (KNN) in the VLM output space for evaluation, also enhance VL.M classification performance.
However, our primary goal is to improve VLMs' general capabilities across a range tasks, not just classification. The universal inference interface for various tasks is text generation, whereas approaches like KNN-based inference are tailored solely to classification and do not generalize to other task types. For fine-tuning, linear probing loss. This approach did not yield improvements on ImageWikiQA, indicating that task-specific fine-tuning does not enhance the overall capabilities of VLMs.
experimented with adding
token and applying
In summary, without changing VLM training or inference to maintain general capabilities for different tasks, adding data is the most promising and effective approach.
D Summary of Contributions
Object recognition is fundamental to the general capability of visually-grounded language models (VLMs), yet current VLMS perform poorly in this We the first to thoroughly investigate this critical issue and propose potential solution to improve it.
Our primary contribution lies in identifying the problem (i.e., VLMs are inadequate image classifiers, significant weakness that has been overlooked) and conducting an in-depth in- vestigation (i.e., understanding why VLMs are poor image classifiers and how to address this issue).
In summary, our contributions are threefold:
We thoroughly evaluated current public and proprietary VLMs on four common classification benchmarks and discovered that their performance significantly lags behind CLIP. This finding is counterintuitive because VLMs often use CLIP as their vision encoders. This finding reveals weakness in VLMs that previous works have not widely noted 12 49]. Understanding the limitations of VLMs is crucial given their increasing deployment in various scenarios.
Thorough Evaluation to Identify the Problem:
Given the poor performance of VLMs in classification, we investigated the underlying reasons. We considered multiple plausible hypotheses related to VLM inference, training, and data perspectives. For example, essential information for classification could be lost during the vision encoder's propagation through multiple LLM layers; VLMS might inherently poor at classification due their text generation training objective compared to the standard cross-entropy objective. Our thorough investigation ruled
Hypothesis Testing to Understand the Problem:
23

---

out these reasons but revealed the major issue: the lack of alignment data. This finding is counterintuitive because other alternative hypotheses also seemed plausible.
Improving VLMs based Our Understanding: Based to enhance VLM performance. We believe that classification is foundational to more advanced capabilities. For example, if a VLM cannot accurately classify mushroom species, it will also struggle with follow-up questions. such as whether particular mushroom is poisonous. Indeed, found that simply adding classification data not only improves VLMs' classification performance but also their general capabilities, demonstrating that accurately identifying objects is a prerequisite for answering complex questions about these objects.
findings, explored ways
Impact: Recent work from Google DeepMind, PaliGemma 6 supports our main conclusion that
data is the critical factor improving VLM performance. They also found that most VLM tasks benefit significantly from longer pre-training with more data (Appendix K). This demonstrates that our analysis can inspire researchers build better VLMs the future.
24

---

Image
Question
Choices
Reference Wikipedia
Which position is par-
A. Fowler's position
The feet can be raised to what is called the Trendelen- burg position, indicated for patients in shock.
ticularly important for
B. Upright position
this object when trans-
C. Trendelenhurg position
porting a patient in
D. Flat position
shock?
Which characteristic is NOT typical for the temperament of this
A. Very intelligent and loyal
Giant Schnauzers are usually quiet breed... It has the po- tential to be aggressive, but Giant Schnauzers usually reserved
B. Easily bored and highly energetic
C. Quiet and suspicious of strangers
ohject?
D. Highly aggressive and loud
What is the traditional
A. Black with white markings
The breed's coat only comes in single colour combina- tion of white with red mark- ings, usually in piebald pat- tern.
color combination of this object's coat?
B. Solid red
C. Red with white markings
D. White with black markings
What steering mech-
A. A rudder
foue-whe vehicle is also steered by the shafts or pole, which are attached to the front axle; this swivels 'turntable" 'fifth wheel' beneath the vehicle.
anism is traditionally
B. Turntable or fifth wheel
associated with this object when pos- sesses four wheels?
C. Flap control
D. A steering wheel
What is notable be- havior of this object during mating?
A. Males compete in physical com-
The female releases pheromone which causes the males to become less aggressive and to begin courtship.
B. Releases a sound to attract mates
C. The female releases pheromone
to reduce aggression in males
D. Both
change color
brighter
Which of the follow-
A. The LEGO Store
Notable Examples. Ham- leys, the world's largest toy shop
ing is known the
B. Hamleys
world's largest this ob- ject?
C. Toys "R" Us
D. FAO Schwarz
In which habitats is this object predomi- nantly found?
A. Low-altitude woodlands and for-
They are principally birds of low-altitude woodlands and forests, and particularly of forest edge and canopy.
est edges
Polar and subpolar zones
C. Temperate forests and grasslands
D. Desert and arid regions
In which type of cli- mates does this object
A. Mediterranean
The plant tolerates seasonal drought, and the Middle East- em and Mediterranean cli- mates are especially suitable to it.
B. Subarctic
predominantly thrive?
C. Tropical rainforest
D. Desert
Which of the follow-
A. Eight-to Eight-to-fourted fourttee modulation
Philips also contributed tight-to-fourteen modulation (EFM), which offers a certain resilience to defects such as scratches and fingerprints.
ing features intro-
(EFM)
duced by Philips the development object?
Diagonal error correction
C. Anti-shock buffering
D. Cross-interleaved Reed-Solomon
Coding (CIRC)
What common
A. Display in museums
Messenger hag: one long strap worn across the hody, inspired by bags worn by urban messengers deliver business mail, a modern houette
contemporary use for this object in urban set- tings?
B. Storing chilled goods for retail
C. Transporting large amounts of heavy equipment
D. Delivering business mail across
the city
Table 14: Examples of Image WikiQA. Each example has image, question, four choices (correct choice highlighted orange), and reference Wikipedia sentences.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims supported in and
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed $
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included all the details in each section and released all the codes and data in httpt::// github.......... com/ con//yhui-- 2h15/VMMCaassifier
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We included all the details in each section and released all the codes and data in https: / :ttp:::////ttttttt github. com com/yuhui- 2h15/VLMClassifier
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We included all the details in each section and released all the codes and data in //ttps://gitiiiii //github/bbb com/yuhut M
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The results are statistically significant, so no error bar is reported.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Discussed in 6
26

---

9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pww.aaaaooo.....c/aaaaaaa
Question:
Answer: [Yes]
Justification: We have carefully reviewed the NeurlPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussed 6
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Question: Are creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the assets (data and models) properly cited in the paper.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We included all the details in S and released all the codes and data in htttp:///github. com/yuhui-20 h VLClassifie
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
27


Reviews:
[{"review_id": "73w3PmHvpg", "reviewer": "Reviewer_fPsb", "summary": "In this paper, the authors explored why Vision-Language Models (VLMs) significantly underperform as image classifiers. They compared several publicly available with proprietary VLMs on several classification benchmark datasets, including ImageNet, Flowers102, StanfordCars, Caltech101, and their newly collected ImageWikiQA dataset. The paper presented multiple hypotheses regarding the underperformance of VLMs, addressing questions related to inference, training objectives, and training/finetuning data. The authors conducted empirical studies and concluded that the performance of VLMs in classification is determined by the data used. They fine-tuned LLaVa1.5-7B on ImageNet and the LLaVa instruction-tuning dataset to enhance classification accuracy.", "questions": "Please respond to the points of weakness I mentioned above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-organized and easy to follow.\n2. The motivation is clear, and the empirical study is thorough.\n3. The authors analyzed the training data of LLaVA1.5 models to show the strong correlation between class frequency over accuracy.", "weaknesses": "1. The authors did not provide much analysis on the proprietary VLMs. Can the authors address the hypotheses related to inference, training objectives, and data for these VLMs?\n2. The authors introduced “open-world setting” and “closed-wold setting” as evaluation protocols in section 2.3. However, the protocol seems to be only applied to Table 1. It is not clear which settings are being used for sections 3 and 4.\n3. The authors need to provide more explanation for Table 2. What is the difference between the “Closed-World Setting” performance of Table 1 and the “Base Prompt w/ Label (Fixed Order)” in Table 2? Why are their accuracies different?\n4. The LLaVA-7B and BLIP2-7B models are not large enough to demonstrate intrinsic capabilities such as chain-of-thought reasoning [1]. The authors could include the prompt variation results of Table 2 for one of the proprietary VLMs to show the impact on performance.\n5. In Lines 178-179, the authors observed that the information necessary for classification is preserved in VLMs' latent space; however, it cannot be effectively decoded. If the VLMs have the essential information, then why are they not able to decode it? The authors should explain the possible reasons for this. \n6. In Table 4, the finetuned VLM performance is worse than the zero-shot performance of GPT4  on ImageWikiQA. What are the possible reasons for this? Why is the “Finetuned on ImageNet” model performance (30.6) worse than the non-finetuned model performance (37.8)? Does finetuning always enhance the general capabilities of the VLMs?\n\n[1] Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. \"Multimodal chain-of-thought reasoning in language models.\" arXiv preprint arXiv:2302.00923 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer fPsb for their positive comments and thoughtful feedback. We address Reviewer fPsb’s questions below.\n\n---\n\n**Analysis of proprietary VLMs**\n\n> Can the authors address the hypotheses related to inference, training objectives, and data for proprietary VLMs?\n\nThank you for your question. **Proprietary VLMs haven’t released the training details and data usage, so it is impossible to directly address training-related or data-related hypotheses.** For inference-related hypotheses, we conduct additional experiments with GPT4 (**see PDF in general response**), and the conclusion is the same as the public VLMs.\n\n**Recent work from DeepMind** [1] shows that most VLM tasks benefit significantly from longer pre-training with more data (in Figure 4 and Appendix K of [1]). This **aligns with our main conclusion that data is the critical factor in improving VLM performance.** \n\n[1] PaliGemma: A versatile 3B VLM for transfer.\n\n---\n\n**Open-world vs closed-world setting**\n\n> It is not clear “open-world” vs “closed-world” settings are being used for sections 3 and 4.\n\nGreat question! **Sec. 3 and 4 all use “open-world” settings (not providing the label set in the prompt), except for prompt variation analysis and label set size analysis in Sec. 3.1, using “closed-world” settings (providing the label set in the prompt).**\n\nNaturally, the “closed-world” setting narrows the model’s generation space and increases accuracy. When the model can easily predict the class in the label set by modifying the inference space (probabilistic inference in Sec. 3.1 and probing VLM in Sec. 3.2) or training VLMs with more classification data (Sec. 4), **the advantage of the “closed-world” setting doesn’t exist**. Thus, for these experiments, we use the “open-world” setting.\n\nWe will clarify this in the revised paper.\n\n---\n\n**Clarification for Table 2**\n\n> What is the difference between the “Closed-World Setting” performance of Table 1 and the “Base Prompt w/ Label (Fixed Order)” in Table 2? Why are they different?\n\n**Table 1’s “Closed-World Setting” is equivalent to Table 2’s “Base Prompt w/ Label (Random Order)”.** In this setting, we concatenate label candidates in a random order for each question: “What’s in the image? Choose one from random_shuffle([cat, dog, pig])”. The accuracy for these two settings is identical between Table 1 and Table 2.\n\nTable 2’s “Base Prompt w/ Label (Fixed Order)” concatenates label candidates in a fixed order for each question: “What’s in the image? Choose one from [cat, dog, pig]”. This setting is used to rule out the possibility that the order of labels might affect model performance.\n\n---\n\n**CoT with larger VLMs**\n\n> The LLaVA-7B and BLIP2-7B models are not large enough to demonstrate intrinsic capabilities such as chain-of-thought reasoning [1].\n\nThank you for your suggestion! We have now evaluated GPT-4’s performance on ImageNet using chain-of-thought (CoT) reasoning. \n\nIn Table 2, GPT-4 achieves 60.6% accuracy on ImageNet without CoT. **With CoT, GPT-4 achieves 62.2% accuracy**, demonstrating that CoT has a limited impact on image classification, even for large VLMs. \n\nWe will add these results and citations in the revised paper.\n\n---\n\n**Cannot decode information**\n\n> If the VLMs have the essential information, then why are they not able to decode it?\n\n\nGreat question! **One possible reason is that the VLM’s decoding space is too large and not aligned with the visual features.** Specifically, VLM decoding is performed through next-word prediction, which usually involves a vocabulary of over 10,000 words. The output text embedding is not aligned with the visual features due to insufficient data to align these spaces.\n\n**In general, having information in a model does not necessarily mean the model can express that information.** This phenomenon is also observed in other research areas. For example, after training ResNets or ViTs with self-supervised learning methods like SimCLR, the model acquires discriminative information for different classes. However, this information can only be decoded by adding and training a linear layer on a new dataset.\n\nSimilarly, we demonstrate that VLMs possess classification information, but it is not readily expressible. **Adding classification-related data helps bridge the gap between “information possession” and “information expression”** (refer to Table 3 in our paper).\n\nWe will include this discussion in the revised paper.\n\n---\n\n**Fine-tuning results**\n\n> In Table 4, why is the “Finetuned on ImageNet” model performance (30.6) worse than the non-finetuned model performance (37.8 [should be 38.0])? Does finetuning always enhance the general capabilities of the VLMs?\n\n\n**Fine-tuning often improves in-distribution performance but does not necessarily enhance out-of-distribution or general capabilities.**\n\n**Table 4 presents results of models fine-tuned on ImageNet and evaluated on ImageWikiQA, which are vastly different datasets.** ImageNet questions only require classification, such as “Q:  <image> What is in the image? A: dog,” while ImageWikiQA questions demand both classification and knowledge/reasoning, such as “Q:  <image> What is the native region of this object? A: South America.”\n\nFine-tuning solely on ImageNet trains the model to classify, but it can lead to a loss of general capabilities like reasoning and knowledge, resulting in lower performance on ImageWikiQA (30.6 fine-tuned vs. 38.0 pre-trained). This phenomenon is known as “catastrophic forgetting” [1].\n\n**However, when we fine-tune on both ImageNet and instruction tuning data, the model learns classification while retaining its original capabilities, leading to a significant performance improvement on ImageWikiQA (49.8 fine-tuned vs. 38.0 pre-trained).**\n\nWe discussed this in Lines 281-285 and will add further clarification.\n\n[1] Overcoming catastrophic forgetting in neural networks.\n\n---\n\nThank you again for your feedback. Please let us know if you have further questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BfvWJ6kS2z", "reviewer": "Reviewer_SHnR", "summary": "The paper presents an interesting observation of VLMs lagging in image classification performance as compared to the visual encoders lie CLIP used within them. Several hypothesis are explored to explain this observation including train-time (information loss, training objective used), inference-time (prompt variations, label set size) and data related reasons. Their analysis shows the primary reason for the observed gap to be data prevalences, showing a correlation between class prevalence during VLM training and performance in those classes for image classification. The paper then simply proposes inclusion of these datasets into VLM training as a way to fix this issue, showing image classification improvements.", "questions": "1. In Figure 2, the label set size does seem to have some effect on the performance gap b/w CLIPs and VLMs. Since this relates to the way VLMs are evaluated for image classification (attaching class options as tokens in the prompt which grow with the label set), do you think this experiment shows the evaluation difference between the two methods (and also how they are originally trained), can explain some of the gap? Also, why do you think the Caltech and ImageNet datasets behave a little differently than Cars and Flowers in Figure 2?\n2. Why do you think CLIP does not suffer from the class prevalence bias but LLava does in Figure 3? What happens if we plot this for linear probed LLava models? What happens if we plot this for linear probed Llava models which are fine-tuned on similarly biased data having similarly skewed frequencies? If those models do not show this strong correlation, it might be more about the fine-tuning and less about the data distribution.", "limitations": "The paper has discussed limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is nicely structured. An interesting performance trend is observed, hypothesis are explored to explain the phenomena, and conclusions drawn from the experiments are further tested. The evaluation and experimental details, the different hypothesis spanning both training and inference, and the well crafted control experiments make it a good study.\n2. The paper is also well motivated and studies a relevant failure mode of VLMs. The results show that even though these modes might encode object or image level concepts, they still suffer from an inability to classify images which is supposed to be a more simpler, and more importantly a more fundamental visual task. This makes it a valuable research problem.\n3. The specific experiments to explore each hypothesis are interesting in their own right. For example studying the effects of prompt ordering or CoT to image classification or different variations of inference or label set size, type of objective used, and linear probing results, all provide useful cues towards the hypothesis, but also showcase interesting VLM behavior.\n4. The paper proposes a very simple and effective way to bridge the VLM performance gap by fine-tuning on the classification data. It shows how doing so leads to not just regaining high image classification performance, but also improves VLM-specific tasks on the classification dataset, for which they curate a new dataset.", "weaknesses": "1. I'm not completely convinced with the paper's final conclusion of data being the reason why CLIP models are superior to VLMs. The fact that linear probes can extract good classification performance from VLMs shows that its a decoding problem since even these models were trained with the same data having skewed prevalences for certain badly performing classes, but the linear probe tuning manages to bring that out to the same degree as CLIP. This implies that either what's missing is a task-aligned fine-tuning (which is what a linear probe does, and which is what the paper does when they fine-tune) or perhaps even a modification of how these are inferred on for image classification which can lead to a more a task-aligned inference (CLIP is trained for text/concept alignment and is evaluated using knn whereas VLMs are trained for vision-text conditioned text generation, but evaluated in a very specific way by attaching tokens of the different options for image classification). In either case, the data argument for explaining the gap might not be the only, or even the strongest reason.\n2. The paper proposes inclusion of the image classification datasets as a solution for this problem. If this VLM limitation stems for other reasons such as multimodal confusion and text feature interference or hallucination problem specific to the generated text space, adding more data for different tasks as a solution might not generalize to similar VML issues. In this case, since the new dataset involves tasks which necessitate good image classification, performance after fine-tuning goes up on both sets of tasks. This might not hold true for other equally fundamental visual tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer SHnR for their positive comments and thoughtful feedback. We address Reviewer SHnR’s questions below.\n\n---\n\n**Main reason and solution**\n\n> I'm not completely convinced with the paper's final conclusion of data being the reason why CLIP models are superior to VLMs… The data argument for explaining the gap might not be the only, or even the strongest reason.\n> \n\nWe clarify that data is not the **primary reason**. The primary reason is that the information required for classification is encoded in the VLM’s latent space but cannot be effectively decoded. Data serves as an **effective solution** to decode this information, as our experiments show that with sufficient data, VLMs can match the accuracy of state-of-the-art classification models.\n\n> This implies that either what's missing is a task-aligned fine-tuning or perhaps even a modification of how these are inferred on for image classification which can lead to a more a task-aligned inference.\n\nThank you for the thought-provoking question. We agree that task-aligned fine-tuning or inference, such as adding a linear probing loss on the VLM output space for training and using KNN in the VLM output space for evaluation, can improve VLM classification performance.\n\n**However, our focus is on enhancing VLM’s general capabilities in solving a variety of tasks rather than the specific classification task.** The only natural inference interface for different tasks is text generation. Task-aligned inference, like KNN, is only applicable to classification and not other tasks.\n\nFor task-aligned fine-tuning, such as adding a new token and a linear probing loss, we observed no improvement when evaluating this fine-tuned VLM on ImageWikiQA. This shows that task-aligned fine-tuning does not improve VLM’s overall capabilities.\n\nIn summary, without changing VLM training or inference to maintain its general capabilities for different tasks, adding data is the most promising and effective approach. We will add these clarifications in the revised paper.\n\n---\n\n**Solution to other problems**\n\n> If this VLM limitation stems for other reasons such as multimodal confusion and text feature interference or hallucination problem specific to the generated text space, adding more data for different tasks as a solution might not generalize to similar VML issues.\n\nThank you for your question. We agree that for issues like multimodal confusion or hallucination, adding data might not resolve the problem, but **addressing these is beyond the scope of our paper.** \n\nThe main contribution of our paper is the formulation of the problem that VLMs are poor image classifiers and a thorough investigation into why this is the case and how to solve it. Our paper demonstrates that the poor classification performance of VLMs is due to the inability to decode the information encoded in VLMs. **Given this reason, data serves as an effective solution to decode the information.** \n\n---\n\n**Label set size analysis**\n\n> Do you think the label set size experiment shows the evaluation difference between CLIPs and VLMs can explain some of the gap?\n\nThank you for your question. **The label set size experiment can partially explain the gap but not entirely.**\n\nBy reducing the number of labels for classification, we can narrow the gap between VLM and CLIP, but a gap remains across all label set sizes, even with just two labels (two-way classification).\n\nMoreover, we find that while the absolute gap between VLMs and CLIPs narrows with reduced label size, the relative gap increases (Appendix Figure 4). For example, in two-way classification on ImageNet, VLMs have a 5.7% error rate, while CLIP has a 0.2% error rate, resulting in a 28.5x gap; for 20 classes, VLMs have an 18.0% error rate, while CLIP has a 2.5% error rate, resulting in a 7.2x gap.\n\n**These results indicate that the VLM-CLIP gap cannot be fully explained by label set size.**\n\n> Why do you think the Caltech and ImageNet datasets behave a little differently than Cars and Flowers in Figure 2?\n\nThe absolute gap on Flowers and Cars is larger than ImageNet and Caltech when reducing the label set size. **This may be because Flowers and Cars are more fine-grained classification datasets, while ImageNet and Caltech are more coarse-grained**. VLMs are weaker in fine-grained classification compared to CLIP, resulting in a larger gap.\n\nWe will add these clarifications in the revised paper.\n\n---\n\n**Class prevalence bias**\n\n> Why do you think CLIP does not suffer from the class prevalence bias but LLava does in Figure 3?\n\nGreat question! **In Figure 3, the class frequency on the x-axis is computed based on the LLaVA pre-training and instruction-tuning dataset, not the CLIP training dataset.** If we change the x-axis to the CLIP pre-training data frequency, CLIP should also show prevalence bias. Please refer to Figure 2 in [1].\n\n[1] No “Zero-Shot” Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance.\n\n> What happens if we plot this for linear probed LLava models?\n\n**We plotted the line for the linear probed LLaVA model, which aligns with the fine-tuned LLaVA (see PDF in general response).** The zero correlation is expected because fine-tuning or probing on the ImageNet dataset equalizes class frequencies, as ImageNet classes are evenly distributed. We will include this line in the updated paper.\n\n> What happens if we plot this for linear probed Llava models which are fine-tuned on similarly biased data having similarly skewed frequencies?\n\nWe have not yet trained a new model on skewed data, but **we found that training on the ImageNet classification dataset does not improve performance on the Flowers dataset, which has drastically different class frequencies.** This further verifies that data, rather than fine-tuning, is the key factor determining performance. We will add this in the revised paper.\n\n---\n\nThank you again for your feedback. Please let us know if you have further questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gyvaK5YBTZ", "reviewer": "Reviewer_8fB5", "summary": "This paper analyzes the issue of large vision language models (VLMs) that perform poorly in common image classification datasets such as ImageNet. The authors analyze different perspectives on the problem, including trying different inference and training methods. For inference, the authors tried using different prompt variations, shrinking the number of classes, and computing conditional probabilities of class names. All these methods still leads to performance gaps between VLMs and CLIP. Furthermore, the authors confirm that sufficient class information is encoded in the output features of the visual encoder in VLMs, via linear probing. The authors also discover VLMs can be trained to generate class labels with a comparable accuracy as their visual encoders. Finally, they analyze the frequency of different classes in the original datasets that were used to train the VLMs, and find that the frequency of samples for a class is positively correlated to the accuracy of the VLMs on that class. This reveals that the poor performance of VLMs in image classification is due to the lack of class labels in their instruction-tuning data. \n\nThey also construct a new question-answering dataset, ImageWikiQA, to test the model ability of answering questions related to the finegrained image class. The authors then finetune VLMs using a dataset that consists of both the image classification data and the original nstruction-tuning data. The resulting VLMs can perform well on the ImageWikiQA dataset.", "questions": "- Can the authors confirm the results in Table 3 (the right sub-table) are the accuracy on the validation/test sets? The accuracy on ImageNet looks very high to me. I have the same question for the accuracy mentioned in line 279. \n- In Table 4, LLaVA1.5-7B with GT Class provide still has a relative low accuracy. Why is that? \n- Can the authors confirm there are no training data in ImageWikiQA (i.e., it is only a testing dataset)?", "limitations": "Please see the Weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper provides a relatively extensive analysis of why SOTA public VLMs perform poorly in image classification. The results generally look correct but there are some issues (as detailed in Weaknesses and Questions) that make them less sound. \n- The authors create a new dataset, ImageWikiQA, to evaluate the question-answering capacity of VLMs related to the fine-grained classes in ImageNet. \n- The authors show that incorporating the ImageNet classification data into VLM's instruction-tuning data can improve the model performance on ImageWikiQA.", "weaknesses": "- It seems that the authors did not consider there may be multiple textual labels for many ImageNet classes. For example, for class with ID n01496331, the class name can be electric ray, crampfish, numbfish, or torpedo. When calculating VLM classification accuracy, the authors can match the model-generated text with any of these class names. \n- It may harm the VLM's instruction-following capacity to incorporate the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data to train the VLM, because the classification data has a single fixed template \"What type of object is in this photo? ASSISTANT: <Class Name>.\" Did the authors perform any evaluation of the VLM's instruction-following capacity? \n- It is expected that the information necessary for classification is largely preserved in the visual encoders in the VLMs, as these visual encoders are typically pre-trained on ImageNet and are kept frozen during the integration into VLMs. If the instruction-tuning data for VLMs do not contain class-specific information, it is unlikely that the VLMs can automatically align the class information in the visual encoder's output to the text generation. This explains why inference methods do not work in the paper. \n- Section 4.2 is a bit unclear. For example, it mentions that the authors randomly sampled at most 3 ImageNet images for each question, but then it mentions there are 2000 multiple-choice questions, each with an image. So how many images are there per question?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 8fB5 for providing detailed and thoughtful feedback on our work. We address Reviewer 8fB5’s questions below.\n\n---\n\n**Multiple textual labels**\n\n> There may be multiple textual labels for many ImageNet classes. For example, n01496331 can be electric ray, crampfish...\n\nThank you for your suggestion. **We have now considered ImageNet label synonyms [1] in our evaluation process.** When evaluating with synonyms, **the accuracy only improves by 1%-3%**, which still results in a significant performance gap compared to CLIP.\n\nWe will include these results in the revised paper.\n\n[1] https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a\n\n| Model | Accuracy w/o Synonyms | Accuracy w/ Synonyms | Delta |\n| - | - | -| - |\n| BLIP2-2.7B | 25.3 | 27.8 | 2.5 |\n| IBLIP-7B | 14.6 | 16.5 | 1.9 |\n| IBLIP-13B | 14.7 | 16.6 | 1.9 |\n| LLaVA1.5-7B | 22.8 | 24.6 | 1.8 |\n| LLaVANeXT-V7B | 29.4 | 32.2 | 2.8 |\n| LLaVA1.5-13B | 24.3 | 26.0 | 1.7 |\n| LLaVANeXT-M7B | 32.3 | 35.1 | 2.8 |\n| Claude3 | 53.6 | 56.3 | 2.7 |\n| GeminiPro | 39.2 | 42.5 | 3.3 |\n| GPT4 | 48.5 | 51.1 | 2.6 |\n\n---\n\n**Instruction-following capacity**\n\n> It may harm the VLM's instruction-following capacity to incorporate the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data.\n\nGreat question! **We have now evaluated the original VLM and our fine-tuned VLM on three additional instruction-following benchmarks:** TextVQA, POPE, and MMVet. We found that **fine-tuned VLM achieves the same accuracy as without fine-tuning**. \n\nWe will release all the codes and models to reproduce these results and include them in the revised paper.\n\n|  | TextVQA | POPE Popular | POPE Adverse | MMVet |\n| --- | --- | --- | --- | --- |\n| LLaVA1.5-7B (Official Released) | 58.2 | 86.1 | 84.2 | 31.1 |\n| LLaVA1.5-7B (Finetuned) | 58.0 | 86.3 | 84.5 | 31.1 |\n\n---\n\n**Expected conclusion** \n\n> It is expected that the information necessary for classification is largely preserved in the visual encoders in the VLMs.\n\nWe agree that it is apparent that the information necessary for classification is preserved in visual encoders. However, **it is not unclear whether the information still remains after propagating through all the LLM layers** (e.g., 32 layers for Vicuna-7B). Our results highlight that information is preserved after the LLM propagation rather than preserved in visual encoders. \n\n> If the instruction-tuning data for VLMs do not contain class-specific information, it is unlikely that the VLMs can automatically align the class information in the visual encoder's output to the text generation\n\n**This claim is actually unknown and controversial.** The key question is: if both vision encoders and language models have seen a specific class during their uni-modal training, do VLMs need to see the exact class in a multi-modal format to align them during multi-modal training, and how much data is required? **Many previous works have shown that this alignment stage is very data-efficient and even unnecessary [1, 2].**\n\n**Our paper demonstrated that 1) multi-modal data is necessary for alignment, and 2) increasing the data amount leads to linearly improved performance.** This provides a critical data-centric view for VLM training. **Recent work from DeepMind echoes our findings.** They found that most VLM tasks benefit significantly from longer pre-training with more data, as shown in Figure 4 and Appendix K [3].\n\nWe will add these clarifications in the revised paper.\n\n[1] Visual Instruction Tuning\n\n[2] Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models\n\n[3] PaliGemma: A versatile 3B VLM for transfer\n\n---\n\n**ImageWikiQA clarification**\n\n> It mentions that the authors randomly sampled at most 3 images for each question, but then it mentions each with an image. How many images are there per question?\n\n**There is only one image per question** (see Appendix Table 11 for examples).\n\nThe confusion arises from our two-stage creation pipeline:\n\n- **1st stage: generate questions at the class level.** For any given class out of the 1000 ImageNet classes, we create questions such as “What is the native region of <class name>?”.\n- **2nd stage: generate questions at the image level.** Each ImageNet class has 50 images, from which we randomly sample 3 images and create 3 questions like “What is the native region of <image i>?”\n\nWe will revise the text for clarity.\n\n---\n\n**Data split clarification**\n\n> Are the results in Table 3 and line 279 are the accuracy on the validation/test sets? The accuracy on ImageNet looks very high to me.\n\n**Both Table 3 and Line 279 are evaluated on the validation set.** The data splits use the official data split (e.g., ImageNet contains 1.28M training images and 50K validation images). Detailed data splits are provided in Appendix Table 6.\n\n**The very high accuracy on ImageNet is one of the significant contributions of our paper.** We found that, after fine-tuning the VLM on ImageNet, there is no more gap between VLM and CLIP, with VLM now being the state-of-the-art classifier.\n\n---\n\n**Low accuracy of LLaVA1.5-7B with GT class**\n\n> In Table 4, LLaVA1.5-7B with GT Class still has a relative low accuracy. Why?\n\nTable 4 reports the accuracy on ImageWikiQA. **These questions come from Wikipedia and require extensive knowledge, which is very challenging even for humans** (e.g., what is the native region of the guinea pig?)\n\n**The low accuracy (55.9%) is because LLaVA1.5-7B lacks sufficient world knowledge for these ImageNet classes.** It is well known that smaller LMs like Vicuna-7B lack world knowledge compared to larger LMs like GPT4.\n\n---\n\n**ImageWikiQA test only**\n\n> No training data in ImageWikiQA?\n\n**Yes, ImageWikiQA only contains a test set.** This dataset serves as an important resource to evaluate VLM’s fine-grained classification capabilities as well as its knowledge and reasoning abilities.\n\n---\n\nThank you again for your feedback. Please let us know if you have further questions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7prDVf6dZz", "reviewer": "Reviewer_CuMn", "summary": "This article notes that current top-performing VLMs, like GPT-4V and LLaVA, are unable to perform image classification tasks like the CLIP model. Despite having a larger number of parameters and incorporating a vision encoder from a pre-trained CLIP model.\n\nThis author thinks the main reason for this underperformance is data-related. Critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data.\n\nThe author proposes to incorporate classification-specific datasets (such as ImageNet) into VLM training; its performance in classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.", "questions": "I would like to know how well the fine-tuned VLMs perform on the original benchmarks such as MME, TextVQA, and others.\nCan we achieve similar results if a classification dataset is included in the second stage of training with LLaVA?", "limitations": "Yes, the author has addressed relevant limitations and discussed the broader impacts adequately.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The article clearly points out the performance issues of the current Visual Language Model (VLM) in image classification tasks. It systematically analyzes the possible reasons, filling the research gap in this field. This author thinks the main reason for this underperformance is data-related. By incorporating classification-specific datasets (such as ImageNet) into VLM training, the performance of classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.", "weaknesses": "The novelty is limited.\nThis article assesses how well current VLMs perform on classification datasets and fine-tunes them using the same datasets.\nThese contributions are insufficient to warrant publication of this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer CuMn for providing thoughtful feedback on our work. We address Reviewer CuMn’s questions below.\n\n---\n\n**Limited novelty**\n\n> The novelty is limited. This article assesses how well current VLMs perform on classification datasets and fine-tunes them using the same datasets. These contributions are insufficient to warrant publication of this paper.\n> \n\nWe want to clarify that our paper is not a method paper proposing new techniques to achieve better performance on standard benchmark leaderboards. Instead, it is an analysis paper.\n\nObject recognition is fundamental to the general reasoning capability of VLMs, yet current VLMs perform poorly in this area. We are the first to thoroughly investigate this critical issue and propose a potential solution to improve it.\n\n**Our primary contribution lies in identifying the problem (i.e., VLMs are inadequate image classifiers, a significant weakness that has been overlooked) and conducting an in-depth investigation (i.e., understanding why VLMs are poor image classifiers and how to address this issue).** This contribution goes far beyond merely assessing current VLMs on classification datasets and fine-tuning them using the same datasets.\n\nIn summary, our contributions are threefold:\n\n1. **Thorough Evaluation to Identify the Problem:** We thoroughly evaluated current public and proprietary VLMs on four common classification benchmarks and discovered that their performance significantly lags behind CLIP. This finding is counterintuitive because VLMs often use CLIP as their vision encoders. **This finding reveals a weakness in VLMs that previous works have not noted. Understanding the limitations of VLMs is crucial given their increasing deployment in various scenarios.**\n2. **Hypothesis Testing to Understand the Problem:** Given the poor performance of VLMs in classification, we investigated the underlying reasons. We considered multiple plausible hypotheses related to VLM inference, training, and data perspectives. For example, **essential information for classification could be lost during the vision encoder’s propagation through multiple LLM layers; VLMs might be inherently poor at classification due to their text generation training objective compared to the standard cross-entropy objective.** Our thorough investigation ruled out these reasons but revealed the major issue: the lack of alignment data. **This finding is counterintuitive because other alternative hypotheses also seemed plausible.**\n3. **Improving VLMs based on Our Understanding:** Based on our findings, we explored ways to enhance VLM performance. We believe that classification is foundational to more advanced capabilities. For example, if a VLM cannot accurately classify mushroom species, it will also struggle with follow-up questions, such as whether a particular mushroom is poisonous. Indeed, **we found that simply adding classification data not only improves VLMs’ classification performance but also their general capabilities, demonstrating that accurately identifying objects is a prerequisite for answering complex questions about these objects.** \n\n**Impact:** recent work from Google DeepMind, PaliGemma [1], supports our main conclusion that data is the critical factor in improving VLM performance. They also found that most VLM tasks benefit significantly from longer pre-training with more data (Figure 4, Appendix K). **This demonstrates that our analysis can inspire researchers to build better VLMs in the future.**\n\n[1] PaliGemma: A versatile 3B VLM for transfer.\n\n---\n\n**Fine-tuned VLM performance on other benchmarks**\n\n> I would like to know how well the fine-tuned VLMs perform on the original benchmarks such as MME, TextVQA, and others. Can we achieve similar results if a classification dataset is included in the second stage of training with LLaVA?\n> \n\nThank you for your question. **Following your suggestion, we evaluated both the original VLM and our fine-tuned VLM on three additional benchmarks and found that the VLM’s performance remain the same.**\n\nThe table below shows the performance of the original LLaVA1.5-7B and the fine-tuned LLaVA1.5-7B on TextVQA, POPE, and MMVet. After fine-tuning, the LLaVA1.5-7B achieves the same accuracy as without fine-tuning. This result is intuitive because we included all the original LLaVA training data during our instruction tuning.\n\nWe did not include MME because the dataset is not publicly available, and our access request has not yet been approved.\n\nWe will release all the codes and model checkpoints to reproduce these results and include them in the revised paper.\n\n|  | TextVQA | POPE Popular | POPE Adverse | MMVet |\n| --- | --- | --- | --- | --- |\n| LLaVA1.5-7B (Official Released) | 58.2 | 86.1 | 84.2 | 31.1 |\n| LLaVA1.5-7B (Further Finetuned) | 58.0 | 86.3 | 84.5 | 31.1 |\n\n---\n\nThank you again for your feedback. Please let us know if you have further questions or concerns!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback on our manuscript. Below, we provide individual responses to each reviewer. Please let us know if you have any further questions or concerns!

**We have also attached a PDF for reviewers fPsb and SHnR, which includes visualizations to further address their questions.**

Meta-review: The work reveals a problem that vision language models (e.g. GPT-4o) including proprietary and open-source ones perform poorly in image classification (e.g. ImageNet) compared to CLIP while they are both trained on the Internet data. The question is at which layer do VLMs contain necessary discriminative features? And they find the answer is the last two layers of VLMs i.e., finetuning a linear prob on VLMs improve the classification accuracy. The experiments are quite extensive: 7 open-source VLMs, 3, proprietary VLMs, 4 different datasets. 

Initially, the reviews were mixed (7, 5, 5, 3). After the rebuttal, no reviewers express any concerns. The AC agrees that the authors have addressed most of the concerns raised and recommends `Accept`.
Decision: Accept (poster)