Title: Why are Visually-Grounded Language Models Bad at Image Classification?
Venue: NeurIPS 2024
Accepted: yes
Keywords: vision-language models

image classification
Abstract: Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.

Why are Visually-Grounded Language Models Bad at Image Classification?

Abstract

Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visuallygrounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is datarelated: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset. 1

1 Introduction

The ability to recognize objects within images is a fundamental capability of machine vision. Over the past 15 years, the field has experienced significant breakthroughs due to deep learning and large-scale datasets [11, 48]. For instance, on the renowned ImageNet dataset, designed to classify images into 1,000 categories, the error rate has dramatically decreased from 47.1% in 2009 to 9.1% in 2024, representing a 5-fold reduction [30, 12]. Consequently, these classification models have superseded most human labelers.

Nowadays, the community has focused on more sophisticated and nuanced capabilities in the quest for visual intelligence. Visually-grounded language models (VLMs), which integrate visual signals from vision encoders with large language models, have recently emerged as a promising paradigm [2, 36, 32]. VLMs like GPT-4V [36], Gemini-1.5 [44], or Claude-3 [3] have demonstrated advanced visual understanding abilities, such as answering math questions from table images or generating HTML code from design sketches.

In this work, we revisit the fundamental task of image classification using VLMs. Surprisingly, we find that various public and proprietary VLMs struggle with image classification in both open-world settings, where the class list is unknown, and closed-world settings, where class names are provided in the context (§2). Despite having many more parameters, there is a significant gap between the

VLMs

a

re b

a

d im

a

ge cl

a

f

iers

Why

a

re VLMs b

a

d?

How to improve VLMs?

0

20

40

60

80

BLIP LL

a

VAGemini GPTCl

a

udeCLIP

Wh

a

t is in the im

a

ge?

Answer: guine

a

pig

VLM

Public VLMs

Propriet

a

ry VLMs

Wh

a

t is in the im

a

ge?

Choose one from…

LM

①

guine

a

pig

②

③

④

⑤

Wh

a

t is the n

a

tive

region of this object?

Cl

a

f

ic

a

tion is found

a

tion

a

l!

LL

a

VA tr

a

ined on

origin

a

l

d

a

t

a

:

LL

a

VA tr

a

ined on

origin

a

l+cl

a

f

ic

a

tion

:

The im

a

ge shows

a

sm

a

ll r

a

bbit.

Inference?

Tr

a

ining?

①

Prompt V

a

ri

a

tion?

②

L

a

bel Sp

a

ce?

③

Inference Str

a

tegy?

④

Inform

a

tion Lost?

⑤

Tr

a

ining Objective?

D

a

t

a

?

⑥

Not Enough D

a

t

a

?

⑥

A. South Americ

a

a

a

a

a

The

a

nim

a

l is

a

guine

a

pig, which

is n

a

tive to South Americ

a

.

Therefore, the

a

nswer is A.

The im

a

ge shows

a

sm

a

ll r

a

bbit.

The n

a

tive region could v

a

ry

depending on the species.

CLIP

performance of VLMs and their commonly used vision encoder CLIP [37]. Our evaluation protocol involves feeding each image and a list of class names (in the closed-world setting) to the VLM as context and asking what is in the image; success is defined by whether the generated output contains the ground-truth class name.

To understand why VLMs underperform in classification settings, we investigate several hypotheses regarding VLMs' inference (such as prompt variations, label set size, inference strategy; §3.1), training (such as information lost, training objective; §3.2), and data (such as data-performance correlation; §3.3). Our extensive analyses suggest that the primary reason for the observed gap is data. We find that the information necessary for classification is encoded in the VLM's latent space but can only be decoded with proper training data. Specifically, there is a strong correlation between class presence during VLM training and performance in those classes. Furthermore, training VLMs on classification datasets achieves the same performance level as state-of-the-art classification models.

Motivated by our analysis, we propose a simple method to enhance VLMs' general capabilities by integrating traditional classification-focused datasets into VLM training (§4). We believe that classification is the foundation for more complex, advanced visual capabilities; for example, recognizing an object is a prerequisite for answering complex questions about it. To verify this, we created ImageWikiQA, which contains complex real-world questions about ImageNet objects. On ImageWikiQA, we find that VLMs fine-tuned on the ImageNet classification dataset achieve substantially higher accuracy in recognizing these objects and provide more accurate answers to these non-classification questions, outperforming pre-trained VLMs by 11.8%. This suggests that classical classification data can be beneficially reused in the VLM training process to enhance VLM performance.

In summary, our contributions are threefold:

Evaluating VLM classification weaknesses: Our evaluations using ten VLMs across four benchmarks show that VLMs significantly lag behind CLIP in classification, uncovering a gap unaddressed by previous research.

Analyzing causes of poor classification performance: Testing various hypotheses, we find that the lack of alignment data, rather than information lost or training objective, is the primary reason for VLMs' underperformance in classification.

Enhancing VLMs with classification data: By adding classification data, we improve VLMs' accuracy in object recognition and overall performance, supporting that classification capability is foundational for complex object-related reasoning.

2 VLMs are Bad at Image Classification

We begin by evaluating state-of-the-art visually-grounded language models (VLMs) using standard image classification benchmarks. Our findings reveal that these VLMs significantly underperform compared to state-of-the-art classification models, such as CLIP.

2.1 Models

VLMs. We selected ten widely-used state-of-the-art VLMs, covering different architectures, training methods, and data. These VLMs include three proprietary ones, GPT-4-Turbo (shortened as GPT4 , same below) [36], Gemini-Pro-Vision ( GeminiPro ) [44], and Claude-3-Opus ( Claude3 ) [3], and seven public ones, LLaVA1.5-Vicuna7B/13B ( LLaVA1.5-7/13B ) [32], LLaVANeXT-Mistral7B/Vicuna7B ( LLaVANeXT-M7B/V7B ) [31], BLIP2-OPT2.7B ( BLIP2-2.7B ) [27], and InstructBLIP-Vicuna7B/13B ( IBLIP-7/13B ) [10]). Details of these models are provided in Appendix §A.1.

CLIPs. For comparison, we used two state-of-the-art image classifiers, CLIP-ViT-L/14-336px (shortened as CLIP-L , same below) [37] and EVA-ViT-G/14 ( EVA-G ) [43]. Notably, CLIP-L and EVA-G are utilized by the LLaV A series [32] and the BLIP series as vision encoders [27], respectively. Therefore, the VLMs should theoretically have the same classification capacity as these vision models. Details are listed in Appendix §A.1.

2.2 Data

We evaluated the aforementioned models on four widely-used image classification benchmarks: ImageNet (shortened as , same below) [11], Flowers102 ( ) [35], StanfordCars ( ) [21], and Caltech101 ( ) [13], which contain 50,000, 6,149, 8,041, and 4,331 test images from 1,000, 102, 196, and 101 classes, respectively. ImageNet and Caltech cover more coarse-grained objects, while Flowers and Cars cover more fine-grained objects. Further details are provided in Appendix §A.2.

2.3 Evaluation Protocol

VLMs. We performed image classification in two settings: an open-world setting where the label set is not provided and a closed-world setting where classes are concatenated in the prompt. We

feed the image and the prompt to the VLM and let the VLM complete the rest of the tokens. One closed-world example is: '<image> What type of object is in this photo? Choose one from <class name A>, <class name B>, ...' We define success on a single example as whether the ground-truth label is included in the VLM generation. We report the success rate of all test examples.

CLIPs. CLIP can only be used in a closed-world setting where the label set is known. Following Radford et al. [37], we used the prompt 'a photo of a <class>' to generate the text feature. For each image, we selected the class with the highest cosine similarity to the image feature. We did not include prompt ensembling to fairly compare with VLM. We report the accuracy of all examples.

2.4 Results

Table 1 reports the performance of different VLMs and CLIP models on these classification datasets.

We find that VLMs exhibit poor performance in image classification, significantly lagging behind CLIP models . For instance, on the ImageNet dataset, the best proprietary VLM, GPT4 , only achieves an accuracy of 60.6% in the closed-world setting, whereas the best CLIP, EVA-G , attains an accuracy of 79.2%. In the open-world setting, the best public VLM, LLaVANeXT-M7B , achieves just 32.3% accuracy. The performance disparity is even more pronounced in fine-grained classification datasets like Flowers102 and StanfordCars. Notably, all LLaVA models use CLIP-L as the vision encoder, and although the total parameter count of these models is at least 20 times greater than that of the vision encoder, they significantly underperform compared to it.

Moreover, we find that closed-world setting often outperforms open-world setting . This is expected as the provided label set narrows the prediction space. However, since closed-world settings require including all class names in the context, they can result in an extremely long context that leads to high costs or even exceeds the VLM's context limit. For example, most public VLMs only support 4K context length, which cannot feed 1K ImageNet classes. We also find that larger and better LMs slightly improve VLM performance. For instance, LLaVA1.5-13B outperforms LLaVA1.5-7B , and LLaVANeXT-M7B outperforms LLaVANeXT-V7B .

3 Why are VLMs Bad Image Classifiers?

Given that visually-grounded language models (VLMs) underperform CLIPs at classification by a large margin, as reported in §2, we seek to understand the reasons behind that. We investigate several hypotheses concerning major differences between VLMs and CLIPs, which can be generally categorized into inference (§3.1), training (§3.2), and data (§3.3):

We start with inference-related questions. For example, does prompt variation, such as chain of thought, affect final performance? Does reducing the label set size in context narrow the gap between VLMs and CLIPs? Does performing probabilistic inference to force the generation into the label set help? We find none of these factors can fully close the gap between VLMs and CLIPs.

Therefore, we switch to training-related questions. For example, is the visual information from the vision encoder still preserved in the VLM's latent space? Is the text generation objective as effective as cross-entropy loss for learning classification? Surprisingly, the results show that the information is preserved, and the text generation objective is adequate for learning classification.

Finally, we investigate data-related questions. For example, does the VLM training data include enough classification data and cover enough classes? We find a strong correlation between class exposure in training and model performance. Moreover, VLMs can achieve the same level of performance as CLIPs when trained with enough data. These results suggest that data is the primary cause and effective solution for the poor classification performance of VLMs.

3.1 Inference

In this section, we investigate three questions related to VLM's inference, including prompt variation, label set size, and inference strategy.

Prompt variation. It is well known that language models (LMs) are sensitive to prompts [7, 16, 53]. To understand the effect of prompts on classification performance, we tested three semantically

ImageNet

Flowers

Cars

Caltech

100

100

100

100

80

80

80

80

60

60

60

60

40

40

40

40

20

20

20

20

0

0

0

0

2

5

20

100

2

5

20

100

2

5

20

100

2

5

20

100

CLIP-L

similar but differently worded prompts, compared feeding the label in dataset order or random order within the context, and leveraged the zero-shot chain-of-thought (CoT) prompting technique by adding 'let's think step by step' at the end of the prompt [47, 20].

We find that prompt variation has a limited impact on the performance. From Table 2, we can see that changing the wording of prompts results in a performance variation within 3% for both LLaVA1.5-7B and BLIP2-2.7B on the ImageNet dataset. Different label orderings impact LLaVA1.5-7B less than BLIP2-2.7B . Chain-of-thought prompting consistently improves performance for instruction-tuned model LLaVA1.5-7B but not for BLIP2-2.7B .

Label set size. The label set size can be very large in practice (e.g., 1000 for ImageNet, 196 for StanfordCars), which results in an extremely long context when we concatenate all the class names. Since LMs often struggle with long contexts [33], we explore reducing the number of classes. For each image, we randomly select K = 2 , 5 , 20 , 100 classes, always including the ground-truth label, and re-evaluate the VLM and CLIP performance.

We find that the performance gap between VLMs and CLIPs narrows with reduced label size, but the gap always exists . As shown in Figure 2, when evaluating LLaVA1.5-7B and CLIP-L on ImageNet, the gap decreases from 46% with 100 classes to 6% with 2 classes. However, the relative gap becomes larger, evidenced by a 23.9x error rate with 2 classes compared to a 7.3x error rate with 100 classes. The performance gap between VLMs and CLIPs always exists in all the settings.

Inference algorithm. The default inference algorithm for classification with VLMs is directly generating the class name given a prompt. As the generation is open-ended, even when provided with a list of candidate choices, the generation may not match one of the pre-defined classes. To mitigate this problem, we employ probabilistic inference techniques for VLMs. Specifically, for each class name,

Model

Feature

Probing

Last Tok

76.9

94.5

81.0

96.7

77.1

96.2

82.8

97.3

BLIP2-2.7B

Last Tok

80.3

98.8

91.0

98.0

BLIP2-2.7B

81.4

98.9

92.6

98.0

Model

Trainable

Fine-tuning

CLIP-L

Linear

85.2

98.6

91.5

97.6

Proj

85.7

97.6

90.4

97.5

Proj+LM

NaN

97.9

90.7

97.8

EVA-G

Linear

86.5

99.2

94.3

98.5

BLIP2-2.7B

Proj

88.0

99.0

93.9

98.8

we compute prob ( class name | image , prompt ) and select the class name with the highest probability as the prediction. Since class names can consist of multiple tokens (e.g., 'guinea pig' consists of two tokens), we either average the probabilities of all tokens or sum up all the tokens [7]. We also explore classifier-free guidance (CFG) techniques by ranking t ∗ prob ( class name | image , prompt ) + (1 -t ) ∗ prob ( class name | prompt ) with varying guidance coefficients t [38, 50].

We find that the probabilistic inference method improves the performance, but the gap persists. As shown in Table 2, LLaVA1.5-7B achieves 35.3% accuracy on ImageNet using probabilistic inference compared to 22.7% using the direct generation method. Adding classifier-free guidance further improves the performance, where LLaVA1.5-7B performance boosts to 47.6% on ImageNet. However, it still leaves around a 30% performance gap between VLMs and CLIPs, as its vision encoder CLIP-L achieves 74.8% on ImageNet. Moreover, the probabilistic inference approach is computationally expensive in practice because we need to compute the probability of each class.

3.2 Training

Since inference-based modifications fail to close the performance gap between VLMs and CLIPs, here we investigate two questions regarding to the training of VLMs. We study whether the visual information is lost in the VLM and whether the text generation objective is suitable for learning the classification task.

Visual information lost in the VLM. VLMs process images using an image encoder, such as CLIP, which has strong classification capabilities. We hypothesize that, during the propagation of image features output from the vision encoder in language model layers, the necessary information for classification is lost. To test this hypothesis, we conduct feature probing experiments. Specifically, the features from the VLM's last layer have a shape corresponding to the length of the inputs (image tokens plus text tokens using the open-world prompt). We take the average of these token features or the last token feature and train a simple linear classifier on top of these frozen feature representations. The linear classifier is trained on the training set and evaluated on the validation set. Training details are provided in Appendix B.4. A higher accuracy indicates that less information is lost.

Surprisingly, we find that the information necessary for classification is largely preserved in the VLM'slatent space; however, it cannot be effectively decoded . In Table 3, we show that the probing accuracy of LLaVA1.5-7B on ImageNet is 77.1%, which is close to the 85.2% probing accuracy of CLIP-L , the vision encoder used by LLaVA1.5-7B . The same conclusion holds for BLIP2-2.7B and its vision encoder EVA ViT-G/14 , demonstrating that most information is preserved during the VLM computational process. However, this information cannot be effectively decoded, as §2 shows that the best accuracy LLaVA1.5-7B can achieve on ImageNet is 47.6%.

Training objective. Since information is encoded in VLMs, we wonder whether we can train them to decode the information. VLMs can only be trained to perform classification by auto-regressively generating text-form labels. We hypothesize that this generative objective may be more difficult and less effective in learning classification compared to the traditional cross-entropy loss. To understand the effectiveness of this training objective, we explore fine-tuning VLMs on classification datasets. We

100

ImageNetAccuracy

80

60

40

LLaVA

20

CLIP

Finetuned LLaVA

0

[0,10)

[10,20)

[20,30)

[30,40)

[40,50)

[50,60)

[60,70)

[70,80)

[80,90)

[90,100)

[100,1K）[1K,10K)[10K,100K)

(438class)(85class)(45class)(28class)(22class)

(8class)

(18 class) (25 class) (17 class) (15 class)(195 class)(83 class) (16 class)

ClassFrequency

convert classification datasets into the text generation format using the template (e.g., '<image> What type of object is in this photo? <class name>'). We fine-tune two model architectures ( LLaVA1.5-7B and BLIP2-2.7B ) across different datasets and compare their accuracy to that of fine-tuned CLIP models. Additionally, we investigate fine-tuning different parts of the models, such as only fine-tuning the projector between vision encoder and language models or fine-tuning the projector along with the language models using LoRA [17]. Training details are provided in Appendix B.5.

We find that the text generation training objective is as effective as the traditional crossentropy for learning classification tasks, which eliminates the performance gap between VLMs and CLIPs, with VLMs now being the state-of-the-art classifier . From Table 3, we find that LLaVA1.5-7B achieves 85.7% accuracy on ImageNet, same as 85.2% accuracy when fine-tuning its vision encoder CLIP-L . The same findings apply to all the VLMs on all the datasets.

Moreover, we find that fine-tuning only the projector is sufficient and has better numerical stability . From Table 3, we can see the same level of accuracy achieved by fine-tuning the projector and fine-tuning projector and LLM using LoRA on the three datasets. We also find that finetuning LLM with LoRA often leads to numerical instabilities, such as spikes in loss. While the loss sometimes returns to normal, other times it does not. For example, despite trying various hyperparameters for ImageNet, instability persisted (see Appendix §B.5). In contrast, fine-tuning only the projector always results in a steadily decreasing loss. Notably, the projector is often much smaller (e.g., a 2-layer MLP in LLaVA) compared to LLMs, suggesting the potential for parameter-efficient prefix tuning for VLMs [28].

3.3 Data

As observed, the visual information is preserved in the VLM's latent space, and the VLM's training objective is sufficient for learning classification tasks. We hypothesize that the poor classification performance of VLMs is due to their training data. For example, there might be insufficient classification data or a lack of diverse classes. To investigate this, we analyze the LLaVA1.5-7B training data [32], the only fully publicly available VLM training dataset. We examined the relationship between the frequency of class occurrence and the VLM's classification performance on those classes 2 .

Our findings indicate that data determines VLM classification performance . As shown in Figure 3, there is a strong correlation between the presence of class labels in the VLM training data and the VLM's classification accuracy for those classes. The LLaVA1.5-7B model achieves 82.7% zero-shot accuracy on ImageNet classes with more than 10,000 occurrences in its training data, but only 3.0% accuracy on classes with fewer than 10 occurrences. The Spearman correlation between class frequency and class performance is very high (0.76 on ImageNet; see Appendix §B.6). In contrast, there is no correlation between its vision encoder CLIP-L 's performance on those classes with the same VLM training data, and after fine-tuning LLaVA1.5-7B on the ImageNet classification data, the strong correlation disappears. Combining all of our results, we conclude that data plays a critical role in determining the VLM's classification performance.

Given that data availability is critical for VLM classification performance, we now examine whether the data type specifically impacts results. For instance, to train a VLM to classify a class like 'guinea pig,' should it rely on classification-focused data (e.g., '<image> What type of object is in this photo?

Guinea pig') or can it use broader types, such as caption-focused data (e.g., '<image> Generate a caption for the image. A guinea pig playing in the grass') or VQA-focused data (e.g., '<image> What is the native region of this guinea pig? South America')?

In previous experiments (see §3.2), we fine-tuned the VLM with a classification-focused template. Here, we fine-tune the VLM on caption-focused data generated by prompting GPT4 to produce captions incorporating each class's ground-truth name. Specifically, we used the prompt: 'Generate a short caption for the image. The caption must include the ground-truth label of the image, which is <class name>.' This approach produces diverse captions per class, such as 'A guinea pig playing in the grass' or 'A close-up view of a guinea pig in an indoor environment.' We then fine-tune the VLM on this caption-focused data using identical experimental settings and evaluate it against the model trained with classification-focused data. Success is determined by whether the generated caption for a validation image includes the ground-truth class name.

We find that data presence is the primary factor for VLM classification performance, with data type playing a minimal role . From Table 4, we observe that models fine-tuned on both classificationand caption-focused data perform similarly on Flowers102, StanfordCars, and Caltech101, achieving accuracy gains of at least 48.6% over the non-fine-tuned model. This suggests that enabling a VLM to classify a specific class does not require classification-specific data; any data containing the class name suffices.

These findings emphasize a critical data-centric view of VLM training . A common question for VLM training is whether alignment between vision encoder outputs and text generation necessitates multi-modal data with specific class labels, despite both vision and language components seeing these classes in uni-modal pre-training. Previous studies have suggested that this alignment stage may be data-efficient or even unnecessary [32, 18], likely due to their evaluation settings not requiring visioncentric fine-grained recognition. Our work, however, shows that 1) multi-modal data is essential for alignment and 2) increasing data quantity linearly boosts performance. Concurrent work from DeepMind supports our findings, showing that most VLM tasks benefit from pre-training on larger datasets [6].

4 Improving VLM with Classification Data

Based on the analysis presented in §3, in this section, we discuss the enhancement of visuallygrounded language models (VLMs) by integrating classification-focused data into their training. We demonstrate that this data intervention not only boosts the VLM's classification performance but also enhances its general capabilities.

4.1 Motivation

Classification is fundamental to enabling more advanced capabilities of VLMs, such as visual question answering and reasoning. For example, suppose a virtual assistant is helping visually impaired individuals prepare mushrooms as food. In that case, the model must correctly identify the mushroom species to answer questions like 'Is this mushroom poisonous?' The poor classification performance of VLMs lays a weak foundation for their advanced capabilities.

From §3, we identify the primary cause of poor classification performance is the lack of data. Therefore, we propose a straightforward solution: integrating classification-focused data into the VLM training process. We hope that incorporating classification data not only enhances classification accuracy but also improves general capabilities.

Model

Random

GPT4

w/

GPT4

w/o

Naive Baselines

GT Class

Image

Human w/

GT

Class+Wiki

w/

GT

Class

Proprietary VLM

GeminiPro

[44]

Claude3

[3]

GPT4

[36]

Acc

25.0

25.9

100.0

0.0

96.5

55.9

49.1

54.3

61.2

4.2 ImageWikiQA

To verify our hypothesis, we need a dataset that evaluates both the classification and advanced capabilities of VLMs. However, current visual question answering benchmarks, such as VQAv2 [15], MM-Vet [51], and MMMU [52], primarily focus on advanced capabilities, such as reasoning and knowledge grounding, rather than classification. The objects in these datasets are relatively simple, and questions can often be answered by identifying only the general category of an image, such as 'mushroom' or 'flower', rather than specific types of mushrooms or flowers. In contrast, classification datasets have no questions related to more advanced capabilities.

To bridge the gap between classification and advanced capabilities, we introduce ImageWikiQA, an object-centric, knowledge-intensive question answering dataset that combines both worlds. Each question in ImageWikiQA is a multiple-choice question with four options and one correct answer (e.g., '<image showing a guinea pig> What is the native region of this object? A. South America; B. Africa; C. Asia; D. Australia'). Although each question does not directly ask for the category of the object within the image, the question can only be accurately answered if the class of the object is correctly identified. Example questions from ImageWikiQA can be found in Figure 1 (right) and Appendix §C.1.

ImageWikiQA is created by generating questions based on Wikipedia pages of ImageNet classes using GPT4 . Specifically, for each class in ImageNet, we parsed the Wikipedia content of the class following Bujwid et al. [8], and then provided this content along with a prompt to GPT4 to generate five questions per class. We instructed GPT4 to replace the class name with 'this object' in the questions so that the ground-truth class name is not provided. We retained all questions that GPT4 could answer correctly with the ground-truth class name and could not guess the answer without the class name. To ensure the question quality generated by GPT4 , four human annotators attempted to answer the questions with the ground-truth class name and Wikipedia page and achieved an accuracy of 96.5%. Afterward, we randomly sampled at most 3 ImageNet images for each question, rebalanced the choice distribution, and composed the final ImageWikiQA dataset. In total, there are 2000 multiple-choice questions, each with an image, question, four candidate choices, and a reference to Wikipedia sentences, with a random guess accuracy of 25.0% and max frequency accuracy of 25.9%.

4.3 Results

Table 5 presents the performance of various VLMs on the ImageWikiQA dataset.

We find that current state-of-the-art VLMs perform poorly on answering these questions given images . For example, GPT4 achieves 100% accuracy when the ground-truth class name is provided, but only achieves 61.2% accuracy with images. Similarly, Claude3 and GeminiPro only achieve 54.3% and 49.1% accuracy, respectively. These results indicate that the poor classification performance of VLMs is a fundamental limitation for more advanced capabilities.

Furthermore, we find that integrating classification data into VLM training improves its classification and general capabilities . We fine-tuned LLaVA1.5-7B on the ImageNet 1.28M classification data and original 665K LLaVA instruction-tuning data (training detail in §C.2), which is able to achieve 84.4% accuracy on ImageNet classification compared to 22.8% for non-fine-tuned models. This improvement in classification translates to an 11.8% accuracy boost on ImageWikiQA, demonstrating classification is indeed a foundation for VLM's advanced capabilities. However, it is worth noting that fine-tuning solely on the classification task harms general capabilities. When fine-tuning LLaVA1.5-7B on ImageNet only, their accuracy on ImageWikiQA drops to 30.6%. Therefore, finetuning should be performed on a combined dataset. Developing fine-tuning methods that prevent such catastrophic forgetting is a promising future research direction.

5 Related Work

Visually-grounded language models. Visually-grounded language models (VLMs) refer to a large family of models that integrate visual signals into language models by modeling p ( y t | y <t , x ) , where y i is a text token and x is a visual input such as an image or video. Recently, many powerful VLMs have been developed, including proprietary ones like GPT-4V [36], Gemini [44], Claude [3], Flamingo [2] and public ones like LLaVA [32, 31], BLIP [27, 10], OpenFlamingo [4], Otter [26], Fuyu [1], QwenVL [5]. The typical architecture of VLMs comprises three components: a visual encoder (often CLIP [37]), a language model, and a projector that bridges the visual outputs and language model inputs. The projector can be as simple as a linear layer or MLP (e.g., LLaVA [32]) or a complex architecture such as Transformer with cross-modal attention (e.g., BLIP [27]). VLMs are usually trained on image-text captioning data [40, 39] and carefully designed instruction-tuning data [32], with both the vision encoder and language model initialized from pre-trained versions. In this work, we evaluate widely used VLMs in classification settings and analyze two prominent architectures, LLaVA [32] and BLIP [27].

Analysis of visually-grounded language models. While VLMs have demonstrated impressive performance, many questions remain unanswered. Recent works have explored their limitations from various perspectives, including architectures [34], training recipes [18, 24], data [14, 46], vision encoders [45], language models [2, 44], input resolution [41], and proposed solutions for these limitations. For instance, Tong et al. [45] discovered that CLIP vision encoders, employed by most VLMs, often fail to distinguish between certain image pairs despite their apparent visual differences. They suggested utilizing alternative vision encoders, such as DINO, to address this problem. Our work aligns with these studies in understanding VLM limitations and proposing solutions. We find that current VLMs struggle with image classification, and we thoroughly investigate the reasons behind this, proposing a simple solution to integrate classification-focused data into VLM training.

Image classification. Image classification is one of the most fundamental capabilities of machine intelligence. Starting in the 1990s, researchers collected datasets like MNIST [25] for digit classification and CIFAR [22] for object classification, using various machine learning algorithms such as SVMs and MLPs to tackle the problem. A significant milestone was the ImageNet [11], which elevated the scale and quality of datasets to a new level, driving advancements in deep learning. With the rise of deep supervised and self-supervised learning [23, 9], performance on these datasets soon began to saturate. As a result, classification models have superseded most human labelers, leading researchers to focus on more advanced visual intelligence tasks like visual reasoning. In this work, we revisit this simple yet fundamental task, discovering that current VLMs struggle with it. We demonstrate that classification remains essential, as it forms the foundation for more advanced capabilities, and enhancing VLMs' classification capabilities improves their overall performance.

6 Conclusion

In this work, we explored the use of visually-grounded language models (VLMs) as image classifiers. We found that their performance is limited across various datasets. We then analyzed the reasons behind these limitations and, based on our findings, trained a VLM with enhanced general capabilities.

References

Adept. Fuyu-8b: A multimodal architecture for ai agents, 2023.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022.

Anthropic. Introducing the next generation of claude, 2024.

Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An opensource framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 , 2023.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 , 2023.

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 , 2024.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS , 2020.

Sebastian Bujwid and Josephine Sullivan. Large-scale zero-shot image classification from rich and diverse textual descriptions. In EACL Workshop , 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NeurIPS , 2023.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshop , 2004.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. In NeurIPS , 2023.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR , 2017.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR , 2020.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR , 2021.

Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865 , 2024.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. PNAS , 2017.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS , 2022.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshop , 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS , 2012.

Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? arXiv preprint arXiv:2405.02246 , 2024.

Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998.

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726 , 2023.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL , 2021.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In EMNLP , 2023.

Yuanqing Lin, Fengjun Lv, Shenghuo Zhu, Ming Yang, Timothee Cour, Kai Yu, Liangliang Cao, and Thomas Huang. Large-scale image classification: Fast feature extraction and svm training. In CVPR , 2011.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS , 2023.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. TACL , 2023.

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611 , 2024.

M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing , 2008.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. Stay on topic with classifier-free guidance. arXiv preprint arXiv:2306.17806 , 2023.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL , 2018.

Baifeng Shi, Ziyang Wu, Maolin Mao, Xin Wang, and Trevor Darrell. When do we not need larger vision models? arXiv preprint arXiv:2403.13043 , 2024.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR , 2019.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209 , 2024.

Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge. No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance. arXiv preprint arXiv:2404.04125 , 2024.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS , 2022.

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML , 2022.

Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang, and Jingdong Wang. Gpt4vis: what can gpt-4 do for zero-shot visual recognition? arXiv preprint arXiv:2311.15732 , 2023.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. TMLR , 2022.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 , 2023.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR , 2024.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In ICML , 2021.

Broader Impacts and Ethics Statement

In this work, we identify a core limitation of visually-grounded language models (VLMs) and propose a simple solution to address this issue. We find that VLMs underperform in image classification. By integrating classification data into VLM training, we have demonstrated that this intervention not only enhances classification accuracy but also improves the general capabilities of VLMs. This advancement holds promise for real-world applications, such as virtual assistants for visually impaired individuals. However, there are potential risks associated with incorporating race-based or genderbased classification data, which could amplify discriminatory biases in VLMs. We strongly emphasize the importance of ethical and responsible use of our approach to ensure it contributes positively to our digital ecosystem.

Reproducibility Statement

Weprovide an open-source implementation of our work and have released the ImageWikiQA dataset at https://github.com/yuhui-zh15/VLMClassifier . This will enable researchers to reproduce all experiments described in this paper and conduct their own analyses on additional VLMs and datasets.

Limitations

Due to computational constraints, we cannot evaluate all possible VLMs and datasets. We select two representative ones, LLaVA and BLIP, and conduct analyses on four datasets. Additionally, as we find that VLMs have information in their latent space, we conduct large-scale training to decode them. Developing a zero-shot method to decode this information without extensive training could be a promising direction for future research.

Compute Resources

We use four NVIDIA L40S GPUs for all experiments. Most inference experiments take 5-20 hours on a single L40S GPU. The most computationally intensive step is fine-tuning. Fine-tuning LLaVA1.5-7B on 1.28M ImageNet data takes 1.4 days for one epoch using four GPUs. Fine-tuning LLaVA1.5-7B on 1.28M ImageNet data combined with 665K original LLaVA instruction-tuning data takes 3.5 days for one epoch using four GPUs. For the 13B model, the cost is approximately 40% higher.

Summary of Appendix

We provide additional details for each section in the main paper.

In §A, we provide details of models and data for §2.

In §B, we provide details of experimental settings and results for §3.

In §C, we provide details of our collected ImageWikiQA dataset for §4.

In §D, we summarize the contributions of our work.

A Supplementary for Section 2

A.1 Model

We include model details in Table 6.

A.2 Data

We include data details in Table 7.

Table 8: Model accuracy with and without label synonyms on ImageNet, and corresponding improvement.

A.3 Synonymous Text Labels in ImageNet Evaluation

In ImageNet, many classes have multiple valid textual labels. For instance, the class identifier n01496331 corresponds to both 'electric ray' and 'crampfish'. To address this, we incorporated label synonyms 3 in our evaluation. We treat the classification as correct if any textual label is included in the VLM generation. Including synonyms improves accuracy by only 1%-3% (Table 8), still leaving a notable performance gap when compared to CLIP.

B Supplementary for Section 3

B.1 Prompt Variation Detail

Prompt details. We use three different prompts with varying wordings. Base Prompt: 'What type of object is in this photo?' Prompt Alternative 1: 'Identify the object in this image.' Prompt Alternative 2: 'What is the main object depicted in this photograph?'

Label order details. For the fixed order, we use the default dataset label order for each image.

ImageNet

Flowers

Cars

Caltech

30

10

100

20

80

25

8

60

20

15

6

40

15

10

20

10

4

2

5

20

100

2

5

20

100

2

5

20

100

2

5

20

100

RelativeErrorRatebetweenLLaVA1.5-7BandCLIP-L

B.2 Label Set Size Details

Performance details. The table version of Figure 2 is provided in Table 9. By reducing the number of labels for classification, we can narrow the gap between VLM and CLIP, but a gap remains across all label set sizes, even with just two labels (two-way classification).

Relative performance gaps. We find that while the absolute gap between VLMs and CLIPs narrows with reduced label size, the relative gap increases. For example, in two-way classification on ImageNet, VLMs have a 5.7% error rate, while CLIP has a 0.2% error rate, resulting in a 28.5x gap; for 20 classes, VLMs have an 18.0% error rate, while CLIP has a 2.5% error rate, resulting in a 7.2x gap. This trend is indicated by the error rates between VLMs and CLIP shown in Figure 4.

Larger gap on fine-grained classification datasets. The absolute gap on Flowers and Cars is larger than ImageNet and Caltech when reducing the label set size. This may be because Flowers and Cars are more fine-grained classification datasets, while ImageNet and Caltech are more coarse-grained. VLMs are weaker in fine-grained classification compared to CLIP, resulting in a larger gap.

B.3 Inference Strategy Details

Experimental details. Probabilistic inference methods require separate computations for each class, which introduces significant computational costs. For example, on ImageNet with 1000 classes, each example takes 1000 times longer to process than with a direct generation pipeline. Therefore, we randomly sampled 1000 examples from each dataset's validation set for evaluation.

Table 10: Probing the last token or the average token results in much better performance than probing other token positions. Experiments are done using LLaVA1.5-7B on the Flowers102 dataset.

B.4 Information Loss Details

CLIP global vs. local features. The global feature of CLIP is commonly used in traditional classification settings, which has 1 token for each image. However, VLMs utilize local patch features for classification, which has 576 tokens for each image using CLIP-L . One might hypothesize that local features contain less information than global features. In practice, VLMs use the second-to-last layer of CLIP local features rather than the last layer. Since the last layer's global feature is the weighted average of the second-to-last layer's features through self-attention computation, we can theoretically conclude that local features have at least the same level of information as CLIP global features.

Feature extraction details. We extract features from VLM's last layer by feeding the LLaVA default prompt 'USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT:' into the LLaVA, or BLIP default prompt '<Image Tokens> Question: What type of object is in this photo? Answer:' into the BLIP. We use either the last token feature (i.e., the token ':') or the average feature over all the tokens.

Linear probing details. We train the linear layer on the training set with a batch size of 512, a learning rate of 1e-3 using the Adam optimizer, and for 500 epochs. After training, we report the best performance on the validation set.

Probing position. In practice, we find that probing the last token or the average token results in much better performance than probing other token positions, as shown in Table 10. We leave this as an interesting question for future research.

B.5 Training Objective Detail

LLaVA fine-tuning details. We convert each image and class label into the text format using the LLaVA default template 'USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT: <Class Name>.' We conduct two settings for fine-tuning LLaVA. In the first setting, we only fine-tune the MLP projector between CLIP and the language model (LM). The projector is trained on the training set with a batch size of 64 and a learning rate of 2e-5 using the AdamW optimizer for 50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03. In the second setting, we fine-tune both the MLP projector and the LM using LoRA. The projector and LM are trained on the training set with a batch size of 64, a learning rate of 2e-5 for the projector and 2e-4 for the LM, using the AdamW optimizer for 50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03, a LoRA rank of 128, and a LoRA alpha of 256. For both settings, we report the best performance on the validation set after training. Fine-tuning on ImageNet for 1 epoch requires 130 L40 GPU hours.

BLIP fine-tuning details. We convert each image and class label into the text format using the BLIP default template '<Image Tokens> Question: What type of object is in this photo? Answer: <Class Name>.' We train the Q-former projector between CLIP and the LM on the training set with a batch size of 64, a learning rate of 2e-5, and a weight decay of 0.05 using the AdamW optimizer for 100 epochs (10 epochs for ImageNet), with 1000 warmup steps. After training, we report the best performance on the validation set. Fine-tuning on ImageNet for 10 epochs requires 120 L40 GPU hours.

CLIP fine-tuning details. We initialize a linear layer on top of the CLIP image encoder. The CLIP model is frozen, and only the linear layer is trained. The linear layer is trained on the training set with a batch size of 512, a learning rate of 1e-3, and no weight decay using the Adam optimizer for 300 epochs (40 epochs for ImageNet). After training, we report the best performance on the validation set.

train/loss

— finetunellava7b-imagenet-lora-4th — finetunellava7b-imagenet-lora-3rd — finetunellava7b-imagenet-lora-2nd — finetunellava7b-imagenet-lora

-finetunellava7b-imagenet-projector

14

12

10

8

6

0

5k

10k

15k

20

train/loss

finetunellava7b-caltech-lora-finetunellava7b-cars-lora

-finetunellava7b-flowers-lora-finetunellava7b-caltech-projector

-finetunellava7b-cars-projector

-finetunellava7b-flowers-projector

6

train/global_step

200

400

600

800

Numerical instability. When fine-tuning both the projector and the LM with LoRA for LLaVA, we experience significant numerical instability. On ImageNet, despite adjusting three different hyperparameters such as the learning rate and batch size, the loss consistently peaks during training and does not recover. On the StanfordCars dataset, numerical instability is also observed, but the loss returns to normal during training. This can be seen in Figure 5.

ImageNet

Flowers

Cars

Caltech

100

100

100

100

90

80

80

90

80

60

60

70

40

80

40

60

20

70

50

20

0

2

5

20

100

2

5

20

100

2

5

20

100

2

5

20

100

CLIP-L

GPT4

B.6 Data Detail

Data processing details. The training of LLaVA comprises two stages: pre-training and instruction tuning. The pre-training stage utilizes noisy image captioning data, whereas the instruction tuning stage employs meticulously curated multi-turn conversation data. We tokenize all sentences within the data, compute the frequency for each class, and then calculate the accuracy for each class. This results in 1000 (frequency, accuracy) pairs for ImageNet's 1000 classes. This process is similarly applied to all other datasets.

Correlation results. We compute the Spearman and Pearson correlations between frequency and accuracy of zero-shot LLaVA1.5-7B . The results are presented in Table 11. For comparison, we conducted the same analysis for CLIP-L and fine-tuned LLaVA1.5-7B .

Correlation for two stages. Given that LLaVA training involves two stages, we examine whether pre-training or instruction-tuning data has a greater impact on classification performance. We compute the Spearman and Pearson correlations separately for pre-training and instruction-tuning data. Our findings indicate that the correlation is similar for both stages. For instance, on ImageNet, the Spearman and Pearson correlations are 0.73 and 0.34 for pre-training, and 0.76 and 0.33 for instruction tuning. These values are close to the combined data correlations, which are 0.76 and 0.35 for Spearman and Pearson correlations, respectively, as reported in Table 11.

B.7 Analysis of Proprietary VLMs

In the main paper, we analyzed two public VLMs about why they are bad at image classification. For proprietary VLMs, they haven't released the training details and data usage, so it is impossible to directly address training-related or data-related hypotheses.

For inference-related hypotheses, we conduct additional experiments with GPT4 (Table 12, Figure 6), and the conclusion is the same as the public VLMs. For example, GPT-4 achieves 60.6% accuracy on ImageNet without CoT. With CoT, GPT-4 achieves 62.2% accuracy, demonstrating that CoT has a limited impact on image classification, even for large VLMs.

For training-related and data-related hypotheses, recent work PaliGemma [6] from DeepMind shows that most VLM tasks benefit significantly from longer pre-training with more data (in its Figure 4 and Appendix K). This aligns with our main conclusion that data is the critical factor in improving VLM performance.

B.8 Open-world vs. Closed-world Setting

§3 and §4 all use 'open-world' settings (not providing the label set in the prompt), except for prompt variation analysis and label set size analysis in §3.1, using 'closed-world' settings (providing the label set in the prompt).

Naturally, the 'closed-world' setting narrows the model's generation space and increases accuracy. When the model can easily predict the class in the label set by modifying the inference space (probabilistic inference in §3.1 and probing VLM in §3.2) or training VLMs with more classification data (§4), the advantage of the 'closed-world' setting doesn't exist. Thus, for these experiments, we use the 'open-world' setting.

B.9 Number Comparison of Table 1 vs. Table 2

Table 2's 'Base Prompt w/ Label (Random Order)' is equivalent to Table 1's 'Closed-World Setting'. In this setting, we concatenate label candidates in a random order for each question: 'What's in the image? Choose one from random_shuffle([cat, dog, pig])'. The accuracy for these two settings is identical between Table 2 and Table 1.

Table 2's 'Base Prompt w/ Label (Fixed Order)' concatenates label candidates in a fixed order for each question: 'What's in the image? Choose one from [cat, dog, pig]'. This setting is used to rule out the possibility that the order of labels might affect model performance.

B.10 Why Do VLMs Have Information but Cannot Decode?

In 3, we find that VLMs have the essential information for classification. If the VLMs have the information, then why are they not able to decode it?

One possible reason is that the VLM's decoding space is too large and not aligned with the visual features. Specifically, VLM decoding is performed through next-word prediction, which usually involves a vocabulary of over 10,000 words. The output text embedding is not aligned with the visual features due to insufficient data to align these spaces.

In general, having information in a model does not necessarily mean the model can express that information. This phenomenon is also observed in other research areas. For example, after training ResNets or ViTs with self-supervised learning methods like SimCLR, the model acquires discriminative information for different classes. However, this information can only be decoded by adding and training a linear layer on a new dataset.

Similarly, we demonstrate that VLMs possess classification information, but it is not readily expressible. Adding classification-related data helps bridge the gap between 'information possession' and 'information expression' (refer to Table 3).

B.11 Why Does Fine-tuning Only on ImageNet Get Worse Performance on ImageWikiQA?

In Table 5, the 'Finetuned on ImageNet' model performance (30.6) is worse than the non-finetuned model performance (38.0). This is because fine-tuning often improves in-distribution performance but does not necessarily enhance out-of-distribution or general capabilities.

Table 5 presents the results of models fine-tuned on ImageNet and evaluated on ImageWikiQA, which are vastly different datasets. ImageNet questions only require classification, such as 'Q: <image> What is in the image? A: dog,' while ImageWikiQA questions demand both classification and knowledge/reasoning, such as 'Q: <image> What is the native region of this object? A: South America.'

Fine-tuning solely on ImageNet trains the model to classify, but it can lead to a loss of general capabilities like reasoning and knowledge, resulting in lower performance on ImageWikiQA (30.6 fine-tuned vs. 38.0 pre-trained). This phenomenon is known as 'catastrophic forgetting' [19].

100

ImageNetAccuracy

80

60

40

LLaVA

CLIP

20

Finetuned LLaVA

LinearProbedLLaVA

0

[0,10)

[10,20)

[20,30)

[30,40)

[40,50)

[50,60)

[60,70)

[70,80)

[80,90)

[90,100)

[100,1K）

[1K, 10K) [10K,100K)

(438class)(85 class)(45class)(28class)(22class)

(8class)

(18 class) (25 class) (17 class) (15 class) (195 class) (83 class) (16 class)

ClassFrequency

However, when we fine-tune on both ImageNet and instruction tuning data, the model learns classification while retaining its original capabilities, leading to a significant performance improvement on ImageWikiQA (49.8 fine-tuned vs. 38.0 pre-trained).

B.12 Analysis of Linear Probed VLM from Data Perspective

Figure 7 plots results for the linear probed LLaV A model, which aligns closely with the fine-tuned LLaVA. This alignment shows a zero correlation with class frequency since fine-tuning or probing on the balanced ImageNet dataset equalizes class frequencies, where each class is evenly represented.

C Supplementary for Section 4

C.1 ImageWikiQA Details

Data collection details. ImageWikiQA only contains a test set, providing a valuable benchmark for assessing both fine-grained classification and knowledge-based reasoning in VLMs.

The ImageWikiQA dataset was constructed in two stages. In the first stage, we generated questions at the class level for each of the 1,000 ImageNet classes. For instance, questions like 'What is the native region of <class name>?' were created to capture high-level class information. In the second stage, we refined these questions at the image level. For each ImageNet class, we randomly sampled 3 images from the 50 available images from the ImageNet validation set and formulated specific questions like 'What is the native region of <image i>?'. Each question in ImageWikiQA is paired with a single image, ensuring only one image per question.

The ImageWikiQA dataset contains 2,000 unique questions. A question like 'What is the native region of <image i>?' is counted multiple times if it is asked for different images or classes. For example, if the question is asked for 2 cat images and 2 dog images, it would count as 4 questions in total. To avoid over-representing any particular question, a question like 'What is the native region of <image i>?' is limited to a maximum of 3 occurrences across the 2,000 questions.

Data generation prompt. We provide the prompt used to generate ImageWikiQA in Figure 8 and the prompt used to filter ImageWikiQA in Figure 9.

More examples. We provide more examples of the ImageWikiQA dataset in Table 14.

C.2 Results Details

LLaVA fine-tuning details. For ImageNet, we convert each image and class label into the text format using the LLaVA default template 'USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT: <Class Name>.' Then, we concatenate the original LLaVA 665K instruction tuning dataset with the 1.28M ImageNet classification dataset. Due to numerical stability concerns, we train only the projector layer of LLaV A. The projector is trained on the combined dataset with a batch size of 64, a learning rate of 2e-5, using the AdamW optimizer, for 1 epoch, with a warmup ratio of 0.03. The training process takes approximately 340 hours on a single L40 GPU.

C.3 Performance of Fine-tuned VLM on Other Datasets

To understand whether incorporating the ImageNet 1.28M classification data into the original LLaV A instruction-tuning data will harm the model's general capability, we further evaluated the performance of our fine-tuned visually-grounded language model (VLM) on additional benchmarks to ensure its robustness across a variety of tasks. In particular, we tested the model on TextVQA [42], POPE [29], and MMVet [51]. The results, presented in Table 13, show that the fine-tuned LLaVA1.5-7B maintains almost identical performance compared to the original version.

This result aligns with expectations since our fine-tuning process incorporated all of the original LLaVA training data during instruction tuning. For instance, on TextVQA, the fine-tuned model achieved an accuracy of 58.0%, closely matching the original model's 58.2%. Additionally, the finetuned model slightly outperformed the original on POPE Popular and POPE Adverse benchmarks, showing small improvements of 0.2% and 0.3%, respectively. Performance on MMVet remained

unchanged at 31.1%. These findings demonstrate that the fine-tuning process preserves the model's strengths across these varied benchmarks.

C.4 Low Accuracy on ImageWikiQA with Ground-truth Class Names

Table 5 presents the accuracy on ImageWikiQA, which includes questions derived from Wikipedia requiring extensive world knowledge (e.g., 'What is the native region of the guinea pig?').

The modest accuracy of 55.9% by LLaVA1.5-7B with provided ground-truth class names reflects its limited world knowledge for ImageNet classes. This limitation aligns with findings that smaller models, like Vicuna-7B , lack world knowledge compared to larger models like GPT-4.

C.5 Other Fine-Tuning and Inference Strategies

Other fine-tuning or inference modifications, such as applying a linear probing loss in the VLM output space during training or using k-nearest neighbors (KNN) in the VLM output space for evaluation, can also enhance VLM classification performance.

However, our primary goal is to improve VLMs' general capabilities across a range of tasks, not just classification. The universal inference interface for various tasks is text generation, whereas approaches like KNN-based inference are tailored solely to classification and do not generalize to other task types. For fine-tuning, we experimented with adding a new token and applying a linear probing loss. This approach did not yield improvements on ImageWikiQA, indicating that task-specific fine-tuning does not enhance the overall capabilities of VLMs.

In summary, without changing VLM training or inference to maintain its general capabilities for different tasks, adding data is the most promising and effective approach.

D Summary of Contributions

Object recognition is fundamental to the general capability of visually-grounded language models (VLMs), yet current VLMs perform poorly in this area. We are the first to thoroughly investigate this critical issue and propose a potential solution to improve it.

Our primary contribution lies in identifying the problem (i.e., VLMs are inadequate image classifiers, a significant weakness that has been overlooked) and conducting an in-depth investigation (i.e., understanding why VLMs are poor image classifiers and how to address this issue).

In summary, our contributions are threefold:

Thorough Evaluation to Identify the Problem: We thoroughly evaluated current public and proprietary VLMs on four common classification benchmarks and discovered that their performance significantly lags behind CLIP. This finding is counterintuitive because VLMs often use CLIP as their vision encoders. This finding reveals a weakness in VLMs that previous works have not widely noted [2, 49]. Understanding the limitations of VLMs is crucial given their increasing deployment in various scenarios.

Hypothesis Testing to Understand the Problem: Given the poor performance of VLMs in classification, we investigated the underlying reasons. We considered multiple plausible hypotheses related to VLM inference, training, and data perspectives. For example, essential information for classification could be lost during the vision encoder's propagation through multiple LLM layers; VLMs might be inherently poor at classification due to their text generation training objective compared to the standard cross-entropy objective. Our thorough investigation ruled

out these reasons but revealed the major issue: the lack of alignment data. This finding is counterintuitive because other alternative hypotheses also seemed plausible.

Improving VLMs based on Our Understanding: Based on our findings, we explored ways to enhance VLM performance. We believe that classification is foundational to more advanced capabilities. For example, if a VLM cannot accurately classify mushroom species, it will also struggle with follow-up questions, such as whether a particular mushroom is poisonous. Indeed, we found that simply adding classification data not only improves VLMs' classification performance but also their general capabilities, demonstrating that accurately identifying objects is a prerequisite for answering complex questions about these objects.

Impact: Recent work from Google DeepMind, PaliGemma [6], supports our main conclusion that data is the critical factor in improving VLM performance. They also found that most VLM tasks benefit significantly from longer pre-training with more data (Appendix K). This demonstrates that our analysis can inspire researchers to build better VLMs in the future.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims supported in §2 and §3.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in §6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We included all the details in each section and released all the codes and data in https://github.com/yuhui-zh15/VLMClassifier .

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We included all the details in each section and released all the codes and data in https://github.com/yuhui-zh15/VLMClassifier .

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We included all the details in each section and released all the codes and data in https://github.com/yuhui-zh15/VLMClassifier .

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The results are statistically significant, so no error bar is reported.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Discussed in §6.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed in §6.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the assets (data and models) are properly cited in the paper.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We included all the details in §4 and released all the codes and data in https://github.com/yuhui-zh15/VLMClassifier .

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article notes that current top-performing VLMs, like GPT-4V and LLaVA, are unable to perform image classification tasks like the CLIP model. Despite having a larger number of parameters and incorporating a vision encoder from a pre-trained CLIP model.\n\nThis author thinks the main reason for this underperformance is data-related. Critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data.\n\nThe author proposes to incorporate classification-specific datasets (such as ImageNet) into VLM training; its performance in classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.\n\nSTRENGTHS:\nThe article clearly points out the performance issues of the current Visual Language Model (VLM) in image classification tasks. It systematically analyzes the possible reasons, filling the research gap in this field. This author thinks the main reason for this underperformance is data-related. By incorporating classification-specific datasets (such as ImageNet) into VLM training, the performance of classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.\n\nWEAKNESSES:\nThe novelty is limited.\nThis article assesses how well current VLMs perform on classification datasets and fine-tunes them using the same datasets.\nThese contributions are insufficient to warrant publication of this paper.\n\nQUESTIONS:\nI would like to know how well the fine-tuned VLMs perform on the original benchmarks such as MME, TextVQA, and others.\nCan we achieve similar results if a classification dataset is included in the second stage of training with LLaVA?\n\nLIMITATIONS:\nYes, the author has addressed relevant limitations and discussed the broader impacts adequately.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the issue of large vision language models (VLMs) that perform poorly in common image classification datasets such as ImageNet. The authors analyze different perspectives on the problem, including trying different inference and training methods. For inference, the authors tried using different prompt variations, shrinking the number of classes, and computing conditional probabilities of class names. All these methods still leads to performance gaps between VLMs and CLIP. Furthermore, the authors confirm that sufficient class information is encoded in the output features of the visual encoder in VLMs, via linear probing. The authors also discover VLMs can be trained to generate class labels with a comparable accuracy as their visual encoders. Finally, they analyze the frequency of different classes in the original datasets that were used to train the VLMs, and find that the frequency of samples for a class is positively correlated to the accuracy of the VLMs on that class. This reveals that the poor performance of VLMs in image classification is due to the lack of class labels in their instruction-tuning data. \n\nThey also construct a new question-answering dataset, ImageWikiQA, to test the model ability of answering questions related to the finegrained image class. The authors then finetune VLMs using a dataset that consists of both the image classification data and the original nstruction-tuning data. The resulting VLMs can perform well on the ImageWikiQA dataset.\n\nSTRENGTHS:\n- This paper provides a relatively extensive analysis of why SOTA public VLMs perform poorly in image classification. The results generally look correct but there are some issues (as detailed in Weaknesses and Questions) that make them less sound. \n- The authors create a new dataset, ImageWikiQA, to evaluate the question-answering capacity of VLMs related to the fine-grained classes in ImageNet. \n- The authors show that incorporating the ImageNet classification data into VLM's instruction-tuning data can improve the model performance on ImageWikiQA.\n\nWEAKNESSES:\n- It seems that the authors did not consider there may be multiple textual labels for many ImageNet classes. For example, for class with ID n01496331, the class name can be electric ray, crampfish, numbfish, or torpedo. When calculating VLM classification accuracy, the authors can match the model-generated text with any of these class names. \n- It may harm the VLM's instruction-following capacity to incorporate the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data to train the VLM, because the classification data has a single fixed template \"What type of object is in this photo? ASSISTANT: <Class Name>.\" Did the authors perform any evaluation of the VLM's instruction-following capacity? \n- It is expected that the information necessary for classification is largely preserved in the visual encoders in the VLMs, as these visual encoders are typically pre-trained on ImageNet and are kept frozen during the integration into VLMs. If the instruction-tuning data for VLMs do not contain class-specific information, it is unlikely that the VLMs can automatically align the class information in the visual encoder's output to the text generation. This explains why inference methods do not work in the paper. \n- Section 4.2 is a bit unclear. For example, it mentions that the authors randomly sampled at most 3 ImageNet images for each question, but then it mentions there are 2000 multiple-choice questions, each with an image. So how many images are there per question?\n\nQUESTIONS:\n- Can the authors confirm the results in Table 3 (the right sub-table) are the accuracy on the validation/test sets? The accuracy on ImageNet looks very high to me. I have the same question for the accuracy mentioned in line 279. \n- In Table 4, LLaVA1.5-7B with GT Class provide still has a relative low accuracy. Why is that? \n- Can the authors confirm there are no training data in ImageWikiQA (i.e., it is only a testing dataset)?\n\nLIMITATIONS:\nPlease see the Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an interesting observation of VLMs lagging in image classification performance as compared to the visual encoders lie CLIP used within them. Several hypothesis are explored to explain this observation including train-time (information loss, training objective used), inference-time (prompt variations, label set size) and data related reasons. Their analysis shows the primary reason for the observed gap to be data prevalences, showing a correlation between class prevalence during VLM training and performance in those classes for image classification. The paper then simply proposes inclusion of these datasets into VLM training as a way to fix this issue, showing image classification improvements.\n\nSTRENGTHS:\n1. The paper is nicely structured. An interesting performance trend is observed, hypothesis are explored to explain the phenomena, and conclusions drawn from the experiments are further tested. The evaluation and experimental details, the different hypothesis spanning both training and inference, and the well crafted control experiments make it a good study.\n2. The paper is also well motivated and studies a relevant failure mode of VLMs. The results show that even though these modes might encode object or image level concepts, they still suffer from an inability to classify images which is supposed to be a more simpler, and more importantly a more fundamental visual task. This makes it a valuable research problem.\n3. The specific experiments to explore each hypothesis are interesting in their own right. For example studying the effects of prompt ordering or CoT to image classification or different variations of inference or label set size, type of objective used, and linear probing results, all provide useful cues towards the hypothesis, but also showcase interesting VLM behavior.\n4. The paper proposes a very simple and effective way to bridge the VLM performance gap by fine-tuning on the classification data. It shows how doing so leads to not just regaining high image classification performance, but also improves VLM-specific tasks on the classification dataset, for which they curate a new dataset.\n\nWEAKNESSES:\n1. I'm not completely convinced with the paper's final conclusion of data being the reason why CLIP models are superior to VLMs. The fact that linear probes can extract good classification performance from VLMs shows that its a decoding problem since even these models were trained with the same data having skewed prevalences for certain badly performing classes, but the linear probe tuning manages to bring that out to the same degree as CLIP. This implies that either what's missing is a task-aligned fine-tuning (which is what a linear probe does, and which is what the paper does when they fine-tune) or perhaps even a modification of how these are inferred on for image classification which can lead to a more a task-aligned inference (CLIP is trained for text/concept alignment and is evaluated using knn whereas VLMs are trained for vision-text conditioned text generation, but evaluated in a very specific way by attaching tokens of the different options for image classification). In either case, the data argument for explaining the gap might not be the only, or even the strongest reason.\n2. The paper proposes inclusion of the image classification datasets as a solution for this problem. If this VLM limitation stems for other reasons such as multimodal confusion and text feature interference or hallucination problem specific to the generated text space, adding more data for different tasks as a solution might not generalize to similar VML issues. In this case, since the new dataset involves tasks which necessitate good image classification, performance after fine-tuning goes up on both sets of tasks. This might not hold true for other equally fundamental visual tasks.\n\nQUESTIONS:\n1. In Figure 2, the label set size does seem to have some effect on the performance gap b/w CLIPs and VLMs. Since this relates to the way VLMs are evaluated for image classification (attaching class options as tokens in the prompt which grow with the label set), do you think this experiment shows the evaluation difference between the two methods (and also how they are originally trained), can explain some of the gap? Also, why do you think the Caltech and ImageNet datasets behave a little differently than Cars and Flowers in Figure 2?\n2. Why do you think CLIP does not suffer from the class prevalence bias but LLava does in Figure 3? What happens if we plot this for linear probed LLava models? What happens if we plot this for linear probed Llava models which are fine-tuned on similarly biased data having similarly skewed frequencies? If those models do not show this strong correlation, it might be more about the fine-tuning and less about the data distribution.\n\nLIMITATIONS:\nThe paper has discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors explored why Vision-Language Models (VLMs) significantly underperform as image classifiers. They compared several publicly available with proprietary VLMs on several classification benchmark datasets, including ImageNet, Flowers102, StanfordCars, Caltech101, and their newly collected ImageWikiQA dataset. The paper presented multiple hypotheses regarding the underperformance of VLMs, addressing questions related to inference, training objectives, and training/finetuning data. The authors conducted empirical studies and concluded that the performance of VLMs in classification is determined by the data used. They fine-tuned LLaVa1.5-7B on ImageNet and the LLaVa instruction-tuning dataset to enhance classification accuracy.\n\nSTRENGTHS:\n1. The paper is well-organized and easy to follow.\n2. The motivation is clear, and the empirical study is thorough.\n3. The authors analyzed the training data of LLaVA1.5 models to show the strong correlation between class frequency over accuracy.\n\nWEAKNESSES:\n1. The authors did not provide much analysis on the proprietary VLMs. Can the authors address the hypotheses related to inference, training objectives, and data for these VLMs?\n2. The authors introduced “open-world setting” and “closed-wold setting” as evaluation protocols in section 2.3. However, the protocol seems to be only applied to Table 1. It is not clear which settings are being used for sections 3 and 4.\n3. The authors need to provide more explanation for Table 2. What is the difference between the “Closed-World Setting” performance of Table 1 and the “Base Prompt w/ Label (Fixed Order)” in Table 2? Why are their accuracies different?\n4. The LLaVA-7B and BLIP2-7B models are not large enough to demonstrate intrinsic capabilities such as chain-of-thought reasoning [1]. The authors could include the prompt variation results of Table 2 for one of the proprietary VLMs to show the impact on performance.\n5. In Lines 178-179, the authors observed that the information necessary for classification is preserved in VLMs' latent space; however, it cannot be effectively decoded. If the VLMs have the essential information, then why are they not able to decode it? The authors should explain the possible reasons for this. \n6. In Table 4, the finetuned VLM performance is worse than the zero-shot performance of GPT4  on ImageWikiQA. What are the possible reasons for this? Why is the “Finetuned on ImageNet” model performance (30.6) worse than the non-finetuned model performance (37.8)? Does finetuning always enhance the general capabilities of the VLMs?\n\n[1] Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. \"Multimodal chain-of-thought reasoning in language models.\" arXiv preprint arXiv:2302.00923 (2023).\n\nQUESTIONS:\nPlease respond to the points of weakness I mentioned above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}