Why are Visually-Grounded Language Models Bad at Image Classification?

Authors: yuhui  zhang
alyssa  unell
xiaohan  wang
dhruba  ghosh
yuchang  su
ludwig  schmidt
serena  yeung-levy
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-26 19:52:49

Abstract: Image classification is one of the most fundamental capabilities of machine vision intelligence. In this work, we revisit the image classification task using visually-grounded language models (VLMs) such as GPT-4V and LLaVA. We find that existing proprietary and public VLMs, despite often using CLIP as a vision encoder and having many more parameters, significantly underperform CLIP on standard image classification benchmarks like ImageNet. To understand the reason, we explore several hypotheses concerning the inference algorithms, training objectives, and data processing in VLMs. Our analysis reveals that the primary cause is data-related: critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data. Specifically, there is a strong correlation between the frequency of class exposure during VLM training and instruction-tuning and the VLM's performance in those classes; when trained with sufficient data, VLMs can match the accuracy of state-of-the-art classification models. Based on these findings, we enhance a VLM by integrating classification-focused datasets into its training, and demonstrate that the enhanced classification performance of the VLM transfers to its general capabilities, resulting in an improvement of 11.8% on the newly collected ImageWikiQA dataset.

Main paper: 
 introduction: 
 The ability to recognize objects within images is a fundamental capability of machine vision. Over the past 15 years, the field has experienced significant breakthroughs due to deep learning and large-scale datasets [11, 48] . For instance, on the renowned ImageNet dataset, designed to classify images into 1,000 categories, the error rate has dramatically decreased from 47.1% in 2009 to 9.1% in 2024, representing a 5-fold reduction [30, 12] . Consequently, these classification models have superseded most human labelers.Nowadays, the community has focused on more sophisticated and nuanced capabilities in the quest for visual intelligence. Visually-grounded language models (VLMs), which integrate visual signals from vision encoders with large language models, have recently emerged as a promising paradigm [2, 36, 32] . VLMs like GPT-4V [36] , Gemini-1.5 [44] , or Claude-3 [3] have demonstrated advanced visual understanding abilities, such as answering math questions from table images or generating HTML code from design sketches.In this work, we revisit the fundamental task of image classification using VLMs. Surprisingly, we find that various public and proprietary VLMs struggle with image classification in both open-world settings, where the class list is unknown, and closed-world settings, where class names are provided in the context ( §2). Despite having many more parameters, there is a significant gap between the performance of VLMs and their commonly used vision encoder CLIP [37] . Our evaluation protocol involves feeding each image and a list of class names (in the closed-world setting) to the VLM as context and asking what is in the image; success is defined by whether the generated output contains the ground-truth class name.To understand why VLMs underperform in classification settings, we investigate several hypotheses regarding VLMs' inference (such as prompt variations, label set size, inference strategy; §3. 1) , training (such as information lost, training objective; §3.2), and data (such as data-performance correlation; §3.3). Our extensive analyses suggest that the primary reason for the observed gap is data. We find that the information necessary for classification is encoded in the VLM's latent space but can only be decoded with proper training data. Specifically, there is a strong correlation between class presence during VLM training and performance in those classes. Furthermore, training VLMs on classification datasets achieves the same performance level as state-of-the-art classification models.Motivated by our analysis, we propose a simple method to enhance VLMs' general capabilities by integrating traditional classification-focused datasets into VLM training ( §4). We believe that classification is the foundation for more complex, advanced visual capabilities; for example, recognizing an object is a prerequisite for answering complex questions about it. To verify this, we created ImageWikiQA, which contains complex real-world questions about ImageNet objects. On ImageWik-iQA, we find that VLMs fine-tuned on the ImageNet classification dataset achieve substantially higher accuracy in recognizing these objects and provide more accurate answers to these non-classification questions, outperforming pre-trained VLMs by 11.8%. This suggests that classical classification data can be beneficially reused in the VLM training process to enhance VLM performance.In summary, our contributions are threefold:• Evaluating VLM classification weaknesses: Our evaluations using ten VLMs across four benchmarks show that VLMs significantly lag behind CLIP in classification, uncovering a gap unaddressed by previous research.• Analyzing causes of poor classification performance: Testing various hypotheses, we find that the lack of alignment data, rather than information lost or training objective, is the primary reason for VLMs' underperformance in classification.• Enhancing VLMs with classification data: By adding classification data, we improve VLMs' accuracy in object recognition and overall performance, supporting that classification capability is foundational for complex object-related reasoning. VLMs exhibit poor performance in image classification, significantly lagging behind CLIP models. =ImageNet [11] , =Flowers102 [35] , =StanfordCars [21] , =Caltech101 [13] . 
vlms are bad at image classification: 
 We begin by evaluating state-of-the-art visually-grounded language models (VLMs) using standard image classification benchmarks. Our findings reveal that these VLMs significantly underperform compared to state-of-the-art classification models, such as CLIP.VLMs. We selected ten widely-used state-of-the-art VLMs, covering different architectures, training methods, and data. These VLMs include three proprietary ones, GPT-4-Turbo (shortened as GPT4, same below) [36] , Gemini-Pro-Vision (GeminiPro) [44] , and Claude-3-Opus (Claude3) [3] , and seven public ones, LLaVA1.5-Vicuna7B/13B (LLaVA1.5-7/13B) [32] , LLaVANeXT-Mistral7B/Vicuna7B (LLaVANeXT-M7B/V7B) [31] , BLIP2-OPT2.7B (BLIP2-2.7B) [27] , and InstructBLIP-Vicuna7B/13B (IBLIP-7/13B) [10] ). Details of these models are provided in Appendix §A.1.For comparison, we used two state-of-the-art image classifiers, CLIP-ViT-L/14-336px (shortened as CLIP-L, same below) [37] and EVA-ViT-G/14 (EVA-G) [43] . Notably, CLIP-L and EVA-G are utilized by the LLaVA series [32] and the BLIP series as vision encoders [27] , respectively. Therefore, the VLMs should theoretically have the same classification capacity as these vision models. Details are listed in Appendix §A.1.We evaluated the aforementioned models on four widely-used image classification benchmarks: ImageNet (shortened as , same below) [11] , Flowers102 ( ) [35] , StanfordCars ( ) [21] , and Caltech101 ( ) [13] , which contain 50,000, 6,149, 8,041, and 4,331 test images from 1,000, 102, 196, and 101 classes, respectively. ImageNet and Caltech cover more coarse-grained objects, while Flowers and Cars cover more fine-grained objects. Further details are provided in Appendix §A.2.VLMs. We performed image classification in two settings: an open-world setting where the label set is not provided and a closed-world setting where classes are concatenated in the prompt. We feed the image and the prompt to the VLM and let the VLM complete the rest of the tokens. One closed-world example is: "<image> What type of object is in this photo? Choose one from <class name A>, <class name B>, ..." We define success on a single example as whether the ground-truth label is included in the VLM generation. We report the success rate of all test examples.CLIPs. CLIP can only be used in a closed-world setting where the label set is known. Following Radford et al. [37] , we used the prompt "a photo of a <class>" to generate the text feature. For each image, we selected the class with the highest cosine similarity to the image feature. We did not include prompt ensembling to fairly compare with VLM. We report the accuracy of all examples.Table 1 reports the performance of different VLMs and CLIP models on these classification datasets.We find that VLMs exhibit poor performance in image classification, significantly lagging behind CLIP models. For instance, on the ImageNet dataset, the best proprietary VLM, GPT4, only achieves an accuracy of 60.6% in the closed-world setting, whereas the best CLIP, EVA-G, attains an accuracy of 79.2%. In the open-world setting, the best public VLM, LLaVANeXT-M7B, achieves just 32.3% accuracy. The performance disparity is even more pronounced in fine-grained classification datasets like Flowers102 and StanfordCars. Notably, all LLaVA models use CLIP-L as the vision encoder, and although the total parameter count of these models is at least 20 times greater than that of the vision encoder, they significantly underperform compared to it. Moreover, we find that closed-world setting often outperforms open-world setting. This is expected as the provided label set narrows the prediction space. However, since closed-world settings require including all class names in the context, they can result in an extremely long context that leads to high costs or even exceeds the VLM's context limit. For example, most public VLMs only support 4K context length, which cannot feed 1K ImageNet classes. We also find that larger and better LMs slightly improve VLM performance. For instance, LLaVA1.5-13B outperforms LLaVA1.5-7B, and LLaVANeXT-M7B outperforms LLaVANeXT-V7B. 
why are vlms bad image classifiers?: 
 Given that visually-grounded language models (VLMs) underperform CLIPs at classification by a large margin, as reported in §2, we seek to understand the reasons behind that. We investigate several hypotheses concerning major differences between VLMs and CLIPs, which can be generally categorized into inference ( §3.1), training ( §3.2), and data ( §3.3):1. We start with inference-related questions. For example, does prompt variation, such as chain of thought, affect final performance? Does reducing the label set size in context narrow the gap between VLMs and CLIPs? Does performing probabilistic inference to force the generation into the label set help? We find none of these factors can fully close the gap between VLMs and CLIPs. 2. Therefore, we switch to training-related questions. For example, is the visual information from the vision encoder still preserved in the VLM's latent space? Is the text generation objective as effective as cross-entropy loss for learning classification? Surprisingly, the results show that the information is preserved, and the text generation objective is adequate for learning classification. 3. Finally, we investigate data-related questions. For example, does the VLM training data include enough classification data and cover enough classes? We find a strong correlation between class exposure in training and model performance. Moreover, VLMs can achieve the same level of performance as CLIPs when trained with enough data. These results suggest that data is the primary cause and effective solution for the poor classification performance of VLMs.In this section, we investigate three questions related to VLM's inference, including prompt variation, label set size, and inference strategy.Prompt variation. It is well known that language models (LMs) are sensitive to prompts [7, 16, 53] .To understand the effect of prompts on classification performance, we tested three semantically similar but differently worded prompts, compared feeding the label in dataset order or random order within the context, and leveraged the zero-shot chain-of-thought (CoT) prompting technique by adding "let's think step by step" at the end of the prompt [47, 20] .We find that prompt variation has a limited impact on the performance. From Table 2 , we can see that changing the wording of prompts results in a performance variation within 3% for both LLaVA1.5-7B and BLIP2-2.7B on the ImageNet dataset. Different label orderings impact LLaVA1.5-7B less than BLIP2-2.7B. Chain-of-thought prompting consistently improves performance for instruction-tuned model LLaVA1.5-7B but not for BLIP2-2.7B.Label set size. The label set size can be very large in practice (e.g., 1000 for ImageNet, 196 for StanfordCars), which results in an extremely long context when we concatenate all the class names. Since LMs often struggle with long contexts [33] , we explore reducing the number of classes. For each image, we randomly select K = 2, 5, 20, 100 classes, always including the ground-truth label, and re-evaluate the VLM and CLIP performance.We find that the performance gap between VLMs and CLIPs narrows with reduced label size, but the gap always exists. As shown in Figure 2 , when evaluating LLaVA1. we compute prob(class name|image, prompt) and select the class name with the highest probability as the prediction. Since class names can consist of multiple tokens (e.g., "guinea pig" consists of two tokens), we either average the probabilities of all tokens or sum up all the tokens [7] . We also explore classifier-free guidance (CFG) techniques by ranking t * prob(class name|image, prompt) + (1 -t) * prob(class name|prompt) with varying guidance coefficients t [38, 50] .We find that the probabilistic inference method improves the performance, but the gap persists.As shown in Table 2 , LLaVA1.5-7B achieves 35.3% accuracy on ImageNet using probabilistic inference compared to 22.7% using the direct generation method. Adding classifier-free guidance further improves the performance, where LLaVA1.5-7B performance boosts to 47.6% on ImageNet. However, it still leaves around a 30% performance gap between VLMs and CLIPs, as its vision encoder CLIP-L achieves 74.8% on ImageNet. Moreover, the probabilistic inference approach is computationally expensive in practice because we need to compute the probability of each class.Since inference-based modifications fail to close the performance gap between VLMs and CLIPs, here we investigate two questions regarding to the training of VLMs. We study whether the visual information is lost in the VLM and whether the text generation objective is suitable for learning the classification task.Visual information lost in the VLM. VLMs process images using an image encoder, such as CLIP, which has strong classification capabilities. We hypothesize that, during the propagation of image features output from the vision encoder in language model layers, the necessary information for classification is lost. To test this hypothesis, we conduct feature probing experiments. Specifically, the features from the VLM's last layer have a shape corresponding to the length of the inputs (image tokens plus text tokens using the open-world prompt). We take the average of these token features or the last token feature and train a simple linear classifier on top of these frozen feature representations.The linear classifier is trained on the training set and evaluated on the validation set. Training details are provided in Appendix B.4. A higher accuracy indicates that less information is lost.Surprisingly, we find that the information necessary for classification is largely preserved in the VLM's latent space; however, it cannot be effectively decoded. In Table 3 , we show that the probing accuracy of LLaVA1.5-7B on ImageNet is 77.1%, which is close to the 85.2% probing accuracy of CLIP-L, the vision encoder used by LLaVA1.5-7B. The same conclusion holds for BLIP2-2.7B and its vision encoder EVA ViT-G/14, demonstrating that most information is preserved during the VLM computational process. However, this information cannot be effectively decoded, as §2 shows that the best accuracy LLaVA1.5-7B can achieve on ImageNet is 47.6%.Training objective. Since information is encoded in VLMs, we wonder whether we can train them to decode the information. VLMs can only be trained to perform classification by auto-regressively generating text-form labels. We hypothesize that this generative objective may be more difficult and less effective in learning classification compared to the traditional cross-entropy loss. To understand the effectiveness of this training objective, we explore fine-tuning VLMs on classification datasets. We convert classification datasets into the text generation format using the template (e.g., "<image> What type of object is in this photo? <class name>"). We fine-tune two model architectures (LLaVA1.5-7B and BLIP2-2.7B) across different datasets and compare their accuracy to that of fine-tuned CLIP models. Additionally, we investigate fine-tuning different parts of the models, such as only fine-tuning the projector between vision encoder and language models or fine-tuning the projector along with the language models using LoRA [17] . Training details are provided in Appendix B.5.We find that the text generation training objective is as effective as the traditional crossentropy for learning classification tasks, which eliminates the performance gap between VLMs and CLIPs, with VLMs now being the state-of-the-art classifier. From Table 3 , we find that LLaVA1.5-7B achieves 85.7% accuracy on ImageNet, same as 85.2% accuracy when fine-tuning its vision encoder CLIP-L. The same findings apply to all the VLMs on all the datasets.Moreover, we find that fine-tuning only the projector is sufficient and has better numerical stability. From Table 3 , we can see the same level of accuracy achieved by fine-tuning the projector and fine-tuning projector and LLM using LoRA on the three datasets. We also find that finetuning LLM with LoRA often leads to numerical instabilities, such as spikes in loss. While the loss sometimes returns to normal, other times it does not. For example, despite trying various hyperparameters for ImageNet, instability persisted (see Appendix §B.5). In contrast, fine-tuning only the projector always results in a steadily decreasing loss. Notably, the projector is often much smaller (e.g., a 2-layer MLP in LLaVA) compared to LLMs, suggesting the potential for parameter-efficient prefix tuning for VLMs [28] .As observed, the visual information is preserved in the VLM's latent space, and the VLM's training objective is sufficient for learning classification tasks. We hypothesize that the poor classification performance of VLMs is due to their training data. For example, there might be insufficient classification data or a lack of diverse classes. To investigate this, we analyze the LLaVA1.5-7B training data [32] , the only fully publicly available VLM training dataset. We examined the relationship between the frequency of class occurrence and the VLM's classification performance on those classes2 .Our findings indicate that data determines VLM classification performance. As shown in Figure 3 , there is a strong correlation between the presence of class labels in the VLM training data and the VLM's classification accuracy for those classes. The LLaVA1.5-7B model achieves 82.7% zero-shot accuracy on ImageNet classes with more than 10,000 occurrences in its training data, but only 3.0% accuracy on classes with fewer than 10 occurrences. The Spearman correlation between class frequency and class performance is very high (0.76 on ImageNet; see Appendix §B.6). In contrast, there is no correlation between its vision encoder CLIP-L's performance on those classes with the same VLM training data, and after fine-tuning LLaVA1.5-7B on the ImageNet classification data, the strong correlation disappears. Combining all of our results, we conclude that data plays a critical role in determining the VLM's classification performance.Given that data availability is critical for VLM classification performance, we now examine whether the data type specifically impacts results. For instance, to train a VLM to classify a class like "guinea pig," should it rely on classification-focused data (e.g., "<image> What type of object is in this photo? Table 4 : Analysis of data types. We fine-tune the VLM on the caption-focused data generated by GPT4 using the same experimental settings as Table 3 and find that data is the main determining factor for VLM performance, and the data type does not matter much.Guinea pig") or can it use broader types, such as caption-focused data (e.g., "<image> Generate a caption for the image. A guinea pig playing in the grass") or VQA-focused data (e.g., "<image> What is the native region of this guinea pig? South America")?In previous experiments (see §3.2), we fine-tuned the VLM with a classification-focused template.Here, we fine-tune the VLM on caption-focused data generated by prompting GPT4 to produce captions incorporating each class's ground-truth name. Specifically, we used the prompt: "Generate a short caption for the image. The caption must include the ground-truth label of the image, which is <class name>." This approach produces diverse captions per class, such as "A guinea pig playing in the grass" or "A close-up view of a guinea pig in an indoor environment." We then fine-tune the VLM on this caption-focused data using identical experimental settings and evaluate it against the model trained with classification-focused data. Success is determined by whether the generated caption for a validation image includes the ground-truth class name.We find that data presence is the primary factor for VLM classification performance, with data type playing a minimal role. From Table 4 , we observe that models fine-tuned on both classificationand caption-focused data perform similarly on Flowers102, StanfordCars, and Caltech101, achieving accuracy gains of at least 48.6% over the non-fine-tuned model. This suggests that enabling a VLM to classify a specific class does not require classification-specific data; any data containing the class name suffices.These findings emphasize a critical data-centric view of VLM training. A common question for VLM training is whether alignment between vision encoder outputs and text generation necessitates multi-modal data with specific class labels, despite both vision and language components seeing these classes in uni-modal pre-training. Previous studies have suggested that this alignment stage may be data-efficient or even unnecessary [32, 18] , likely due to their evaluation settings not requiring visioncentric fine-grained recognition. Our work, however, shows that 1) multi-modal data is essential for alignment and 2) increasing data quantity linearly boosts performance. Concurrent work from DeepMind supports our findings, showing that most VLM tasks benefit from pre-training on larger datasets [6] . 
improving vlm with classification data: 
 Based on the analysis presented in §3, in this section, we discuss the enhancement of visuallygrounded language models (VLMs) by integrating classification-focused data into their training. We demonstrate that this data intervention not only boosts the VLM's classification performance but also enhances its general capabilities.Classification is fundamental to enabling more advanced capabilities of VLMs, such as visual question answering and reasoning. For example, suppose a virtual assistant is helping visually impaired individuals prepare mushrooms as food. In that case, the model must correctly identify the mushroom species to answer questions like "Is this mushroom poisonous?" The poor classification performance of VLMs lays a weak foundation for their advanced capabilities.From §3, we identify the primary cause of poor classification performance is the lack of data. Therefore, we propose a straightforward solution: integrating classification-focused data into the VLM training process. We hope that incorporating classification data not only enhances classification accuracy but also improves general capabilities. 37.0 IBLIP-13B [10] 37.5 LLaVA1.5-13B [32] 37.8 LLaVA1.5-7B [32] 38.0 LLaVANeXT-M7B [31] 41.9Finetuned VLM LLaVA1.5-7B Finetuned on ImageNet 30.6 LLaVA1.5-7B Finetuned on ImageNet+LLaVA 49.8Table 5 : Evaluations of VLMs on ImageWikiQA. ImageWikiQA is a multiple-choice questionanswering dataset collected by feeding the Wikipedia pages of ImageNet classes to GPT-4. We find that current VLMs perform poorly in answering these questions, suggesting that their poor classification performance is a fundamental limitation for more advanced capabilities. Integrating classification data into VLM training enhances both their classification and overall capabilities.To verify our hypothesis, we need a dataset that evaluates both the classification and advanced capabilities of VLMs. However, current visual question answering benchmarks, such as VQAv2 [15] , MM-Vet [51] , and MMMU [52] , primarily focus on advanced capabilities, such as reasoning and knowledge grounding, rather than classification. The objects in these datasets are relatively simple, and questions can often be answered by identifying only the general category of an image, such as "mushroom" or "flower", rather than specific types of mushrooms or flowers. In contrast, classification datasets have no questions related to more advanced capabilities.To bridge the gap between classification and advanced capabilities, we introduce ImageWikiQA, an object-centric, knowledge-intensive question answering dataset that combines both worlds. Each question in ImageWikiQA is a multiple-choice question with four options and one correct answer (e.g., "<image showing a guinea pig> What is the native region of this object? A. South America; B. Africa; C. Asia; D. Australia"). Although each question does not directly ask for the category of the object within the image, the question can only be accurately answered if the class of the object is correctly identified. Example questions from ImageWikiQA can be found in Figure 1 (right) and Appendix §C.1.ImageWikiQA is created by generating questions based on Wikipedia pages of ImageNet classes using GPT4. Specifically, for each class in ImageNet, we parsed the Wikipedia content of the class following Bujwid et al. [8] , and then provided this content along with a prompt to GPT4 to generate five questions per class. We instructed GPT4 to replace the class name with "this object" in the questions so that the ground-truth class name is not provided. We retained all questions that GPT4 could answer correctly with the ground-truth class name and could not guess the answer without the class name. To ensure the question quality generated by GPT4, four human annotators attempted to answer the questions with the ground-truth class name and Wikipedia page and achieved an accuracy of 96.5%. Afterward, we randomly sampled at most 3 ImageNet images for each question, rebalanced the choice distribution, and composed the final ImageWikiQA dataset. In total, there are 2000 multiple-choice questions, each with an image, question, four candidate choices, and a reference to Wikipedia sentences, with a random guess accuracy of 25.0% and max frequency accuracy of 25.9%.Table 5 presents the performance of various VLMs on the ImageWikiQA dataset.We find that current state-of-the-art VLMs perform poorly on answering these questions given images. For example, GPT4 achieves 100% accuracy when the ground-truth class name is provided, but only achieves 61.2% accuracy with images. Similarly, Claude3 and GeminiPro only achieve 54.3% and 49.1% accuracy, respectively. These results indicate that the poor classification performance of VLMs is a fundamental limitation for more advanced capabilities. Furthermore, we find that integrating classification data into VLM training improves its classification and general capabilities. We fine-tuned LLaVA1.5-7B on the ImageNet 1.28M classification data and original 665K LLaVA instruction-tuning data (training detail in §C.2), which is able to achieve 84.4% accuracy on ImageNet classification compared to 22.8% for non-fine-tuned models. This improvement in classification translates to an 11.8% accuracy boost on ImageWikiQA, demonstrating classification is indeed a foundation for VLM's advanced capabilities. However, it is worth noting that fine-tuning solely on the classification task harms general capabilities. When fine-tuning LLaVA1.5-7B on ImageNet only, their accuracy on ImageWikiQA drops to 30.6%. Therefore, finetuning should be performed on a combined dataset. Developing fine-tuning methods that prevent such catastrophic forgetting is a promising future research direction. 
related work: 
 Visually-grounded language models. Visually-grounded language models (VLMs) refer to a large family of models that integrate visual signals into language models by modeling p(y t |y <t , x), where y i is a text token and x is a visual input such as an image or video. Recently, many powerful VLMs have been developed, including proprietary ones like GPT-4V [36] , Gemini [44] , Claude [3] , Flamingo [2] and public ones like LLaVA [32, 31] , BLIP [27, 10] , OpenFlamingo [4] , Otter [26] , Fuyu [1] , QwenVL [5] . The typical architecture of VLMs comprises three components: a visual encoder (often CLIP [37] ), a language model, and a projector that bridges the visual outputs and language model inputs. The projector can be as simple as a linear layer or MLP (e.g., LLaVA [32] ) or a complex architecture such as Transformer with cross-modal attention (e.g., BLIP [27] ). VLMs are usually trained on image-text captioning data [40, 39] and carefully designed instruction-tuning data [32] , with both the vision encoder and language model initialized from pre-trained versions. In this work, we evaluate widely used VLMs in classification settings and analyze two prominent architectures, LLaVA [32] and BLIP [27] .Analysis of visually-grounded language models. While VLMs have demonstrated impressive performance, many questions remain unanswered. Recent works have explored their limitations from various perspectives, including architectures [34] , training recipes [18, 24] , data [14, 46] , vision encoders [45] , language models [2, 44] , input resolution [41] , and proposed solutions for these limitations. For instance, Tong et al. [45] discovered that CLIP vision encoders, employed by most VLMs, often fail to distinguish between certain image pairs despite their apparent visual differences. They suggested utilizing alternative vision encoders, such as DINO, to address this problem. Our work aligns with these studies in understanding VLM limitations and proposing solutions. We find that current VLMs struggle with image classification, and we thoroughly investigate the reasons behind this, proposing a simple solution to integrate classification-focused data into VLM training.Image classification. Image classification is one of the most fundamental capabilities of machine intelligence. Starting in the 1990s, researchers collected datasets like MNIST [25] for digit classification and CIFAR [22] for object classification, using various machine learning algorithms such as SVMs and MLPs to tackle the problem. A significant milestone was the ImageNet [11] , which elevated the scale and quality of datasets to a new level, driving advancements in deep learning. With the rise of deep supervised and self-supervised learning [23, 9] , performance on these datasets soon began to saturate. As a result, classification models have superseded most human labelers, leading researchers to focus on more advanced visual intelligence tasks like visual reasoning. In this work, we revisit this simple yet fundamental task, discovering that current VLMs struggle with it. We demonstrate that classification remains essential, as it forms the foundation for more advanced capabilities, and enhancing VLMs' classification capabilities improves their overall performance. 
conclusion: 
 In this work, we explored the use of visually-grounded language models (VLMs) as image classifiers. We found that their performance is limited across various datasets. We then analyzed the reasons behind these limitations and, based on our findings, trained a VLM with enhanced general capabilities. 
 
 Appendix: In this work, we identify a core limitation of visually-grounded language models (VLMs) and propose a simple solution to address this issue. We find that VLMs underperform in image classification. By integrating classification data into VLM training, we have demonstrated that this intervention not only enhances classification accuracy but also improves the general capabilities of VLMs. This advancement holds promise for real-world applications, such as virtual assistants for visually impaired individuals. However, there are potential risks associated with incorporating race-based or genderbased classification data, which could amplify discriminatory biases in VLMs. We strongly emphasize the importance of ethical and responsible use of our approach to ensure it contributes positively to our digital ecosystem. In ImageNet, many classes have multiple valid textual labels. For instance, the class identifier n01496331 corresponds to both "electric ray" and "crampfish". To address this, we incorporated label synonyms3 in our evaluation. We treat the classification as correct if any textual label is included in the VLM generation. Including synonyms improves accuracy by only 1%-3% (Table 8 ), still leaving a notable performance gap when compared to CLIP.Prompt details. We use three different prompts with varying wordings. Base Prompt: "What type of object is in this photo?" Prompt Alternative 1: "Identify the object in this image." Prompt Alternative 2: "What is the main object depicted in this photograph?"Label order details. For the fixed order, we use the default dataset label order for each image. 2 is provided in Table 9 . By reducing the number of labels for classification, we can narrow the gap between VLM and CLIP, but a gap remains across all label set sizes, even with just two labels (two-way classification).Relative performance gaps. We find that while the absolute gap between VLMs and CLIPs narrows with reduced label size, the relative gap increases. For example, in two-way classification on ImageNet, VLMs have a 5.7% error rate, while CLIP has a 0.2% error rate, resulting in a 28.5x gap; for 20 classes, VLMs have an 18.0% error rate, while CLIP has a 2.5% error rate, resulting in a 7.2x gap. This trend is indicated by the error rates between VLMs and CLIP shown in Figure 4 .Larger gap on fine-grained classification datasets. The absolute gap on Flowers and Cars is larger than ImageNet and Caltech when reducing the label set size. This may be because Flowers and Cars are more fine-grained classification datasets, while ImageNet and Caltech are more coarse-grained. VLMs are weaker in fine-grained classification compared to CLIP, resulting in a larger gap.Experimental details. Probabilistic inference methods require separate computations for each class, which introduces significant computational costs. For example, on ImageNet with 1000 classes, each example takes 1000 times longer to process than with a direct generation pipeline. Therefore, we randomly sampled 1000 examples from each dataset's validation set for evaluation. Feature extraction details. We extract features from VLM's last layer by feeding the LLaVA default prompt "USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT:" into the LLaVA, or BLIP default prompt "<Image Tokens> Question: What type of object is in this photo? Answer:" into the BLIP. We use either the last token feature (i.e., the token ":") or the average feature over all the tokens.Linear probing details. We train the linear layer on the training set with a batch size of 512, a learning rate of 1e-3 using the Adam optimizer, and for 500 epochs. After training, we report the best performance on the validation set.Probing position. In practice, we find that probing the last token or the average token results in much better performance than probing other token positions, as shown in Table 10 . We leave this as an interesting question for future research.LLaVA fine-tuning details. We convert each image and class label into the text format using the LLaVA default template "USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT: <Class Name>." We conduct two settings for fine-tuning LLaVA. In the first setting, we only fine-tune the MLP projector between CLIP and the language model (LM). The projector is trained on the training set with a batch size of 64 and a learning rate of 2e-5 using the AdamW optimizer for 50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03. In the second setting, we fine-tune both the MLP projector and the LM using LoRA. The projector and LM are trained on the training set with a batch size of 64, a learning rate of 2e-5 for the projector and 2e-4 for the LM, using the AdamW optimizer for 50 epochs (1 epoch for ImageNet), with a warmup ratio of 0.03, a LoRA rank of 128, and a LoRA alpha of 256. For both settings, we report the best performance on the validation set after training. Fine-tuning on ImageNet for 1 epoch requires 130 L40 GPU hours.BLIP fine-tuning details. We convert each image and class label into the text format using the BLIP default template "<Image Tokens> Question: What type of object is in this photo? Answer: <Class Name>." We train the Q-former projector between CLIP and the LM on the training set with a batch size of 64, a learning rate of 2e-5, and a weight decay of 0.05 using the AdamW optimizer for 100 epochs (10 epochs for ImageNet), with 1000 warmup steps. After training, we report the best performance on the validation set. Fine-tuning on ImageNet for 10 epochs requires 120 L40 GPU hours.CLIP fine-tuning details. We initialize a linear layer on top of the CLIP image encoder. The CLIP model is frozen, and only the linear layer is trained. The linear layer is trained on the training set with a batch size of 512, a learning rate of 1e-3, and no weight decay using the Adam optimizer for 300 epochs (40 epochs for ImageNet). After training, we report the best performance on the validation set. Numerical instability. When fine-tuning both the projector and the LM with LoRA for LLaVA, we experience significant numerical instability. On ImageNet, despite adjusting three different hyperparameters such as the learning rate and batch size, the loss consistently peaks during training and does not recover. On the StanfordCars dataset, numerical instability is also observed, but the loss returns to normal during training. This can be seen in Figure 5 . 2 , we added the results for the proprietary GPT4 model. We explore prompt variation such as label order and chainof-thought (CoT) and find it has a limited impact on the performance, which applies to proprietary VLMs such as GPT4.Figure 6 : Analysis of the label set size. Compared to Figure 2 , we added the line for the proprietary GPT4 model. LLaVA1.5-7B uses CLIP-L as its vision encoder, while GPT4 presumably employs a superior vision encoder. Therefore, comparing GPT4 to CLIP-L is not entirely fair. However, we observe that GPT4 performs worse than CLIP-L on ImageNet even when the number of classes is reduced to 2. Correlation results. We compute the Spearman and Pearson correlations between frequency and accuracy of zero-shot LLaVA1.5-7B. The results are presented in Table 11 . For comparison, we conducted the same analysis for CLIP-L and fine-tuned LLaVA1.5-7B.Correlation for two stages. Given that LLaVA training involves two stages, we examine whether pre-training or instruction-tuning data has a greater impact on classification performance. We compute the Spearman and Pearson correlations separately for pre-training and instruction-tuning data. Our findings indicate that the correlation is similar for both stages. For instance, on ImageNet, the Spearman and Pearson correlations are 0.73 and 0.34 for pre-training, and 0.76 and 0.33 for instruction tuning. These values are close to the combined data correlations, which are 0.76 and 0.35 for Spearman and Pearson correlations, respectively, as reported in Table 11 .In the main paper, we analyzed two public VLMs about why they are bad at image classification. For proprietary VLMs, they haven't released the training details and data usage, so it is impossible to directly address training-related or data-related hypotheses.For inference-related hypotheses, we conduct additional experiments with GPT4 (Table 12 , Figure 6 ), and the conclusion is the same as the public VLMs. For example, GPT-4 achieves 60.6% accuracy on ImageNet without CoT. With CoT, GPT-4 achieves 62.2% accuracy, demonstrating that CoT has a limited impact on image classification, even for large VLMs.For training-related and data-related hypotheses, recent work PaliGemma [6] from DeepMind shows that most VLM tasks benefit significantly from longer pre-training with more data (in its Figure 4 and Appendix K). This aligns with our main conclusion that data is the critical factor in improving VLM performance.B.8 Open-world vs. Closed-world Setting §3 and §4 all use "open-world" settings (not providing the label set in the prompt), except for prompt variation analysis and label set size analysis in §3.1, using "closed-world" settings (providing the label set in the prompt).Naturally, the "closed-world" setting narrows the model's generation space and increases accuracy.When the model can easily predict the class in the label set by modifying the inference space (probabilistic inference in §3. In 3, we find that VLMs have the essential information for classification. If the VLMs have the information, then why are they not able to decode it?One possible reason is that the VLM's decoding space is too large and not aligned with the visual features. Specifically, VLM decoding is performed through next-word prediction, which usually involves a vocabulary of over 10,000 words. The output text embedding is not aligned with the visual features due to insufficient data to align these spaces.In general, having information in a model does not necessarily mean the model can express that information. This phenomenon is also observed in other research areas. For example, after training ResNets or ViTs with self-supervised learning methods like SimCLR, the model acquires discriminative information for different classes. However, this information can only be decoded by adding and training a linear layer on a new dataset.Similarly, we demonstrate that VLMs possess classification information, but it is not readily expressible. Adding classification-related data helps bridge the gap between "information possession" and "information expression" (refer to Table 3 ).In Table 5 , the "Finetuned on ImageNet" model performance (30.6 ) is worse than the non-finetuned model performance (38.0) . This is because fine-tuning often improves in-distribution performance but does not necessarily enhance out-of-distribution or general capabilities.Table 5 presents the results of models fine-tuned on ImageNet and evaluated on ImageWikiQA, which are vastly different datasets. ImageNet questions only require classification, such as "Q: <image> What is in the image? A: dog," while ImageWikiQA questions demand both classification and knowledge/reasoning, such as "Q: <image> What is the native region of this object? A: South America."Fine-tuning solely on ImageNet trains the model to classify, but it can lead to a loss of general capabilities like reasoning and knowledge, resulting in lower performance on ImageWikiQA (30.6 fine-tuned vs. 38.0 pre-trained). This phenomenon is known as "catastrophic forgetting" [19] . However, when we fine-tune on both ImageNet and instruction tuning data, the model learns classification while retaining its original capabilities, leading to a significant performance improvement on ImageWikiQA (49.8 fine-tuned vs. 38.0 pre-trained).B.12 Analysis of Linear Probed VLM from Data Perspective The ImageWikiQA dataset was constructed in two stages. In the first stage, we generated questions at the class level for each of the 1,000 ImageNet classes. For instance, questions like "What is the native region of <class name>?" were created to capture high-level class information. In the second stage, we refined these questions at the image level. For each ImageNet class, we randomly sampled 3 images from the 50 available images from the ImageNet validation set and formulated specific questions like "What is the native region of <image i>?". Each question in ImageWikiQA is paired with a single image, ensuring only one image per question.The ImageWikiQA dataset contains 2,000 unique questions. A question like "What is the native region of <image i>?" is counted multiple times if it is asked for different images or classes. For example, if the question is asked for 2 cat images and 2 dog images, it would count as 4 questions in total. To avoid over-representing any particular question, a question like "What is the native region of <image i>?" is limited to a maximum of 3 occurrences across the 2,000 questions.Data generation prompt. We provide the prompt used to generate ImageWikiQA in Figure 8 and the prompt used to filter ImageWikiQA in Figure 9 .More examples. We provide more examples of the ImageWikiQA dataset in Table 14 .LLaVA fine-tuning details. For ImageNet, we convert each image and class label into the text format using the LLaVA default template "USER: <576 Image Tokens> What type of object is in this photo? ASSISTANT: <Class Name>." Then, we concatenate the original LLaVA 665K instruction tuning dataset with the 1.28M ImageNet classification dataset. Due to numerical stability concerns, we train only the projector layer of LLaVA. The projector is trained on the combined dataset with a batch size of 64, a learning rate of 2e-5, using the AdamW optimizer, for 1 epoch, with a warmup ratio of 0.03. The training process takes approximately 340 hours on a single L40 GPU.Come up with five multiple -choice questions for a { classname } to → examine the knowledge of an expert .The questions should come from the following Wikipedia articles on → { classname }: ''' { wikipedia page } ''' ** Instruction **: Each question should have four choices , one of which is the correct → answer . Note that the Wikipedia articles will not be accessible to the → test -takers , so please do not reference specific details from → the articles in the questions . Use " this object " rather than "{ classname }" in the questions . We want → to give test -takers an image of this object , not the word → itself . Each question should have four fields : " question " ( str ) , " choices " → ( list [ str ]) , " answer " ( int , starting from 0) ) , and " reference " → ( str , original sentences from Wikipedia articles ) . Output in JSON format . To understand whether incorporating the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data will harm the model's general capability, we further evaluated the performance of our fine-tuned visually-grounded language model (VLM) on additional benchmarks to ensure its robustness across a variety of tasks. In particular, we tested the model on TextVQA [42] , POPE [29] , and MMVet [51] . The results, presented in Table 13 , show that the fine-tuned LLaVA1.5-7B maintains almost identical performance compared to the original version. This result aligns with expectations since our fine-tuning process incorporated all of the original LLaVA training data during instruction tuning. For instance, on TextVQA, the fine-tuned model achieved an accuracy of 58.0%, closely matching the original model's 58.2%. Additionally, the finetuned model slightly outperformed the original on POPE Popular and POPE Adverse benchmarks, showing small improvements of 0.2% and 0.3%, respectively. Performance on MMVet remained Table 13 : Performance of LLaVA1.5-7B before and after fine-tuning on TextVQA, POPE, and MMVet datasets. Fine-tuning resulted in consistent performance across all benchmarks.unchanged at 31.1%. These findings demonstrate that the fine-tuning process preserves the model's strengths across these varied benchmarks.C.4 Low Accuracy on ImageWikiQA with Ground-truth Class NamesTable 5 presents the accuracy on ImageWikiQA, which includes questions derived from Wikipedia requiring extensive world knowledge (e.g., "What is the native region of the guinea pig?").The modest accuracy of 55.9% by LLaVA1.5-7B with provided ground-truth class names reflects its limited world knowledge for ImageNet classes. This limitation aligns with findings that smaller models, like Vicuna-7B, lack world knowledge compared to larger models like GPT-4.Other fine-tuning or inference modifications, such as applying a linear probing loss in the VLM output space during training or using k-nearest neighbors (KNN) in the VLM output space for evaluation, can also enhance VLM classification performance.However, our primary goal is to improve VLMs' general capabilities across a range of tasks, not just classification. The universal inference interface for various tasks is text generation, whereas approaches like KNN-based inference are tailored solely to classification and do not generalize to other task types. For fine-tuning, we experimented with adding a new token and applying a linear probing loss. This approach did not yield improvements on ImageWikiQA, indicating that task-specific fine-tuning does not enhance the overall capabilities of VLMs.In summary, without changing VLM training or inference to maintain its general capabilities for different tasks, adding data is the most promising and effective approach.Object recognition is fundamental to the general capability of visually-grounded language models (VLMs), yet current VLMs perform poorly in this area. We are the first to thoroughly investigate this critical issue and propose a potential solution to improve it.Our primary contribution lies in identifying the problem (i.e., VLMs are inadequate image classifiers, a significant weakness that has been overlooked) and conducting an in-depth investigation (i.e., understanding why VLMs are poor image classifiers and how to address this issue).In summary, our contributions are threefold:• Thorough Evaluation to Identify the Problem: We thoroughly evaluated current public and proprietary VLMs on four common classification benchmarks and discovered that their performance significantly lags behind CLIP. This finding is counterintuitive because VLMs often use CLIP as their vision encoders. This finding reveals a weakness in VLMs that previous works have not widely noted [2, 49] . Understanding the limitations of VLMs is crucial given their increasing deployment in various scenarios.• Hypothesis Testing to Understand the Problem: Given the poor performance of VLMs in classification, we investigated the underlying reasons. We considered multiple plausible hypotheses related to VLM inference, training, and data perspectives. For example, essential information for classification could be lost during the vision encoder's propagation through multiple LLM layers; VLMs might be inherently poor at classification due to their text generation training objective compared to the standard cross-entropy objective. Our thorough investigation ruled out these reasons but revealed the major issue: the lack of alignment data. This finding is counterintuitive because other alternative hypotheses also seemed plausible.• Improving VLMs based on Our Understanding: Based on our findings, we explored ways to enhance VLM performance. We believe that classification is foundational to more advanced capabilities. For example, if a VLM cannot accurately classify mushroom species, it will also struggle with follow-up questions, such as whether a particular mushroom is poisonous. Indeed, we found that simply adding classification data not only improves VLMs' classification performance but also their general capabilities, demonstrating that accurately identifying objects is a prerequisite for answering complex questions about these objects.Impact: Recent work from Google DeepMind, PaliGemma [6] , supports our main conclusion that data is the critical factor in improving VLM performance. They also found that most VLM tasks benefit significantly from longer pre-training with more data (Appendix K). This demonstrates that our analysis can inspire researchers to build better VLMs in the future.Project page: https://yuhui-zh15.github.io/VLMClassifier-Website/.LLaVA training contains two stages. Here we combine the data from the two stages. More in Appendix B.6.https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a 


Decision: The work reveals a problem that vision language models (e.g. GPT-4o) including proprietary and open-source ones perform poorly in image classification (e.g. ImageNet) compared to CLIP while they are both trained on the Internet data. The question is at which layer do VLMs contain necessary discriminative features? And they find the answer is the last two layers of VLMs i.e., finetuning a linear prob on VLMs improve the classification accuracy. The experiments are quite extensive: 7 open-source VLMs, 3, proprietary VLMs, 4 different datasets. 

Initially, the reviews were mixed (7, 5, 5, 3). After the rebuttal, no reviewers express any concerns. The AC agrees that the authors have addressed most of the concerns raised and recommends `Accept`.

Reviews: review: paper_summary: paper_summary: In this paper, the authors explored why Vision-Language Models (VLMs) significantly underperform as image classifiers. They compared several publicly available with proprietary VLMs on several classification benchmark datasets, including ImageNet, Flowers102, StanfordCars, Caltech101, and their newly collected ImageWikiQA dataset. The paper presented multiple hypotheses regarding the underperformance of VLMs, addressing questions related to inference, training objectives, and training/finetuning data. The authors conducted empirical studies and concluded that the performance of VLMs in classification is determined by the data used. They fine-tuned LLaVa1.5-7B on ImageNet and the LLaVa instruction-tuning dataset to enhance classification accuracy.
strength_weakness: strength_weakness: 1. The authors did not provide much analysis on the proprietary VLMs. Can the authors address the hypotheses related to inference, training objectives, and data for these VLMs?
2. The authors introduced “open-world setting” and “closed-wold setting” as evaluation protocols in section 2.3. However, the protocol seems to be only applied to Table 1. It is not clear which settings are being used for sections 3 and 4.
3. The authors need to provide more explanation for Table 2. What is the difference between the “Closed-World Setting” performance of Table 1 and the “Base Prompt w/ Label (Fixed Order)” in Table 2? Why are their accuracies different?
4. The LLaVA-7B and BLIP2-7B models are not large enough to demonstrate intrinsic capabilities such as chain-of-thought reasoning [1]. The authors could include the prompt variation results of Table 2 for one of the proprietary VLMs to show the impact on performance.
5. In Lines 178-179, the authors observed that the information necessary for classification is preserved in VLMs' latent space; however, it cannot be effectively decoded. If the VLMs have the essential information, then why are they not able to decode it? The authors should explain the possible reasons for this. 
6. In Table 4, the finetuned VLM performance is worse than the zero-shot performance of GPT4  on ImageWikiQA. What are the possible reasons for this? Why is the “Finetuned on ImageNet” model performance (30.6) worse than the non-finetuned model performance (37.8)? Does finetuning always enhance the general capabilities of the VLMs?

[1] Zhang, Zhuosheng, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. "Multimodal chain-of-thought reasoning in language models." arXiv preprint arXiv:2302.00923 (2023).
questions: questions: Please respond to the points of weakness I mentioned above.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents an interesting observation of VLMs lagging in image classification performance as compared to the visual encoders lie CLIP used within them. Several hypothesis are explored to explain this observation including train-time (information loss, training objective used), inference-time (prompt variations, label set size) and data related reasons. Their analysis shows the primary reason for the observed gap to be data prevalences, showing a correlation between class prevalence during VLM training and performance in those classes for image classification. The paper then simply proposes inclusion of these datasets into VLM training as a way to fix this issue, showing image classification improvements.
strength_weakness: strength_weakness: 1. I'm not completely convinced with the paper's final conclusion of data being the reason why CLIP models are superior to VLMs. The fact that linear probes can extract good classification performance from VLMs shows that its a decoding problem since even these models were trained with the same data having skewed prevalences for certain badly performing classes, but the linear probe tuning manages to bring that out to the same degree as CLIP. This implies that either what's missing is a task-aligned fine-tuning (which is what a linear probe does, and which is what the paper does when they fine-tune) or perhaps even a modification of how these are inferred on for image classification which can lead to a more a task-aligned inference (CLIP is trained for text/concept alignment and is evaluated using knn whereas VLMs are trained for vision-text conditioned text generation, but evaluated in a very specific way by attaching tokens of the different options for image classification). In either case, the data argument for explaining the gap might not be the only, or even the strongest reason.
2. The paper proposes inclusion of the image classification datasets as a solution for this problem. If this VLM limitation stems for other reasons such as multimodal confusion and text feature interference or hallucination problem specific to the generated text space, adding more data for different tasks as a solution might not generalize to similar VML issues. In this case, since the new dataset involves tasks which necessitate good image classification, performance after fine-tuning goes up on both sets of tasks. This might not hold true for other equally fundamental visual tasks.
questions: questions: 1. In Figure 2, the label set size does seem to have some effect on the performance gap b/w CLIPs and VLMs. Since this relates to the way VLMs are evaluated for image classification (attaching class options as tokens in the prompt which grow with the label set), do you think this experiment shows the evaluation difference between the two methods (and also how they are originally trained), can explain some of the gap? Also, why do you think the Caltech and ImageNet datasets behave a little differently than Cars and Flowers in Figure 2?
2. Why do you think CLIP does not suffer from the class prevalence bias but LLava does in Figure 3? What happens if we plot this for linear probed LLava models? What happens if we plot this for linear probed Llava models which are fine-tuned on similarly biased data having similarly skewed frequencies? If those models do not show this strong correlation, it might be more about the fine-tuning and less about the data distribution.
limitations: limitations: The paper has discussed limitations.
paper_summary: paper_summary: This paper analyzes the issue of large vision language models (VLMs) that perform poorly in common image classification datasets such as ImageNet. The authors analyze different perspectives on the problem, including trying different inference and training methods. For inference, the authors tried using different prompt variations, shrinking the number of classes, and computing conditional probabilities of class names. All these methods still leads to performance gaps between VLMs and CLIP. Furthermore, the authors confirm that sufficient class information is encoded in the output features of the visual encoder in VLMs, via linear probing. The authors also discover VLMs can be trained to generate class labels with a comparable accuracy as their visual encoders. Finally, they analyze the frequency of different classes in the original datasets that were used to train the VLMs, and find that the frequency of samples for a class is positively correlated to the accuracy of the VLMs on that class. This reveals that the poor performance of VLMs in image classification is due to the lack of class labels in their instruction-tuning data. 

They also construct a new question-answering dataset, ImageWikiQA, to test the model ability of answering questions related to the finegrained image class. The authors then finetune VLMs using a dataset that consists of both the image classification data and the original nstruction-tuning data. The resulting VLMs can perform well on the ImageWikiQA dataset.
strength_weakness: strength_weakness: - It seems that the authors did not consider there may be multiple textual labels for many ImageNet classes. For example, for class with ID n01496331, the class name can be electric ray, crampfish, numbfish, or torpedo. When calculating VLM classification accuracy, the authors can match the model-generated text with any of these class names. 
- It may harm the VLM's instruction-following capacity to incorporate the ImageNet 1.28M classification data into the original LLaVA instruction-tuning data to train the VLM, because the classification data has a single fixed template "What type of object is in this photo? ASSISTANT: <Class Name>." Did the authors perform any evaluation of the VLM's instruction-following capacity? 
- It is expected that the information necessary for classification is largely preserved in the visual encoders in the VLMs, as these visual encoders are typically pre-trained on ImageNet and are kept frozen during the integration into VLMs. If the instruction-tuning data for VLMs do not contain class-specific information, it is unlikely that the VLMs can automatically align the class information in the visual encoder's output to the text generation. This explains why inference methods do not work in the paper. 
- Section 4.2 is a bit unclear. For example, it mentions that the authors randomly sampled at most 3 ImageNet images for each question, but then it mentions there are 2000 multiple-choice questions, each with an image. So how many images are there per question?
questions: questions: - Can the authors confirm the results in Table 3 (the right sub-table) are the accuracy on the validation/test sets? The accuracy on ImageNet looks very high to me. I have the same question for the accuracy mentioned in line 279. 
- In Table 4, LLaVA1.5-7B with GT Class provide still has a relative low accuracy. Why is that? 
- Can the authors confirm there are no training data in ImageWikiQA (i.e., it is only a testing dataset)?
limitations: limitations: Please see the Weaknesses.
paper_summary: paper_summary: This article notes that current top-performing VLMs, like GPT-4V and LLaVA, are unable to perform image classification tasks like the CLIP model. Despite having a larger number of parameters and incorporating a vision encoder from a pre-trained CLIP model.

This author thinks the main reason for this underperformance is data-related. Critical information for image classification is encoded in the VLM's latent space but can only be effectively decoded with enough training data.

The author proposes to incorporate classification-specific datasets (such as ImageNet) into VLM training; its performance in classification and complex visual tasks can be significantly improved. For example, after fine-tuning ImageNet, VLM's performance on the ImageWikiQA dataset improved by 11.8%.
strength_weakness: strength_weakness: The novelty is limited.
This article assesses how well current VLMs perform on classification datasets and fine-tunes them using the same datasets.
These contributions are insufficient to warrant publication of this paper.
questions: questions: I would like to know how well the fine-tuned VLMs perform on the original benchmarks such as MME, TextVQA, and others.
Can we achieve similar results if a classification dataset is included in the second stage of training with LLaVA?
limitations: limitations: Yes, the author has addressed relevant limitations and discussed the broader impacts adequately.

Comments: title: Gentle request for discussion
Gentle request for discussion
Gentle request for discussion
Gentle request for discussion
Please review our response
comment: We are pleased to hear that our response has addressed your concerns, and we will certainly revise the related text as you suggested. Thank you for your time and consideration!
Thanks you for the elaborate discussion and answering all my questions related to label set size and class prevalences. The new experiments you added were very helpful.

Some responses:
> We clarify that data is not the primary reason. The primary reason is that the information required for classification is encoded in the VLM’s latent space but cannot be effectively decoded. Data serves as an effective solution to decode this information, as our experiments show that with sufficient data, VLMs can match the accuracy of state-of-the-art classification models.

Thanks for clarifying this! The paper mentions it in multiple places that data is the reason why VLMs are bad at classification. Here's one of the prominent concluding statement at the end of Section 3: *"These results suggest that data is the primary cause of the poor classification performance of VLMs."*
If the main reason is the inability to decode the latent information, and the solution to it is to fine-tune with classification data, then I would advise the authors to explicitly state that in the introduction to prevent confusion. Currently Figure 1 simply says "Not Enough Data" as the reason which is not true. It's more about being able to align the VLM features using the right kind of data.

The paper successfully shows that VLMs hold the information needed for performing well at image classification (so do their CLIP-based visual encoders), but require some kind of data-aligned fine-tuning on classification or instruction tuning to do well on those tasks. This kind of tuning being necessary is not surprising since you need some way to align the VLM to classes seen at classification time. It's a known observation and often seen in other models as well, but the paper conducts hypothesis-driven experimentation to rule out other possibilities, and in the process generates new ablative results and a new dataset, which is valuable to the community. Illuminating the general problem of VLM needing alignment for classification is a useful direction for future work. I will increase my rating.
Thank you! We will incorporate these in the revised text and are happy to answer any other concerns you have in mind.
Thanks for the response. I will raise my rating to 5.
Dear Reviewer SHnR,

With just 1 day left in the response period, we would greatly appreciate it if you could kindly review our responses soon. We are still looking to discuss any remaining concerns.

Thank you for your time!
Dear Reviewer fPsb,

With just 1 day left in the response period, we would greatly appreciate it if you could kindly review our responses soon. We are still looking to discuss any remaining concerns.

Thank you for your time!
Thank you for your response! Could you clarify which of your concerns remain unaddressed? If you have any other questions or concerns, please feel free to let us know.
Thank the authors for the response.

After reading the rebuttal, I would like to keep the original rating.
Thank you for your reply! We are glad to hear that our response has solved most of your concerns.

> How many unique questions are there in the ImageWikiQA dataset? For example, does a question like “What is the native region of ?” count only once, even if it’s asked for different images and classes?

The ImageWikiQA dataset contains 2,000 unique questions. A question like “What is the native region of <image i>?” is counted multiple times if it is asked for different images or classes. For example, if the question is asked for 2 cat images and 2 dog images, it would count as 4 questions in total. To avoid over-representing any particular question, a question like “What is the native region of <image i>?” is limited to a maximum of 3 occurrences across the 2,000 questions.

Please let us know if you have any further questions!
Thank you for the responses. Most of my concerns are addressed. I have one follow-up question. How many unique questions are there in the ImageWikiQA dataset? For example, a question like “What is the native region of <image i>?” counts only once even though it can be asked for different images and classes.
Dear Reviewer,

We kindly request your feedback on our rebuttal. We believe we have thoroughly addressed your concerns regarding other VLM benchmarks and the novelty of our work.

If our rebuttal has resolved your concerns, we would greatly appreciate it if you could reconsider your scores for our paper. Should you have any further questions or concerns, we are eager to discuss them.

Thank you for your time and consideration!
Dear Reviewer,

We kindly ask for your feedback on our rebuttal. We have conducted all the requested experiments (ImageNet multiple textual labels and instruction following capacity) and clarified concerns that due to misunderstanding.

If our rebuttal has addressed some of your concerns, we would appreciate it if you could reconsider your scores for our paper. Should you have any further questions, please do not hesitate to reach out to us.

Thank you for your time and consideration!
Dear Reviewer,

We kindly request your consideration of our rebuttal. We believe we have thoroughly explained why the data presents a promising solution rather than the root of the problem, and why altering the training or inference objective would not be appropriate. Additionally, we have addressed the other questions raised. 

Thank you for your time and consideration!
Dear reviewer,

We would kindly ask for a response to our rebuttal. We believe that some of your concerns are due to a misunderstanding and are not weaknesses, and we would appreciate it if you would revisit your evaluation of our work. Thank you for your time and consideration!
Dear Reviewers,

We appreciate your valuable feedback and have provided detailed responses to all your comments. We kindly ask you to review our responses. Should you have any further questions, please do not hesitate to initiate a discussion with us.

Thank you for your time and consideration!