What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models

## Abstract

Abstract Large language models (LLMs) have been effectively used for many computer vision tasks, including image classification. In this paper, we present a simple yet effective approach for zero-shot image classification using multimodal LLMs. By employing multimodal LLMs, we generate comprehensive textual representations from input images. These textual representations are then utilized to generate fixed-dimensional features in a cross-modal embedding space. Subsequently, these features are fused together to perform zero-shot classification using a linear classifier. Our method does not require prompt engineering for each dataset; instead, we use a single, straightforward, set of prompts across all datasets. We evaluated our method on several datasets, and our results demonstrate its remarkable effectiveness, surpassing benchmark accuracy on multiple datasets. On average over ten benchmarks, our method achieved an accuracy gain of 4.1 percentage points, with an increase of 6.8 percentage points on the ImageNet dataset, compared to prior methods. Our findings highlight the potential of multimodal LLMs to enhance computer vision tasks such as zero-shot image classification, offering a significant improvement over traditional methods.

## 1 Introduction

Zero-shot image classification aims to categorize images into classes unseen during training, presenting a significant challenge in computer vision. Recent approaches leverage the power of large language models (LLMs) like GPT-4 (Brown et al. ()) to generate prompts for target classes, often in conjunction with vision-language models such as CLIP (Radford et al. ()) to embed images and text in a common space. Open-vocabulary models like CLIP (Radford et al. ()) and VirTex (Desai & Johnson ()) have shown promise in this area due to their ability to generalize to unseen classes. These models learn to match images with captions from vast amounts of image-text data, allowing for dynamic classification without retraining. Early works like DeViSE (Frome et al. ()) pioneered the concept of a joint embedding space for images and text, enabling generalization to unseen classes. Approaches like CLIP (Radford et al. ()), based on contrastive learning, and ALIGN (Jia et al. ()), employing a two-stage framework, further refined the alignment of image and text representations. More recent approaches for zero-shot image classification (e.g., Pratt et al. (); Menon & Vondrick ()) have utilized LLMs to generate prompts (i.e., captions or descriptions) for the target classes to further improve the classification accuracy.

Figure: Figure 1: An illustration of the difference between our work and prior work (e.g., CuPL (Pratt et al. ())) in terms of using LLMs. Prior works use LLMs to describe class labels while we use multimodal LLMs to describe input images and class labels as well as making initial predictions. The shown image is sourced from the ImageNet dataset (Deng et al. ()).
Refer to caption: x1.png

However, relying solely on visual features of the input images during inference can limit accuracy, as these features may not be sufficient to fully capture the nuances present in textual descriptions. To address this, we propose a novel method that leverages the capabilities of multimodal LLMs to generate rich textual representations of the input images. Multimodal LLMs, such as GPT-4 (Brown et al. ()) and Gemini (Gemini Team Google ()), have demonstrated remarkable abilities in various tasks. They are capable of processing and integrating information from various sources like text, images, and audio. This allows them to perform tasks that were previously impossible, such as generating detailed image descriptions, answering complex visual questions, and even creating realistic images from text. Inspired by these advancements, we utilize a straightforward set of prompts to generate detailed textual descriptions of the input images, eliminating the need for complex prompt engineering seen in previous works (e.g., Radford et al. (); Guo et al. ()). These textual representations are then fused with visual features to perform zero-shot classification. See Figure 1.

Our method offers several key advantages: it significantly improves classification accuracy by incorporating richer textual information extracted directly from the input images; it employs a simple and universal set of prompts, eliminating the need for dataset-specific prompt engineering; and it outperforms existing methods on a variety of benchmark datasets.
By employing multimodal LLMs and a straightforward set of prompts, our method outperforms previous zero-shot image classification methods. Specifically, our method achieves an average accuracy gain of 4.1 percentage points over ten image classification benchmark datasets and an accuracy increase of 6.8% on the ImageNet dataset (Deng et al. ()).

In the following sections, we detail our proposed approach for zero-shot image classification using LLMs (Section 2), present experimental results across ten benchmark datasets (Section 3), analyze the computational resources used (Section 4), discuss limitations (Section 5), and conclude with future directions (Section 6).

## 2 Method

Figure: Figure 2: We propose a zero-shot image classification method that leverages multimodal large language models (LLMs) to enhance the accuracy of standard zero-shot classification. Our method employs a set of engineered prompts to generate image description and initial class prediction by the LLM. Subsequently, we encode this data along with the input testing image using a cross-modal embedding encoder to project the inputs into a common feature space. Finally, we fuse the generated features to produce the final query feature, which is then utilized by a standard zero-shot linear image classifier to predict the final class. The shown image is sourced from the Pets dataset (Parkhi et al. ()).
Refer to caption: x2.png

Given an input image, X, containing object(s) belonging to a single class label from a finite set of class labels, $\{l_{i}\}_{i=1}^{m}$, our objective is to classify X without any dataset-specific training process for image classification. The overview of our method is illustrated in Figure 2. As shown in Figure 2, our approach relies on a cross-modal embedding encoder models (image encoder, $f_{i}$, and text encoder, $f_{t}$), trained to learn joint representations of images and text, as demonstrated in prior works, such as Radford et al. (); Frome et al. (); Jia et al. (). Additionally, we utilize a multimodal LLM, $g$, which is pre-trained on a large corpus of multimodal data. This model, $g$, is designed to generate responses that align with both textual and visual inputs, effectively integrating information from both modalities for enhanced predictions (e.g., Gemini Team Google ()).

### 2.1 Class label features

Our zero-shot classifier utilizes class label features of the target dataset. We first encode the set of class labels, $\{l_{i}\}_{i=1}^{m}$, in some embedding space by the cross-modal encoder models, $f_{i}$, and $f_{t}$, such that $\{\textbf{L}_{i}\}_{i=1}^{m}$ refers to the set of encoded class label features, where $\textbf{L}_{i}\in\mathbb{R}^{n}$ is a normalized $n$-D class label feature of the class label $l_{i}$ and $n$ is the dimensionality of the embedded features.

Our zero-shot classifier uses the embedded class labels by representing $\{\textbf{L}_{i}\}$ as a 2D matrix, $\textbf{M}\in\mathbb{R}^{n\times m}$, by stacking all encoded class label features:

$$ $\textbf{M}=[\textbf{L}_{1},\dots,\textbf{L}_{m}].$ (1) $$

Such an encoded feature matrix can be generated using one of three options: (1) directly from the textual class labels, (2) using a human-designed template – e.g., ‘‘A photo of {class_label}” (Radford et al. (); Guo et al. ()), where {class_label} refers to the textual label of one of our classes $\{l_{i}\}_{i=1}^{m}$, or (3) LLM-generated class description(s) (Pratt et al. ()). In option (3), the LLM-generated class description(s) are then converted to embedded features, followed by fusion (e.g., averaging) to generate a single embedded feature for each class label in the dataset.
Optionally, all features from the three options can be fused together (e.g., averaged) for increased robustness.

Figure: Figure 3: Our method utilizes image description and initial class prediction generated by LLM, in addition to the input image, to improve the zero-shot classification accuracy of cross-modal embedding models, such as CLIP (Radford et al. ()). The shown image is from the ImageNet dataset (Deng et al. ()).

### 2.2 Cross-modal input features

To predict the final class, we first encode the input image by the cross-modal image encoder model, $f_{i}$, to generate the image feature $\textbf{X}_{\texttt{if}}\in\mathbb{R}^{n}$ as described below:

$$ $\widetilde{\textbf{X}}_{\texttt{if}}=f_{i}\left(\textbf{X}\right),$ (2) $$

$$ $\textbf{X}_{\texttt{if}}=\frac{1}{\|\widetilde{\textbf{X}}_{\texttt{if}}\|} \widetilde{\textbf{X}}_{\texttt{if}},$ (3) $$

where $\|\cdot\|$ performs vector normalization. Traditionally, the image feature serves as the sole input to prior zero-shot image classifiers (Pratt et al. (); Radford et al. (); Guo et al. ()). Our method enhances this input by incorporating the LLM, $g$, which generates additional textual-based inputs for our zero-shot image classifier (see Figure 3). To achieve this, we employ an engineered prompt that instructs the LLM to describe the input image, X, and perform initial image classification using the textual names of the class label set $\{l_{i}\}_{i=1}^{m}$. Denoting $p_{\texttt{d}}$ and $p_{\texttt{c}}$ as our prompts for image description and initial image classification, respectively, we generate two additional embedded features alongside $\textbf{X}_{\texttt{if}}$ by:

$$ $\widetilde{\textbf{X}}_{\texttt{df}}=\left(f_{t}\circ g\right)\left(\textbf{X} ,p_{\texttt{d}}\right),$ (4) $$

$$ $\textbf{X}_{\texttt{df}}=\frac{1}{\|\widetilde{\textbf{X}}_{\texttt{df}}\|} \widetilde{\textbf{X}}_{\texttt{df}},$ (5) $$

$$ $\widetilde{\textbf{X}}_{\texttt{pf}}=\left(f_{t}\circ g\right)\left(\textbf{X} ,p_{\texttt{c}}\right),$ (6) $$

$$ $\textbf{X}_{\texttt{pf}}=\frac{1}{\|\widetilde{\textbf{X}}_{\texttt{pf}}\|} \widetilde{\textbf{X}}_{\texttt{pf}},$ (7) $$

where $\textbf{X}_{\texttt{df}}\in\mathbb{R}^{n}$ and $\textbf{X}_{\texttt{pf}}\in\mathbb{R}^{n}$ refer to the image description feature and initial class prediction feature, respectively. Notably, in our method, unlike prior methods (e.g., Pratt et al. (); Radford et al. (); Guo et al. ()), such prompts $p_{\texttt{d}}$ and $p_{\texttt{c}}$ do not require dataset-specific engineering for each dataset. Instead, we employ fixed prompts: the first prompt instructs the LLM to provide a generic image description, while the second prompt includes the textual class labels of the target dataset. Further details regarding our prompts are provided in the supplemental materials (Appendix A).

After generating the three input features (image feature, description feature, and initial prediction feature), we fuse them to generate our final query feature, $\textbf{X}_{\texttt{q}}$. One can interpret this fusion as an ensemble of different candidate features to generate a more precise query feature (see Figure 4). We adopted a simple fusion, where the final query feature, $\textbf{X}_{\texttt{q}}$, is generated by:

$$ $\widetilde{\textbf{X}}_{\texttt{q}}={\textbf{X}_{\texttt{if}}+\textbf{X}_{ \texttt{df}}+\textbf{X}_{\texttt{pf}}}\ ,$ (8) $$

$$ $\textbf{X}_{\texttt{q}}=\frac{1}{\|\widetilde{\textbf{X}}_{\texttt{q}}\|} \widetilde{\textbf{X}}_{\texttt{q}}\ .$ (9) $$

We found that this simple fusion yields good results compared to alternative fusion approaches. Refer to Section 3.2 for ablation studies.

### 2.3 Class label prediction

After computing our query feature, $\textbf{X}_{\texttt{q}}$, we apply our zero-shot linear classifier weights, M, to the fused query feature, $\textbf{X}_{\texttt{q}}$, to generate the final similarity scores (i.e., “logits”) of our prediction. This process can be described as follows:

Figure: Figure 4: Our query feature is a fusion of features extracted from the input image, image description, and initial prediction. This fusion operates similarly to ensembling, where our fused query feature demonstrates better robustness, achieving higher accuracy compared to traditional image features used in cross-modal-based zero-shot image classification; e.g., CLIP (Radford et al. ()). On the right, the t-SNE (Van der Maaten & Hinton ()) plot shows the class-embedded features of the ImageNet dataset (Deng et al. ()) (in gray, with the ground-truth class of the shown image in green), our input features (in red), and the query feature after fusion (in blue).
Refer to caption: x4.png

$$ $\textbf{W}=\textbf{X}_{\texttt{q}}^{T}\textbf{M},$ (10) $$

where $T$ represents the vector transpose operation to transform $\textbf{X}_{\texttt{q}}$ into a row vector of shape $1\times n$, and $\textbf{W}\in\mathbb{R}^{1\times m}$ contains the similarity scores of the generated query feature to the class label features in the target dataset. The index of the final predicted class is then computed as $\texttt{argmax}\left(\textbf{W}\right)$ that corresponds to the maximum similarity score.

## 3 Experiments

In our experiments, we employed Gemini Pro (Gemini Team Google ()) as our multimodal LLM, $g$, for generating image descriptions and initial predictions. We utilized CLIP (ViT-L/14) (Radford et al. ()) as our cross-modal embedding encoder models, $f_{i}$ and $f_{t}$, to encode input testing images, image descriptions, and initial predictions generated by Gemini Pro.

We explored four different versions of class label features, M, in Equation 1. Specifically, we used CLIP to encode the following representations of class labels: 1) class label names, 2) the text template ‘‘A photo of {class_label}’’, where {class_label} denotes each class label in each dataset, and 3) class descriptions, similar to those generated by CuPL (Pratt et al. ()), and 4) a combination of the aforementioned three options—akin to the fusion of our input features, we combined the three encoded features of each class and computed their average feature. The class descriptions were produced by prompting Gemini Pro to describe each class label in the dataset 50 times, resulting in 50 different class descriptions for each class. Subsequently, we utilized CLIP to encode the 50 class descriptions for each class and compute the average encoded feature vector to represent each class label. The class description features are generated once as described in Section 2. The exact prompts used to generate image descriptions, initial predictions, and class description are detailed in Appendix A.

### 3.1 Results

We evaluated our method on the following datasets: ImageNet (Deng et al. ()), Pets (Parkhi et al. ()), Places365 (Zhou et al. ()), Food-101 (Bossard et al. ()), SUN397 (Xiao et al. (; )), Stanford Cars (Krause et al. ()), Describable Textures Dataset (DTD) (Cimpoi et al. ()), Caltech-101 (Fei-Fei et al. ()), CIFAR-10 (Krizhevsky et al. ()), and CIFAR-100 (Krizhevsky et al. ()). We compared our results against the following zero-shot classification methods: CLIP (Radford et al. ()), SLIP (Mu et al. ()), PyramidCLIP (Gao et al. ()), nCLIP (Zhou et al. ()), NLIP (Huang et al. ()), UniCLIP (Lee et al. ()), ALIP (Yang et al. ()), CALIP (Guo et al. ()), and CuPL (Pratt et al. ()). For CuPL (Pratt et al. ()), we utilized Gemini Pro (Gemini Team Google ()) for computing class descriptions instead of GPT-3 (Brown et al. ()), ensuring a fair comparison with our method, which also employs Gemini Pro. Additionally, it is worth mentioning that the results reported in the original paper of CuPL (Pratt et al. ()) use different class labels for the ImageNet dataset (Deng et al. ()) compared to the original class labels of the dataset. Consequently, we decided to re-compute the results for CuPL using Gemini Pro with the standard ImageNet class labels.

**Table 1: Comparison of classification accuracy between our method and prior work across various datasets, including ImageNet (Deng et al. ()), CIFAR-10 (C-10) (Krizhevsky et al. ()), CIFAR-100 (C-100) (Krizhevsky et al. ()), Food-101 (Bossard et al. ()), SUN397 (Xiao et al. (; )), Cars (Krause et al. ()), DTD (Cimpoi et al. ()), Caltech-101 (Fei-Fei et al. ()), Pets (Parkhi et al. ()), and Places (Zhou et al. ()). We report our results with the following class label features: 1) class descriptions, 2) class labels, 3) the template “A photo of {class}”, and combined features of (1-3). The symbol $\ast$ denotes training-free methods, whereas the symbol $\diamondsuit$ represents few-shot methods. The best results are highlighted in yellow, and the best zero-shot classification results are highlighted in bold.**
|  | Dataset (number of classes/number of testing images) |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | ImageNet<br>(1K/50K) | C-10<br>(10/10K) | C-100<br>(100/10K) | Food<br>(101/25.3K) | SUN<br>(397/19.9K) | Cars<br>(196/8K) | DTD<br>(47/1.9K) | Caltech<br>(101/8.7K) | Pets<br>(37/3.7K) | Places<br>(365/36.5K) |
| CLIP (ViT-L/14) (Radford et al. ()) | 65.1 | 87.6 | 54.3 | 86.8 | 61.2 | 65.1 | 46.7 | 83.4 | 87.9 | 37.3 |
| CLIP (ViT-B/32) (Radford et al. ()) | 47.0 | 73.4 | 41.4 | 70.8 | 56.5 | 43.8 | 37.0 | 84.4 | 72.9 | 35.6 |
| CLIP (ViT-B/16) (Radford et al. ()) | 54.9 | 66.5 | 37.6 | 80.1 | 58.5 | 52.6 | 37.9 | 85.3 | 80.4 | 36.3 |
| CLIP (RN50) (Radford et al. ()) | 43.0 | 42.7 | 16.0 | 57.4 | 46.3 | 34.3 | 30.5 | 79.1 | 64.7 | 29.7 |
| CLIP (RN101) (Radford et al. ()) | 44.0 | 49.0 | 22.4 | 64.3 | 50.2 | 44.5 | 34.0 | 83.2 | 66.3 | 30.9 |
| SLIP (Mu et al. ()) | 47.9 | 87.5 | 54.2 | 69.2 | 56.0 | 9.0 | 29.9 | 80.9 | 41.6 | - |
| PyramidCLIP (Gao et al. ()) | 47.8 | 81.5 | 53.7 | 67.8 | 65.8 | 65.0 | 47.2 | 81.7 | 83.7 | - |
| nCLIP (Zhou et al. ()) | 48.8 | 83.4 | 54.5 | 65.8 | 59.9 | 18.0 | 57.1 | 73.9 | 33.2 | - |
| NLIP (Huang et al. ()) | 47.4 | 81.9 | 47.5 | 59.2 | 58.7 | 7.8 | 32.9 | 79.5 | 39.2 | - |
| UniCLIP (Lee et al. ()) | 54.2 | 87.8 | 56.5 | 64.6 | 61.1 | 19.5 | 36.6 | 84.0 | 69.2 | - |
| ALIP (Yang et al. ()) | 40.3 | 83.8 | 51.9 | 45.4 | 47.8 | 3.4 | 23.2 | 74.1 | 30.7 | - |
| CALIP (ViT-B/32) (Guo et al. ()) | 60.6 | 76.5 | 44.2 | 77.4 | 58.6 | 56.3 | 42.4 | 87.7 | 86.2 | 36.9 |
| CALIP (ViT-B/16) (Guo et al. ()) | 57.5 | 70.2 | 41.3 | 80.7 | 60.6 | 50.1 | 39.5 | 86.2 | 79.1 | 38.4 |
| CALIP (RN101) (Guo et al. ()) | 50.0 | 49.7 | 24.5 | 67.3 | 51.8 | 39.4 | 35.4 | 84.3 | 70.5 | 33.8 |
| CuPL (Pratt et al. ()) | 66.6 | 86.6 | 57.7 | 89.0 | 65.3 | 63.9 | 49.1 | 90.5 | 80.0 | 39.7 |
| Tip-Adapter ($\ast$) (Zhang et al. ()) | 62.0 | - | - | - | - | - | - | - | - | - |
| SuS-X ($\ast$) (Udandarao et al. ()) | 61.9 | - | - | - | - | - | 50.6 | - | 77.6 | - |
| Tip-Adapter-F ($\diamondsuit$) (Zhang et al. ()) | 65.5 | - | - | - | - | - | - | - | - | - |
| CLIP-Adapter ($\diamondsuit$) (Gao et al. ()) | 61.3 | - | - | - | - | - | 66.1 | 93.4 | - | - |
| APE-T ($\diamondsuit$) (Zhu et al. ()) | 66.1 | - | - | - | - | - | - | - | - | - |
| \hdashlineOurs (descriptions) | 71.3 | 91.2 | 65.3 | 92.5 | 68.8 | 72.0 | 55.1 | 91.3 | 85.0 | 42.0 |
| Ours (class labels) | 69.9 | 91.8 | 65.4 | 92.0 | 66.8 | 74.3 | 53.2 | 88.5 | 90.0 | 41.3 |
| Ours (template) | 69.0 | 93.1 | 65.8 | 89.3 | 64.5 | 73.6 | 52.3 | 84.5 | 87.9 | 39.4 |
| Ours (combined) | 73.4 | 93.4 | 70.2 | 93.0 | 70.6 | 76.6 | 58.0 | 89.4 | 90.9 | 43.4 |

For CLIP (Radford et al. ()) and CALIP (Guo et al. ()), we employed the template “A photo of {class_label}” to encode textual class labels across all datasets, except for Pets (Parkhi et al. ()), DTD (Cimpoi et al. ()), and Cars (Krause et al. ()) datasets. For these exceptions, we employed specific templates: “A photo of {class_label}, a type of pets”, “A photo of {class_label}, a textural category”, and “A photo of {class_label}, a car model”, respectively. This approach was found to enhance results, consistent with prior findings (Radford et al. (); Li et al. (); Allingham et al. (); Popp et al. ()).

For consistency, we resize all images to 224$\times$224 before processing them with our method, CLIP, CuPL, and CALIP methods. Additionally, we report results from training-free and few-shot learning methods for a comprehensive comparison, including: Tip-Adapter (Zhang et al. ()), SuS-X (Udandarao et al. ()), CLIP-Adapter (Gao et al. ()), and APE-T (Zhu et al. ()).

The top-1 accuracy results are reported in Table 1 (see Appendix B for top-5 accuracy results). As can be seen, our method consistently achieves state-of-the-art results when compared with prior work across all datasets in zero-shot image classification methods, and in the majority of datasets when considering other methods (i.e., training-free and few-shot methods). Using the combined class features yields the most promising results across the majority of datasets, with the exception of Caltech-101 (Fei-Fei et al. ()), where the best results were achieved using the class description features.

Figure: Figure 5: Input data highlighted based on its contribution to the final prediction. Examples are shown from the Places (Zhou et al. ()) (first row), ImageNet (Deng et al. ()) (second row), Food-101 (Bossard et al. ()) (third row), and CIFAR-100 (Krizhevsky et al. ()) (last row) datasets.
Refer to caption: x5.png

Figure 5 shows visual examples, where we highlight parts of input modalities (image, initial prediction, and image description) based on their contribution to the final predicted class. To emphasize the significance of each input component in the final prediction, we employ a straightforward approach. Specifically, we utilize a 2D sliding kernel that traverses the image, masking out patches of the image. Subsequently, we measure the difference between the initial prediction and the prediction after masking to highlight areas of the image that contribute most significantly to the final prediction. Similarly, we apply this approach to the textual inputs. Given two distinct input texts – namely, the initial prediction and the image description generated by the LLM (Gemini Team Google ()) – we utilize a sliding kernel with a stride of one word. We mask out words that match the kernel and quantify their importance in our final prediction.
As shown in Figure 5, the three inputs collectively contribute to predicting the final class label. In some cases, one or two inputs exhibit a higher level of influence than the others, as demonstrated in the first, second, and third examples.

It is worth mentioning that, while Gemini’s initial predictions do not match the ground-truth class in the first and second examples, the predictions are contextually sensible. In the first example, Gemini’s prediction was ‘tunnel’ which, while not directly matching any class label in the Places dataset (Zhou et al. ()), conceptually aligns with the displayed ‘catacomb’ image as an underground passage. Similarly, in the second example, Gemini’s initial prediction was ‘Snoopy’, which corresponds to the character drawn on the pen shown in the input image. However, ‘Snoopy’ is not one of the ImageNet (Deng et al. ()) class labels and the correct class of the shown image in second row of Figure 5 is ‘ballpoint pen’. This behavior of LLMs is the reason we cannot use them directly as image classifiers, because they sometimes do not restrict the output class to the provided list of target classes. However, such behavior might be beneficial to other classification tasks that are not restricted to a specific set of classes. Additional examples are provided in the supplemental materials (Appendix B).

### 3.2 Ablation Studies

**Table 2: Ablation study on the impact of features used by our method on the classification accuracy. DF refers to the description feature, PF refers to the prediction feature, and IF refers to the image feature. In all datasets, we employed the best class feature as indicated in Table 1. Specifically, we utilized the combined class feature for all datasets except for Caltech-101 (Fei-Fei et al. ()), where we opted for the class description feature. The best results are highlighted in bold.**
|  | Dataset |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | ImageNet | C-10 | C-100 | Food | SUN | Cars | DTD | Caltech | Pets | Places |
| Ours (DF) | 58.6 | 90.1 | 64.5 | 82.7 | 49.0 | 65.4 | 49.8 | 83.7 | 48.3 | 30.1 |
| Ours (PF) | 55.7 | 94.6 | 73.2 | 89.6 | 60.9 | 70.8 | 57.7 | 89.0 | 87.1 | 36.1 |
| Ours (DF and PF) | 64.5 | 94.4 | 74.0 | 89.8 | 61.6 | 71.6 | 57.7 | 89.3 | 87.5 | 37.0 |
| Ours (DF and IF) | 70.7 | 90.4 | 64.0 | 90.8 | 67.9 | 71.4 | 52.6 | 90.9 | 85.5 | 42.1 |
| Ours (PF and IF) | 71.6 | 92.0 | 67.2 | 92.2 | 69.7 | 74.1 | 56.4 | 91.1 | 90.6 | 42.7 |
| Ours (DF, PF, and IF) | 73.4 | 93.4 | 70.2 | 93.0 | 70.6 | 76.6 | 58.0 | 91.3 | 90.9 | 43.4 |

We conducted a series of ablation studies to explore different versions of our method and investigate the impact of each feature, different fusion approaches, and different cross-modal embedding models. Table 2 presents the results of our method using solely the encoded image description, referred to as the description feature (DF), as our input. We also report the results obtained by using encoded initial predictions, termed as the prediction feature (PF), as our input, as well as using both DF and PF concurrently as inputs. Additionally, Table 2 shows the results of employing image feature (IF) alongside DF or PF as inputs, and finally, we present the results when leveraging all available inputs – specifically, DF, PF, and IF.

From the results in Table 2, it is clear that incorporating all three features (DF, PF, and IF) yields the best performance across most datasets, except for the CIFAR datasets (Krizhevsky et al. ()). This discrepancy may arise from the low resolution of CIFAR images (originally 32$\times$32), where utilizing the IF may degrade accuracy compared to using only DF and PF.

**Table 3: Ablation study on various fusion approaches using 5,000 images randomly selected from the ImageNet dataset (Deng et al. ()). The best results are highlighted in bold.**
|  | CLIP model (Radford et al. ()) |  |  |
| --- | --- | --- | --- |
| Fusion Approach | ViT-L/14 | ViT-B/32 | ViT-B/16 |
| Max similarity | 57.6 | 57.8 | 57.9 |
| Avg similarity | 66.1 | 65.5 | 65.8 |
| Avg feature | 72.5 | 65.7 | 68.5 |

Table 3 shows the results of our second set of ablation studies, where we report the results for 5,000 randomly selected images from the ImageNet dataset (Deng et al. ()). We explore the use of different cross-modal embedding models (CLIP [ViT-L/14], CLIP [ViT-B/32], and CLIP [ViT-B/16]) (Radford et al. ()), and additionally investigate different fusion approaches. Rather than using the mean feature vector of our input features (DF, PF, and IF), we calculated the similarity between each input feature separately and the dataset class label features. Subsequently, we fused the similarity scores to generate a single similarity score for each class label in each dataset. We explored two fusion methods: averaging and taking the maximum for each class label. As shown, averaging our three features (IF, DF, PF) yields the best results.

## 4 Computation resources

Our method relies on a multimodal LLM and cross-modal encoders. The cross-modal encoding takes around 15 ms to encode an image or text on an NVIDIA V100 GPU, while the LLM can be accessed through: 1) Cloud API calls, which do not require local resources to load the model, or 2) loading the model locally for processing, which requires an estimated 16 GPUs/TPUs with approximately 256 GB of memory. Each LLM query takes roughly 700 ms to process. The LLM model is the most intensive operation (as discussed in Section 5), but it can be accelerated using multi-threading.

## 5 Limitations

Our method introduces a new approach by leveraging multimodal LLMs to enhance the accuracy of zero-shot image classification. However, it is important to acknowledge that there are still some limitations inherent in our proposed method. Since our method relies on multiple queries to a multimodal LLM to generate the required features (i.e., DF and PF), there may be potential constraints when running on devices with limited computational power, and it may consume more time compared to other methods. Nevertheless, we believe that advancements in LLMs will lead to models that can run efficiently on lower computational power. This would enable broader accessibility and applicability of such models, such as Gemini (Gemini Team Google ()), GPT (Brown et al. ()), and LLaMA (Touvron et al. ()).

Our method fails in some cases. Figure 6 shows examples of failure cases, where our method misclassify the input image. While the initial prediction and image description features generally enhance classification accuracy, as demonstrated in Table 2, they can sometimes lead to misclassifications. In the first example in Figure 6, both the image description and initial prediction suggest that the image show a ‘lobster’, whereas it actually shows a ‘crayfish’. Similarly, in the second example, the image description lacks specific features of the cat, while the initial prediction suggests the ‘Bengal’ class label, whereas the actual class label is ‘Egyptian Mau’.

Figure: Figure 6: Failure examples of our method, where the initial prediction (and the image description in the first example) adversely influenced our final decision. Results are shown for the Caltech-101 dataset (Fei-Fei et al. ()) (first row) and the Pets dataset (Parkhi et al. ()) (second row).
Refer to caption: x6.png

## 6 Conclusion

In this work, we introduced a zero-shot image classification method that relies on multimodal large language models (LLMs). Our approach involves using a multimodal LLM to describe the input testing image and make an initial class prediction based on input testing image and the target class label names. Subsequently, we fuse the encoded features of the image description, initial LLM’s prediction, and input testing image to retrieve similar encoded features to class labels from the target dataset. Our method is straightforward and easy to implement, resulting in significant improvements in zero-shot classification accuracy when compared with prior methods in this domain.

## Appendix A Additional Details

In the main paper, we presented our method for zero-shot image classification. The inference process of our method is concisely described in Algorithm 1. As part of our method, we employed a set of prompts. Table 4 shows the prompts used for each step discussed in the main paper that employs the LLM (Gemini Pro (Gemini Team Google ())). Specifically, we detail the prompts used to: 1) conduct zero-shot image classification with Gemini Pro (Gemini Team Google ()), 2) describe a given testing image, and 3) generate class labels descriptions.

The class descriptions were generated using five prompts, as shown in Table 4, with 10 responses generated for each prompt and class, resulting in 50 class descriptions per class label. To encourage diversity in Gemini Pro’s responses, we set the temperature parameter to a high value of 0.99, as done in Pratt et al. (). An example of implementing our method, including both classifier construction and the inference process, is shown in Code 1.

**Table 4: Details of prompts utilized in our work. Each row represents one query task to the LLM. For instance, ‘image classification’ indicates the utilization of LLM to conduct initial zero-shot image classification, which serves as one of the features in our method. The {classes} variable refers to the class labels of the dataset. The {predicted_class} refers to Gemini Pro’s output of the image classification prompt. The {class_label} variable denotes one of the class labels in the given dataset.**
| Task | Prompt |
| --- | --- |
| Image classification | You are given an image and a list of class labels. Classify the image given the class labels. Answer using a single word if possible. Here are the class labels: {classes} |
| Image description | What do you see? Describe any object precisely, including its type or class. |
| Class description | 1. Describe what a {class_label} looks like in one or two sentences.<br>2. How can you identify a {class_label} in one or two sentences?<br>3. What does a {class_label} look like? Respond with one or two sentences.<br>4. Describe an image from the internet of a {class_label}. Respond with one or two sentences.<br>5. A short caption of an image of a {class_label}: |

**Table 5: Additional results on 5,000 images from the ImageNet dataset (Deng et al. ()). Best result is highlighted in yellow**
|  | L/14 | B/16 | B/32 | DistilBERT | RoBERTa | ROUGE-N-F1 | ROUGE-F1 | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Top-1 | 52.1 | 52.0 | 55.6 | 43.7 | 28.8 | 54.8 | 54.9 | 70.2 |

In Section 3.1, we visualize examples that highlight the important parts of the inputs contributing to the final predicted class label. In the main paper, we described the approach of sequentially masking out patches from the image and comparing the predicted class with the prediction obtained using the entire unmasked image. Similarly, we follow the same approach for text input by sliding a kernel, masking out words, and comparing the predicted class with our original prediction using inputs without any masking. We used a 2D kernel of size 50$\times$50 pixels with a stride of 10 pixels. If there are no highlighted regions in the image due to the small size of the kernel, we enlarge it by 50 until we reach a kernel size of 200$\times$200 pixels.

For the text kernel, we start with a kernel width of 3 words. If none of the words are highlighted, we reduce it by 1 until we use a 1-word kernel sliding over the text. Each prediction was made using the three inputs: the image, initial prediction, and image description, with one of them having masked out patches or words.

Figure: Algorithm 1 Performs zero-shot image classification.

Figure: Code 1: Example Python implementation of our method. In this example, we utilize the combined class feature, as described in Section 3.

## Appendix B Additional Results

In this section, we provide supplementary results to those presented in the main paper. Figure 7 shows the confusion matrix for CLIP (ViT-L/14) (Radford et al. ()) and our method across two datasets (Caltech-101 (Fei-Fei et al. ()) and CIFAR-100 (Krizhevsky et al. ())). The shown results demonstrate that our method enhances classification accuracy and reduces misclassification rates.

Figure: Figure 7: Confusion matrices for zero-shot image classification results of (A) CLIP (ViT-L/14) (Radford et al. ()) and (B) our method on the Caltech-101 (Fei-Fei et al. ()) and CIFAR-100 (Krizhevsky et al. ()) datasets.
Refer to caption: x7.png

Figure: Figure 8: Additional examples demonstrating the influence of input data on final predictions. Examples are provided from the following datasets: SUN397 (Xiao et al. (; )) (first row), DTD (Cimpoi et al. ()) (second row), and ImageNet (Deng et al. ()) (last three rows).
Refer to caption: x8.png

**Table 6: Top-5 classification accuracy of CLIP (Radford et al. ()), CuPL (Pratt et al. ()), and our method on the following datasets: ImageNet (Deng et al. ()), CIFAR-10 (C-10) (Krizhevsky et al. ()), CIFAR-100 (C-100) (Krizhevsky et al. ()), Food-101 (Bossard et al. ()), SUN397 (Xiao et al. (; )), Cars (Krause et al. ()), DTD (Cimpoi et al. ()), Caltech-101 (Fei-Fei et al. ()), Pets (Parkhi et al. ()), and Places (Zhou et al. ()). We report our results with the following class label features: 1) class descriptions, 2) class labels, 3) the template “A photo of {class}”, and 4) combined features of (1-3). The best results are highlighted in yellow.**
|  | Dataset |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | ImageNet | C-10 | C-100 | Food | SUN | Cars | DTD | Caltech | Pets | Places |
| CLIP (ViT-L/14) (Radford et al. ()) | 88.4 | 98.5 | 77.0 | 97.8 | 89.1 | 93.7 | 72.8 | 95.2 | 96.6 | 64.5 |
| CuPL (Pratt et al. ()) | 91.0 | 98.1 | 79.3 | 98.3 | 92.1 | 94.2 | 77.7 | 99.8 | 96.2 | 68.8 |
| \hdashlineOurs (class descriptions) | 92.7 | 99.3 | 85.4 | 98.9 | 93.8 | 97.6 | 81.0 | 99.9 | 96.9 | 70.7 |
| Ours (class labels) | 89.9 | 99.2 | 84.3 | 98.8 | 91.1 | 97.5 | 77.6 | 98.9 | 98.6 | 67.5 |
| Ours (template) | 89.8 | 99.6 | 84.8 | 97.9 | 90.1 | 97.5 | 77.2 | 96.6 | 97.3 | 65.3 |
| Ours (combined) | 93.0 | 99.6 | 88.5 | 99.0 | 94.4 | 97.9 | 83.8 | 99.9 | 99.5 | 70.9 |

**Table 7: Cohen’s Kappa score of CLIP (Radford et al. ()), CuPL (Pratt et al. ()), and our method on the following datasets: ImageNet (Deng et al. ()), CIFAR-10 (C-10) (Krizhevsky et al. ()), CIFAR-100 (C-100) (Krizhevsky et al. ()), Food-101 (Bossard et al. ()), SUN397 (Xiao et al. (; )), Cars (Krause et al. ()), DTD (Cimpoi et al. ()), Caltech-101 (Fei-Fei et al. ()), Pets (Parkhi et al. ()), and Places (Zhou et al. ()). We report our results with the following class label features: 1) class descriptions, 2) class labels, 3) the template “A photo of {class}”, and 4) combined features of (1-3). The best results are highlighted in yellow.**
|  | Dataset |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | ImageNet | C-10 | C-100 | Food | SUN | Cars | DTD | Caltech | Pets | Places |
| CLIP (ViT-L/14) (Radford et al. ()) | 0.651 | 0.862 | 0.539 | 0.867 | 0.611 | 0.656 | 0.452 | 0.830 | 0.835 | 0.372 |
| CuPL (Pratt et al. ()) | 0.665 | 0.851 | 0.573 | 0.889 | 0.652 | 0.637 | 0.480 | 0.902 | 0.795 | 0.395 |
| \hdashlineOurs (class descriptions) | 0.713 | 0.902 | 0.650 | 0.924 | 0.687 | 0.718 | 0.541 | 0.911 | 0.846 | 0.418 |
| Ours (class labels) | 0.699 | 0.909 | 0.650 | 0.920 | 0.667 | 0.742 | 0.522 | 0.882 | 0.897 | 0.411 |
| Ours (template) | 0.709 | 0.925 | 0.675 | 0.919 | 0.668 | 0.747 | 0.545 | 0.874 | 0.892 | 0.412 |
| Ours (combined) | 0.734 | 0.927 | 0.699 | 0.929 | 0.706 | 0.764 | 0.571 | 0.890 | 0.907 | 0.432 |

In Table 2, encouraging results were demonstrated by utilizing the feature of initial prediction produced by the LLM (i.e., Gemini Pro (Gemini Team Google ())) for zero-shot image classification. Based on these results, one might argue for the direct utilization of Gemini’s class prediction, aiming to match a specific class label from the dataset. However, in several cases, Gemini’s response does not precisely match one of the class labels (as shown in Figure 5). For example, if a ground-truth class label is ‘cat’, Gemini’s response might be ‘The image class is cat’. This discrepancy motivated us to report results of using only Gemini prediction.

In this section, we present additional results from early experiments aimed at utilizing Gemini’s predictions to precisely match one of the class labels in the given dataset. Specifically, we randomly selected 5,000 images from ImageNet (Deng et al. ()) for evaluation. While our method, as presented in the main paper, offers a practical way of utilizing Gemini’s predictions, we also present the results of some alternative approaches aimed at precisely identifying one of the dataset class labels, rather than solely relying on the class prediction text generated by Gemini.

Table 5 show the results on the 5,000 images from ImageNet (Deng et al. ()) of our main method and alternatives that utilize Gemini’s class prediction to conduct similarity matching with the target dataset class labels. Specifically, we report the results of encoding Gemini’s class prediction using an open-vocabulary language model and measuring the similarity with the encoded class label features. Here, we show the results of using CLIP (ViT-L/14, ViT-B/32, ViT-B/16) (Radford et al. ()), DistilBERT (Sanh et al. ()), and RoBERTa (Liu et al. ()).

In addition, we explored classical text similarity metrics – namely, ROUGE-N-F1 and ROUGE-F1 (Lin ()) – rather than encoding both Gemini’s prediction and class labels using an open-vocabulary encoding model. As shown in Table 5, our method, which utilizes Gemini’s class prediction as one of the input features, achieves the best results when compared with the alternative approaches.

In the main paper, we reported the top-1 classification accuracy on several datasets (Deng et al. (); Fei-Fei et al. (); Bossard et al. (); Krause et al. (); Krizhevsky et al. (); Cimpoi et al. (); Zhou et al. (); Xiao et al. (; ); Parkhi et al. ()). Table 6 presents the top-5 classification accuracy of our method compared to prior work, while Table 7 shows the Cohen’s kappa coefficient. As can be seen, our method achieves a notable improvement while remaining simple and easy to implement.

Lastly, Figure 8 shows additional visual examples, where we highlight the most significant contributors from input parts that influence the final predictions of our method.

## Appendix C Broader Impact

Our work introduces a method for zero-shot image classification that leverages the power of multimodal large language models (LLMs) not only during the classifier model construction phase but also at inference time. We achieve this by generating comprehensive textual representations directly from input images. These representations are then combined with the input images for classification, resulting in a significant enhancement in accuracy.

Importantly, our approach eliminates the need for dataset-specific prompt engineering, as commonly required in prior approaches, thereby simplifying the implementation process and enhancing accessibility – effectively acting as a plug-and-play solution. By removing the requirement for dataset-specific customization, our method offers a straightforward and user-friendly approach to zero-shot image classification, making it more accessible to a broader range of users.

By demonstrating its effectiveness across diverse datasets,we illustrate the utility of our method for robust and generalizable real-world computer vision systems reliant on image classification, eliminating the need for dataset-specific training, tuning, or prompt engineering. This approach holds promise for simplifying the deployment of image classification systems and advancing the field of computer vision.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 5
presentation: 4
questions: 1.While the paper provides some qualitative examples of misclassifications, a more comprehensive error analysis could offer insights into potential areas for further improvement. For instance, understanding when and why the textual features lead to incorrect classifications would be valuable.
2.Could other LLM models be used for comparison in addition to Gemini?
soundness: 3
strengths: The proposed method introduces a multimodal LLM-based framework to improve zero-shot image classification by integrating image descriptions and initial class predictions, which is relatively novel and improves on the limitations of using visual features alone.
summary: This paper presents a novel method that leverages multimodal large language models (LLMs) to enhance zero-shot image classification performance. By generating rich textual descriptions for input images and fusing these representations with visual features in a cross-modal embedding space, the approach outperforms several benchmark methods in terms of accuracy. The method uses a consistent set of prompts across datasets, avoiding dataset-specific prompt engineering. The experiments indicate significant performance gains over prior zero-shot approaches, especially in datasets like ImageNet.
weaknesses: The method’s reliance on a multimodal LLM (e.g., Gemini) may restrict its flexibility, as it depends on the capabilities and availability of specific LLMs, which could become a bottleneck.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Authors are encouraged to address the weaknesses in rebuttal/responses. Please refer to the weaknesses for details.
soundness: 2
strengths: - It is reasonable to utilize pretrained foundation models to solve downstream tasks.
- Leveraging generated textual description in visual recognition is a research theme in recent years.
summary: The paper uses pretrained multimodal large language models (LLMs) in zero-shot recognition. It proposes to generate textual represenatation from input images by asking multimodal LLMs such as Gemini Pro. It combines the generated textual representations with features computed by CLIP towards the final zero-shot recognition results. Experiments are done on some standard benchmarks. The paper is limited in novelty and misses citations of relevant works.
weaknesses: - As the paper concerns about "enhancing zero-shot image classification with multimodal large language models" (as suggested in the title), it fails to compare other zero-shot image classification methods which especially leverage pretrained foundation models and their pretraining data, such as [R1-R3].

- The setting is a more serious issue. While datasets used in experiments are standard ones for zero-shot recognition in the literature, images in these datasets are collected on the web, from where one collected data to train the foundation models. Therefore, using output by a (commercial) foundation model such as Gemini Pro is worrisome as it might have information leakage problems. Therefore, a more rigorous setting is to use more highly-specialized tasks which can challenge foundation models.

- While experiments show that the proposed methods outperform the compared ones, it is natural to believe that the benefit is from the use of large pretrained foundation models, especially the commercial Gemini Pro by Google. It makes the paper lack of technical novelties and not ready to publish.

Citations to compare
- [R1] Learning customized visual models with retrieval-augmented knowledge. CVPR 2023
- [R2] Neural priming for sample efficient adaptation. NeurIPS 2023
- [R3] The Neglected Tails of Vision-Language Models. CVPR 2024

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - What could be the reason for CIFAR-10 working best with predicted class features (when those predicted classes are not even guaranteed to be correct), but not working as well with either class description features or image features?
- How often does the failure case in Lines 462-468 observed? Can the authors provide any quantitative measurement for this?
soundness: 3
strengths: - The paper is written very well, effectively communicating the method description and experiments in a concise and easy to understand way.
- The proposed approach shows significant improvement on almost every dataset traditionally used for benchmarking zero-shot image classification.
- It appears to be the first work using multimodal capabilities of MLLMs for this task, which is a natural next step from text-based features used in previous LLM methods for this task.
summary: This paper introduces a novel approach for using the multimodal capabilities of modern LLMs for enhancing zero-shot classification. The proposed method is straightforward in its formulation: instead of only relying on text prompts generated by an LLM for a given class label, use three different multimodal features (image, image description, and an initial prediction) to form a multimodal query. The combined features are then compared against the list of class label features to find the label with maximum similarity, like the CLIP model. Despite its simple formulation, the method shows significant improvement in zero-shot classification across multiple standard datasets for this task.
weaknesses: - The discussion of the results could be improved. For example, in lines 413-420, the results of the ablation study in Table 3 are described. But there is no discussion to explain the results. Particularly, line 420 says averaging features leads to best performance, but it would be better if the authors also provide a likely reason for why other fusion methods do not work.
- In Table 2 caption, the authors state they used the combined class features for all dataset except Caltech where class description features were used. Again, no reason is provided for this decision.
- Table 2 could also benefit from an additional row showing only image feature (IF) input for a more complete picture.
- Line 74-75 implies that feature fusion is illustrated in Figure 1, but the corresponding figure does not show this. The wording of this paragraph should be corrected.
- Lines 32-33 state that recent approaches have utilized LLMs to generate prompts for target classes, but this sentence does not cite these recent approaches. Some examples could be I2MVFormer by Naeem et al, or FocusCLIP by Khan et al. The authors should either use these or any other references which are suitable here.
- I am not completely convinced by the reason provided in Lines 411-412 for why the CIFAR dataset show best performance without the image features.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: I would appreciate clarifications on the choices described in the weaknesses above.
soundness: 3
strengths: The method proposed is simple but effective. The paper is well presented where the method is presented with clarity. Ablations of the method has also been conducted for various contributing factors. Combination of various prompting strategies to the CLIP model for classification and their effects is also a useful contribution.
summary: This paper proposes to improve zero-shot classification by using multimodal LLMs. The proposed method computes additional textual embeddings for each image in the test set by prompting a multimodal LLM and converting the prompt to an embedding using CLIP's text encoder. The paper utilizes two types of prompts for the multimodal LLM -- one which involves generating a description of the image, and the other which involves classifying the image. Results are shown on various datasets where the method improves over the baselines considered.
weaknesses: Missing datasets : Some datasets which are utilized by previous methods such as CUB, FGVC Aircraft, EuroSAT have been overlooked. Is it the case that the multimodal LLM does not possess useful knowledge about these fine-grained datasets and thus cannot improve performance?

Missing baseline : A useful baseline in Table 1 would be the accuracy of the Multimodal LLM in classifying the images. This result will give an insight into the improvements that combining the multimodal LLM and CLIP is bringing.

Comparison to few shot techniques : The authors compare to some few-shot techniques such as CLIP-Adapter, but do not list the accuracies for all datasets. Also it is not clear that given they choose to compare to few-shot techniques, why they choose to not compare with CoOp[1], CoCoOP[2], VDT-Adapter[3], AdaptCLIPZS[4] which are zero-shot techniques trained on "base" classes of a dataset. The last two mentioned works also use LLM generated attributes similar to this method, but have been overlooked.

Computational Overload : Prompting a multimodal LLM twice for each image is highly resource intensive, which brings into question the motivation of using this method in a real-world scenario. Methods such as transduction[5,6], few-shot training[1,2], training on different classes of the same domain[1,2,3,4] seem to be a better choice, therefore comparison with such methods should be conducted.

[1] Zhou, K., Yang, J., Loy, C.C. and Liu, Z., 2022. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9), pp.2337-2348.

[2] Zhou, K., Yang, J., Loy, C.C. and Liu, Z., 2022. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16816-16825).

[3] Maniparambil, M., Vorster, C., Molloy, D., Murphy, N., McGuinness, K. and O'Connor, N.E., 2023. Enhancing clip with gpt-4: Harnessing visual descriptions as prompts. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 262-271).

[4] Saha, O., Van Horn, G. and Maji, S., 2024. Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 17542-17552).

[5] Martin, S., Huang, Y., Shakeri, F., Pesquet, J.C. and Ben Ayed, I., 2024. Transductive Zero-Shot and Few-Shot CLIP. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 28816-28826).

[6] Zanella, M., Gérin, B. and Ayed, I.B., 2024. Boosting Vision-Language Models with Transduction. arXiv preprint arXiv:2406.01837.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: The method introduced in this paper is quite simple. It simple combine the textual and visual inputs together to prompt the accuracy.
This work is lack of theoretical analysis. Why and how much the proposed method can enhance the accuracy?
The experimental section is lack of implementation details.
soundness: 2
strengths: This is a complete work and proposes to enhance the classification accuracy via introducing richer textual information generated via LLMs.
The proposed method employs a simple and universal set of prompts and free from the task of dataset-specific prompt engineering.
The proposed method is experimentally validated.
summary: This paper introduces a method for zero-shot image classification using multimodel LLM. The proposed method first generates comprehensive textual representations from input image, and then utilizes these method textural representations to generate fixed-dimensional features in a cross-modal embedding space, finally these features are used to zero-shot classification.
weaknesses: The method introduced in this paper is quite simple. It simple combine the textual and visual inputs together to prompt the accuracy.
This work is lack of theoretical analysis. Why and how much the proposed method can enhance the accuracy?
The experimental section is lack of implementation details.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - Q1. Have you considered reformulating your evaluation, running experiments with the same backbone model, and comparing the performance with and without your proposed method? Your method is a general plug-and-play approach that could enhance the classification performance of any VLM. I think that empirically showing a robust improvement on many backbones could strengthen your hypothesis.
- Q1bis. It is not clear in Table I which experiments have been recalculated and which backbones have been used for the compared baselines. For example, is CuPL using the same backbone?
- Q2. As a potential future direction, it might be interesting to explore more accessible alternatives to Gemini Pro, such as widely available MLLMs or to compare with other public VLMs. It could even be interesting to see if a simple and more efficient captioner of the input image could improve the performance.
- Q3. My main concern relies on the benefits and improvements arising solely from using Gemini Pro. It would be really insightful to see how Gemini Pro performs on its own for classification. Have you thought about sharing its standalone performance? It might give a clearer picture of its role in the overall method's success.

I would be happy to reconsider my opinion if the authors address my concerns, in particular regarding the unfair comparison using different backbones and the additional computational cost.
soundness: 2
strengths: - The paper is well-organized and clearly written, making the experimental setup easy to follow.
- A key strength of this method is that it eliminates the need for dataset-specific prompt engineering. This simplifies the pipeline and enhances its applicability across various datasets.
- The approach demonstrates significant improvements in zero-shot classification performance across ten benchmark datasets, particularly on ImageNet
summary: The paper introduces a training-free method for improving zero-shot image classification of vision-language models (VLMs) using auxiliary multimodal large language models (MLLMs), i.e. Gemini Pro. With a hand-crafted, dataset-independent prompt, the authors use the MLLM to generate textual descriptions and initial predictions for the input image. The VLM's image and text encoders are then used to extract the input image and the generated text features. These features are fused by averaging. A linear classifier is built using the downstream class names. Classification logits are obtained by multiplying the fused features with the linear classifier. The method achieves an average accuracy gain of 4% across ten datasets.
weaknesses: - One major issue is the use of different backbone models compared to the baselines in the main comparisons (Table 1). The paper employs a CLIP ViT-L/14  for their method (line 243) but compares it to baselines using smaller models like CLIP ViT-B/32, ViT-B/16, and RN50 (and many more), which leads to an unfair advantage. Including smaller backbones for their method, would have enabled a more comprehensive evaluation.
- Additionally, the method’s reliance on Gemini Pro as a multimodal LLM raises concerns about accessibility and generalizability, since not all researchers have access to such powerful models.
- Furthermore, the paper lacks sufficient ablation studies to assess the contributions of different components, particularly the impact of the size of the VLM's backbone model and the role of the MLLM (i.e., the classification performance of Gemini Pro by itself).
- Finally, the greatest weakness is the additional computational cost of using this method in real-world scenarios compared to the baselines. The proposed dataset-independent prompt still requires knowing the training class names and remains image-dependent even at test time. It requires prompting the MLLM to obtain the zero-shot image classification and the description (two MLLM queries). Additionally, the reported results use 50 class descriptions for each label (line 593); these amount to 50,000 more queries (since ImageNet has 1,000 classes) to the MLLM for testing on ImageNet.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. The main motivation of the paper is that "relying solely on visual features of the input images during inference can limit accuracy". However, such a statement is not clear and convincing enough for me to understand the motivation of the paper. I am confused with such a motivation. Could you please make it clear?
2. As stated in the paper, the visual features of the input images are not sufficient enough to fully capture the nuances in the texture descriptions. To make the paper more solid, it would be better to provide some cases here and discuss which kind of features are competent features for the task. 
3. It would be better to provide enough explanations for the notations in the proposed method part. For example,  what does $f_i$ and $f_t$ mean? Current statements are quite confusing.
4. Why is it necessary to generate additional textual-based inputs?
5. There are too many trivial equations in the paper, such as Eq. (2 - 9).
soundness: 2
strengths: - The paper introduces MLLM to perform zero-shot image classification.
- The proposed method is prompt-free compared to previous works.
- The empirical results are good.
summary: The paper mainly focuses on performing zero-shot image classification tasks with multi-modal LLM. Specifically, in this paper, the authors propose a simple yet effective approach to first generate comprehensive textual representations from input images through multi-modal LLMs and then fuse the obtained features to perform zero-shot image classification with a linear classifier. The proposed method is free of prompt engineering. According to the reported results, the proposed method achieves good performance.
weaknesses: - The motivation of the paper is not clear. The concrete information is provided in Q1.
- The descriptions of the proposed method are confusing, see Section 2.
- According to the methodology part, the paper merely fuses the features of three sets of data. The novelty of the paper is extremely weak.

__Minor Concerns:__
- The legend in Fig. 2 is not well defined, which is a little bit confusing.
- The definition of $||\cdot||$ in Eq. (3) is incorrect.

__I have tried my best to read the paper and figure out the motivation and method proposed. However, I failed. The writing of the paper is bad and confusing. The proposed method seems naive. I don't think that this paper reach the bar of ICLR. Thus, I vote to reject.__