Bridge the Modality and Capability Gaps in Vision-Language Model Selection

Authors: chao  yi
yu-hang  he
de-chuan  zhan
han-jia  ye
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:20:20

Abstract: Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zero-shot image classification tasks, a promising strategy is selecting appropriate Pre-Trained VLMs from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the “Modality Gap”—the disparity in VLM’s embeddings across two different modalities, making text a less reliable substitute for images; and the “Capability Gap”— the discrepancy between the VLM’s overall ranking and its ranking for target dataset, hindering direct prediction of a model’s dataset-specific performance from its general performance. We propose VLM Selection With gAp Bridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. It then uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps to obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset’s images. Experiments across various VLMs and image classification datasets validate SWAB’s effectiveness. Code is available at: https://github.com/YCaigogogo/SWAB.

Main paper: 
 introduction: 
 Vision-Language Models (VLMs) [43, 22, 45, 61] have demonstrated impressive image-text matching ability. One notable application of VLMs is zero-shot image classification [43, 37, 14, 35] , where VLMs are leveraged to generate image classifiers using only class names directly. This zero-shot approach has shown considerable success in scenarios with scarce or no training images [33, 18] .Despite the success of VLM in image classification, the performance of a VLM may vary substantially according to the datasets and domains [11] , making it challenging to use a single model to handle all tasks. Fortunately, many open-source VLMs are available [21] , and these VLMs form a vast VLM Zoo. With different architectures, pre-training datasets, or training methods, these VLMs have different strengths. The diverse pre-trained VLMs increase the likelihood of pinpointing at least one VLM that excels in a given target dataset in most cases. 1 To more effectively reuse the VLM Zoo across diverse target tasks and unlock its full potential, we need a model selection method to choose suitable VLMs from the VLM Zoo for the target task. However, in scenarios such as zero-shot image classification, many users might not have labeled images for their target tasks, especially those who are not Machine Learning researchers. They prefer to describe their needs in text and use a Model Search Engine to find the most suitable model. So one solution is identifying the most suitable VLMs in the zoo for a target dataset without access to the dataset's images. This VLM selection is termed as "Language-Only VLM Selection" (LOVM) [64] , and the paradigm is illustrated in Figure 1 .Two key types of information are available for LOVM. One is the target dataset's text data, i.e., names of the target classes and class-related labeled texts generated by LLMs (Details described in Section B.1). The other is the open-source datasets, collected in the form of images with their corresponding class names. Based on these data, the goal is to estimate a VLM's zero-shot image classification capability ranking among the VLM zoo on the target dataset. LOVM encounters two challenges stemming from the inherent heterogeneity in models and datasets. The first challenge is the Modality Gap across different modal features extracted by a VLM. Since the visual and textual features extracted by VLMs tend to cluster into two distinct groups and have gap vectors between them [30] , using text data as image proxies to rank VLMs is inaccurate. The second challenge is the Capability Gap between the VLM's overall ranking and its ranking in the specific dataset. Owing to the VLM's performance variation across different datasets, the VLM's average performance on open-source datasets is hard to reflect its performance on a specific target dataset. Thus, selecting a VLM based solely on its general strength may prove to be a less effective strategy.In this paper, we propose VLM Selection With gAp Bridging (SWAB) to address both gaps. The key idea is to reuse VLMs' statistics from open-source datasets to estimate their statistics on the target dataset, which mitigates the negative impact of these two gaps. In particular, SWAB first uses optimal transport to calculate the transport matrix based on textual similarity between class names of open-source and target datasets. After applying VLMs on open-source datasets to calculate VLMs' statistics, i.e., the class-specific modality gap vectors and performance rankings of different VLMs, SWAB utilizes these statistics to estimate the same type of statistics on the target dataset. After that, SWAB uses the estimated gap vectors to align the features of texts with the features of images from the corresponding category, which bridges the modality gap. Meanwhile, SWAB's estimated VLMs' ranking also improves the prediction of their rankings on the target task, bridging the capability gap. The related work is in the appendix C. The main contributions are:• We analyze two key challenges in LOVM -the modality gap across VLM's modal features and the capability gap between the VLM's overall ranking and its ranking on the target dataset. • We propose SWAB, which utilizes optimal transport to transform useful statistics of VLMs on open-source datasets to the target dataset to bridge two gaps. • Experimental results on a LOVM benchmark composed of a wide range of VLMs and image classification datasets demonstrate the effectiveness of SWAB. 
preliminary: 
 We formally introduce the LOVM setting, a baseline method for LOVM, and analyze the two kinds of gaps in LOVM. We use ∥ • ∥ to represent the Euclidean norm of a vector unless otherwise defined.Zero-Shot Image Classification of VLM. Assume there is a pre-trained VLM f = (f I , f T ) consisting of an image encoder f I and a text encoder f T . Given an image classification dataset T with k T class namesC T = {c T 1 , • • • , c T k T },we input the class names C T (probably with templates like "A photo of {class}") into the VLM's text encoder f T to get the image classifiers { tj } k T j=1 . Then, given a test image x i , we use the image encoder f I to extract its feature xi . Finally, we predict the label via the cosine similarity between the image feature xi and image classifiers { tj } k T j=1 . The class with the highest cosine similarity to the image is selected as the predicted class ŷi . Given xi = f I (x i ), tj = f T (c T j ), Equation 1 describes this zero-shot image classification process:EQUATION)VLM Zoo. In recent years, a large number of (pre-trained) VLMs have emerged. Assume a collection of M VLMs constitute a VLM ZooM = f m = f I m , f T m M m=1. The capability of f m is determined by three key factors: the model architecture (e.g., Transformer [50] , ConvNeXt [32] ), the pre-trained dataset (e.g., LAION-400M [44] , MS-COCO [31] ), and the training method (e.g., contrastive loss [43] , caption loss [60] ). Combinations of these factors result in "good and diverse" VLMs in M. Given a dataset T , it is probable to find a suitable VLM from the VLM zoo with high zero-shot image classification performance on T . T m = 1 |D I T | (xi,yi)∈D I T I (y i = f m (x i , C T )). f m (x i , C T )represents the predicted class with the same manner as Equation 1. I(•) is the indicator function, which outputs 1 if the condition is satisfied, and 0 otherwise. Based on {p T m } M m=1 , we obtain the true ranking of M VLMs r T = [r T 1 , . . . , r T M ] by assigning higher ranking r to models with higher accuracy p. However, in the zero-shot scenario, we can't obtain the test images set D I T in advance. Therefore, the goal of LOVM is to make the predicted ranking rT = [r T 1 , • • • , rT M ] be an accurate estimation of the ground truth ranking r T = [r T 1 , . . . , r T M ] so that the best VLM can be selected. Evaluation of LOVM Methods. We measure the performance of the LOVM algorithm by comparing the ranking similarity between r T and rT . Specifically, we calculate the Top-5 Recall R 5 (ranges from 0 to 1) and Kendall's Rank Correlation τ (ranges from -1 to 1). The larger the better.Non-Learning-based LOVM. There are three main paradigms for LOVM. The first paradigm is to neglect the visual encoder and select VLM solely on texts. In detail, we can utilize ChatGPT [40] to generate auxiliary texts DT based on class names C T of T . More details are described in Section B.1. These class-specific texts act as "image proxies". Then, whether a VLM f m fits T could be measured by transferability metrics, e.g., H-Score [2] and LogME [58] , between the VLM's text encoder f T m and generated text dataset DT . The second paradigm relies on the general performance of a certain VLM f m . We use open-source datasets to measure a VLM's general performance. If f m achieves high zero-shot classification performance over open-source datasets, then it is expected to be competitive on T . These methods assume that a VLM's ranking is relatively consistent across tasks.Learning-based LOVM. The third paradigm is based on the learning process. In detail, the ability of a VLM could be predicted based on a ranker model f R . The input of f R is a vector s T m , depicting the dataset-specific representation of f m on T , while the output of f R is the relative/absolute performance pT m ∈ R of f m on T . The f R could be learned on open-source datasets S [63, 64] . Due to the availability of both class names C S and images D IS in the open-source dataset S such as ImageNet [8] , we can calculate each VLM's representation {s n m } M,N m=1,n=1 and true zero-shot image classification accuracy {p n m } M,N m=1,n=1 . Here N refers to the number of datasets in S. After constructing the train set, the ranker model f R is learned based on the {s n m , p n m } M,N m=1,n=1 :EQUATIONℓ is a loss function that measures the discrepancy between the prediction and the ground truth, which can be Mean Squared Error Loss and Huber Loss, among others. Given T , the learned f R is able to predict the performance{p T m } M m=1 over {s T m } M m=1 via pT m = f R (s T m ). Finally, we can get the predicted VLMs' ranking r based on {p T m } M m=1 . This approach has similarities with metalearning [12, 3] . Meta-learning attempts to use data from multiple datasets to learn a model adaptable to the target task, while Learning-based LOVM employs data from multiple datasets to learn a ranker model for selecting the suitable model from a VLM Zoo for the target task. The representation s T m is one of the keys in this paradigm, and ModelGPT [64] calculates values s T m via the capability of a VLM's text encoder f T m . ModelGPT uses generated text data DT as substitutes for images to calculate some metrics, which measures the zero-shot ability of f m on unseen images by the classification ability of f m on DT :EQUATIONHere Metric i indicates the i-th metrics function such as Top-1 Accuracy and F1-Score. For example, the Top-1 Accuracy s T m,1 could be calculated in a similar manner as Equation 1, with the only difference being that the test samples were replaced with text samples t i instead of image samples x i :EQUATIONBesides, ModelGPT uses some metrics for assessing the features' quality extracted by the VLM's text encoder f T m . More details are in the Section B.2. Moreover, the zero-shot classification performance of f m on ImageNet is also included in s T m as a general ability measure of f m . ModelGPT implements f R as a simple linear model.There are two main challenges that limit the application of the aforementioned paradigms in LOVM. The first is the modality gap across different modalities' features in VLM's feature space, and the second is the capability gap between VLM's overall performance and dataset-specific performance. Modality Gap. As described in Section 2.2, methods like H-Score, LogME, and ModelGPT utilize the ChatGPT generated auxiliary texts DT as image proxies to calculate metrics that measure the zeroshot accuracy on the target dataset T . In other words, the zero-shot classification ability across text and image modalities is estimated by the intra-modality classification ability. The latent assumption is that the generated texts and their corresponding images are closely aligned in VLM's feature space. However, this assumption is difficult to meet [30] , and instances' features are more likely to cluster according to their modalities. In particular, we define the modality gap vector g between the features of an image-text pair (x i , t i ) as g m,i := f I m (x i ) -f T m (t i ). Values in the gap vector are generally not close to zero. We name this phenomenon as Modality Gap in LOVM, which makes the scores on DT hard to reveal the true zero-shot image classification capability of a VLM on a given dataset. We conduct a validation experiment on ImageNet with 43 VLMs. We first generate 50 auxiliary texts per class as DT and then calculate the predicted Top-1 accuracy via Equation 4. Next, we use test images to calculate the VLM's true Top-1 accuracy. The consistency between the predicted Top-1 accuracy and true zero-shot image classification accuracy p m,T is measured by the Kendall Rank Correlation (τ , higher is better) and Mean Absolute Error (MAE, lower is better). It can be observed from the left part of Figure 2 that the predicted accuracy derived from auxiliary texts DT does not closely match the true accuracy, indicating that these generated auxiliary texts in DT are not effective proxies for images.To make the auxiliary texts act as better image proxies, one intuitive idea is to estimate the gap vector g for each image-text pair. Then we can add it to the feature f T m (t i ) of the text t i to eliminate the modality gap, which may lead to more accurate scores s T m,i in Equation 3. However, the gap vector cannot be calculated directly without the target dataset's images. Furthermore, gap vectors for different classes are diverse, so using a shared vector across all datasets may not be a good choice.Capability Gap. To select one VLM from the model zoo given a target dataset, one direct approach is to select the VLM that performs the best on average across multiple datasets. For example, we may first estimate the VLM's zero-shot classification ability on open-source datasets and then select the VLM with the highest performance. The key question is whether a VLM's average ranking on the open-source datasets can reveal its true ranking on the target dataset. Our empirical analyses indicate that there exists a discrepancy between the VLM's overall ranking and its ranking on a specific dataset. We name the discrepancy between the VLM's average ability and its specific ability as the Capability Gap, which results from the fact that a VLM's performance fluctuates significantly across various datasets.To verify the claim, we test 43 VLMs on 23 target datasets provided by [64] and obtain the rankings of each VLM across these datasets. Based on these ranking results, we calculate the average standard deviation and the mean value of the difference between each VLM's maximum and minimum ranking. The experiment process is illustrated in the right part of Figure 2 . We find that the mean difference between one VLM's maximum and the minimum ranking is 38.86. Since the total number of VLMs is 43, such a difference demonstrates that the top-performing VLM in one dataset could likely be among the worst in another.One solution to bridge such a capability gap is to consider the VLMs' ranking on a related dataset. In other words, the ranking of VLMs on datasets from open-source datasets collections that are relevant to the target task may provide more useful insights than a general performance ranking across all tasks. The main challenge is to figure out which open-source datasets are similar to the target dataset and transform the VLM's ranking on these datasets to the target dataset.Step Summary. We emphasize two kinds of gaps in LOVM, i.e., the modality gap across features of different modalities generated by a VLM, and the capability gap between a VLM's overall ranking and its ranking given a specific target dataset. Both two gaps pose obstacles to previous model selection methods, such as LogME and ModelGPT, and degrade their abilities in VLM selection. Moreover, those intuitive approaches to bridge the gaps still face challenges. 
vlm selection with gap bridging: 
 To mitigate the impact of both gaps on LOVM and integrate non-learning-based and learning-based LOVM methods, we propose VLM Selection With gAp Bridging (SWAB). The key idea of SWAB is to bridge modality and capability gaps by utilizing class-level statistics of VLMs from open-source datasets. By measuring the textual similarity between the target dataset's class names and those in open-source datasets, we construct a bridge matrix. Based on it, we estimate the gap vectors between image and text modalities, which rectifies the text-derived scores in ModelGPT. In addition, we predict the VLM's performance ranking for the target dataset based on the bridge matrix and VLM's ranking on the open-source dataset. Both estimated statistics will be used to obtain a more accurate language-only VLM selection. The workflow of SWAB is illustrated in Figure 3 .Benefiting from the open-source datasets, some useful class-level statistics, such as modality gap vectors and zero-shot classification accuracy of a certain VLM, could be calculated, which can help the performance ranking estimation of a VLM on the target dataset. To better utilize these class-level statistics for predicting the corresponding statistics of a VLM on the target task, we introduce semantic relevance information between open-source datasets' classes and target dataset's classes into the statistics reusing process, which is automatically generated through Optimal Transport [7, 42] .Recall that the sets of class names of the open-source datasets and the target dataset areC S = {c S i } k S i=1 and C T = {c T i } k T i=1, respectively. The semantic relevance between two classes could be measured by the textual similarity between their class names. In detail, we use a pre-trained text encoder ϕ (e.g., MPNet [46] ), which extracts text features for these class names, i.e., {ϕ(c S 1 ),• • • , ϕ(c S k S )} and {ϕ(c T 1 ), • • • , ϕ(c T k T )}.Then, we can calculate the cosine similarityϕ(c S i ) ⊤ ϕ(c T j ) ∥ϕ(c S i )∥•∥ϕ(c T j )∥between the text feature of the i-th class in the open-source datasets ϕ(c S i ) and that of the j-th class in the target dataset ϕ(c T j ). The larger the cosine similarity, the more similar the two classes are. After that, we construct the cost matrix in optimal transport via cost ij = 1 -ϕ(c S i ) ⊤ ϕ(c T j ) ∥ϕ(c S i )∥•∥ϕ(c T j )∥ .In practice, we exponentiate each element in cost ∈ R k S ×k T ≥0 using the base e to amplify its differences. We then solve the optimal transport problem with the constructed cost matrix to get the transport matrix γ * . Since optimal transport aims to obtain a transport matrix that minimizes the transmission cost, the transport matrix γ * will reuse more information between semantically similar classes:EQUATIONThe cost matrix quantifies the expense of moving elements between all class pairs, and γ * ∈ R k S ×k T is the transport matrix. OT minimizes the cost indicated by the matrix cost and moves elements from one distribution u to another v. In SWAB, we define u and v as uniformly distributed vectoru = 1/k S ∈ R k S and v = 1/k T ∈ R k T .This indicates that we treat all classes as equally important.We may also incorporate prior knowledge of class importance to define u and v.The solution γ * of the OT problem in Equation 5 could be solved efficiently [13] , and γ * acts as a bridge matrix between open-source datasets' classes and target dataset's classes. Usually, the smaller cost i,j is, the larger the corresponding element γ * i,j obtained by OT, indicating statistics of the i-th class of open-source datasets may help more when we estimate the statistics of the j-th target class.Bridge the Modality Gap. Given the m-th VLM f m in the model zoo, we want to estimate the modality gap g T m,j between the extracted image and text features for the j-th class in the target dataset T to bridge the modality gap. However, in the zero-shot scenario, we can't get the target dataset's images in advance, so we can't directly calculate the gap vectors using image-text pairs. |D I S k | (xi,yi)∈D I S k f I m (xi) ∥f I m (xi)∥ - f T m (c S k ) ∥f T m (c S k )∥ . g S m,T m,1 , • • • , g T m,k T }for the target dataset could be estimated based on the G S m and the transport matrix γ * . If two classes are semantically similar, then we can reuse the gap vector from the similar class. We set the predicted gap vector for the j-th target class ĝT m,j as a weighted sum of G S m , and the weight comes from γ * , which is ĝT m,j = |C T |(γ * :,j ) ⊤ G S m . γ * :,j is the j-th column of γ * . We use scaling factors |C T | to ensure that for each target class, the sum of γ * :,j equals 1. This scale operation has also been used in previous work [55, 56] . After that, we modify the step of ModelGPT in Equation 3, where the metrics over the generated auxiliary texts DT are calculated. We add the gap vector ĝT m,j to the embeddings of the auxiliary texts Dj T from the j-th class in the target dataset:EQUATIONThe modified text embedding tm,i serves as better image proxies. In other words, classification metrics on f T m (t i ) only reveal the discerning ability of the text encoder of f m , which is far from the (cross-modal) zero-shot classification ability due to the modality gap. By bridging such a gap with modified text embedding, classification metrics on tm,i are closer to the classification metrics on images with textual classifier. Therefore, we use { tm,1 EQUATIONEQUATIONNext, we estimate the ranking of a certain VLM f m on the target dataset T . By re-organizing the ranking values in Equation 7, the performance ranking vector off m on all k S open-source classes is r S m = [r S m,1 , • • • , r S m,k S ] ∈ Z k S + .If f m ranks high on certain open-source classes related to the classes in the target dataset, the performance ranking of f m on the target dataset may also be high. Thus, we perform a weighted sum of ranking values in r S m and assign larger weights to those classes related to the target dataset. This can be done by using γ * :EQUATIONElements in rT m ∈ R k T are the predicted ranking of the m-th VLM f m for k T target classes. Since we only need the relative order of ranks, there is no additional scale factor in Equation 8. After that, we average class-specific ranking values in rT m , and use rT ,= Mean rT m to denote the estimated ranking of f m on the target dataset T . In summary, we predict the ranking of models on each class in the target dataset based on their rankings in classes of the open-source datasets, guided by the semantic relevance between the classes. We predict the ranking of models on each category in the target dataset based on their rankings in categories of the open-source dataset, guided by the semantic relevance between the categories. Benefiting from the models' consistently aligned classification performance ranking across similar classes, this approach bridges the capability gap in VLM selection.As is described in Section 3.2, given a target dataset T and a VLM f m , we denote the two performance ranking predictions obtained by bridging the modality gap and capability gap as rT ,(1) m and rT ,(, respectively.rT ,(m is predicted based on ModelGPT with our modified embeddings of the generated auxiliary texts for T .rT ,(m is predicted by the weighted sum of VLM's class-specific ranking values on the open-source datasets. These two predictions, respectively, originate from learning-based and non-learning-based LOVM methods. We ensemble two predictions together and achieve a more accurate model ranking estimation. We utilize the weighted Borda Count to aggregate two rankings:EQUATIONWe set α = 0.5. Ultimately, SWAB determines the final predicted ranking of the VLMs in the VLM Zoo based on rT ,ens m . The pseudo-code of SWAB is listed in Algorithm 1. 
evaluation on lovm benchmark: 
 Setups. We follow LOVM [64] to use its provided VLM Zoo. In addition, to further enhance the diversity of the VLM Zoo, we add some representative VLMs such as BLIP [28] and BEiT-3 [52] to expand the VLM Zoo. Baseline. We select representative methods for each of the three paradigms mentioned in Section 2.2 as our baselines. For the first paradigm, we use four classic model selection methods: H-Score [2] , NCE [48] , LEEP [49] and LogME [58] . For the second paradigm, we use the VLM's ranking on ImageNet (INB) and VLM's average ranking (Avg Rank) on classes of the open-source datasets in the LOVM Benchmark. For the third paradigm, we compare our method with ModelGPT [64] .Evaluations. We use Top-5 Recall and Kendall's Rank Correlation to measure the similarity between the predicted and the ground truth model rankings to evaluate the LOVM method's performance. We also calculate the sum of these two metrics to consider the method's comprehensive capability.Implementation Details. For a fair comparison, SWAB follow ModelGPT [64] to sequentially extract a target dataset from each of the 23 datasets in the LOVM Benchmark and treat the remaining datasets as open-source datasets. Besides, SWAB adopts ModelGPT's approach of adding Gaussian noise to corrupt the target dataset's generated text embeddings. Since LOVM does not provide the specific image data for the 23 datasets, we download these datasets ourselves and adopt their standard data splits. Using the templates provided by LOVM, we construct classifiers and recalculate each VLM's zero-shot image classification accuracy on these datasets. Additionally, we utilize the code provided by LOVM to generate class-related text data using ChatGPT. For H-Score, NCE, LEEP, LogME, INB, and Avg Rank, we follow the practices of previous work and do not add noise to the model's inputs. To ensure reliable results, we conduct ten repeated experiments using random seeds from 1 to 10 and report the mean value and standard deviation of ModelGPT's performance and SWAB's performance in Table 1 .Results Analysis. From Table 1 , we can draw the following conclusions: (1) Metric-based nonlearning model selection methods such as LogME show poor performance on the LOVM Benchmark. This is primarily because such algorithms rely on the target dataset's images, thus the modality gap has a greater negative impact on them when using generated text data as a substitute for images.(2) Using open-source datasets is helpful for LOVM. We find that using open-source datasets in a non-learning way (e.g. INB, Avg Rank) or a learning way (e.g. ModelGPT) all helps LOVM, since their performance significantly surpasses that of methods not utilizing open-source datasets (e.g. LogME). ( 3) Despite leveraging more open-source datasets, the performance of Average Rank is worse than INB. This confirms our analysis of the Capability Gap, which suggests a discrepancy between the average ranking of a VLM and its ranking on a specific dataset. (4) Our SWAB achieves the best performance across all evaluation metrics. Notably, our final performance of R 5 + τ (0.822) represents a significant improvement of 14.8% over the original SoTA method ModelGPT (0.716). We conduct an ablation study to demonstrate that bridging the Modality Gap and Capability Gap are both essential for SWAB. Table 2 presents our experiment results, from which we can observe that SWAB achieves the best performance across all metrics when both gaps are bridged simultaneously. The ablation study confirms our analysis.Will Bridge the Capability Gap Be Beneficial for VLM Selection? We compare the LOVM performance directly using the VLM's average ranking on each class of open-source datasets and weighted-sum ranking based on transport matrix γ * . The results are shown in Table 3 . We can find that utilizing class relevance to bridge the Capability Gap is beneficial for VLM's Model Selection.Will Bridge the Modality Gap Be Beneficial for VLM Selection? To eliminate the interference of other factors, we solely utilize the learning-based predicted rankings rT ,(1) m in SWAB, and the input to the ranker model f m only consists of metrics calculated on the generated text data DT , which serves as substitutes for images. In this way, the method's performance depends solely on the quality of the generated text data DT . From the Table 4 , we can find that the generated text data DT become better substitutes for image data after bridging the Modality Gap. 2) Use the class-level mean gap vector calculated on the corresponding class's image-text pairs. We hope that the gap vectors are as close to each other as possible so that their mean vector can substitute well for the whole set. Based on this idea, we calculate the statistics of the gap vectors within a dataset and within each class. We calculate three metrics which include:(1) the standard deviation of these gap vectors' magnitude; (2) the mean cosine similarity between these gap vectors and their corresponding mean gap vectors; and (3) the standard deviation of these cosine similarities. These metrics reflect the consistency of the gap vectors. Table 5 shows the results. From the Table 5 , we can find that the class-level gap vectors tend to be more consistent, which inspires us to use the class-level mean gap vectors. We also compare the results of SWAB-M on the LOVM Benchmark using the dataset-level mean gap vectors and the class-level mean gap vectors, respectively. The implementation details are the same as Table 2 . Table 6 shows the results, which verifies that using the class-level mean gap vectors is a better choice. 
conclusion: 
 We analyze and address two key challenges in Language-Only VLM Selection (LOVM), which are VLM's modality gap across different modal features and VLM's Capability gap between its overall and dataset-specific rankings. Our key insight is that we can reuse the model's useful statistics on open-source datasets to help the model selection on the target dataset. SWAB utilizes a transport matrix between classes of the target dataset and open-source datasets to transfer VLM's class-specific modality gap vectors and class-specific rank from open-source datasets to the target dataset, which mitigates the negative impacts of these two gaps. Experiment results on the LOVM benchmark show the superiority of our method.In the Appendix, we introduce more details about the LOVM Benchmark as well as our extensions to it. Besides, we introduce ModelGPT's implementation and our SWAB's implementation. We also provide more experimental results of SWAB. The structure of the Appendix is as follows:• In section A, we introduce the relevant information of the 43 models and 23 datasets used in our experiments. We also introduce the evaluation metrics used in LOVM Benchmark [64] . • In section B, we introduce the metrics used in Equation 3.• In section C, we introduce the related work of the paper.• In section D, we provide some details on SWAB's implementation.• In section E, we provide more experimental results of SWAB. 
 
 Appendix: LOVM Benchmark [64] consists of 35 pre-trained VLMs and 23 datasets, with a total of 35×23 = 805 evaluations. To further enhance the diversity of the VLM Zoo, we add some representative VLMs such as BLIP [28] and BEiT-3 [52] to expand the VLM Zoo. The final VLM Zoo contains a total of 43 models, with a total of 43 × 23 = 989 evaluations. For each evaluation, LOVM provides the VLM's zero-shot image classification accuracy on the corresponding dataset. Therefore, we can get the ground truth performance ranking of 43 VLMs on the 23 datasets.To cover as many types of models as possible, the LOVM Benchmark uses OpenCLIP library [21] to get diverse VLMs. These VLMs differ from each other in terms of the model architecture (ResNet [17] , Transformer [50] , ConvNext [32] ), the pre-trained dataset (OpenAI's Data [43] , LAION 2b [44] ), the training method (loss function/hyperparameter/data augmentation) and so on. Table 7 displays the relevant information of each VLM. We further add BLIP and BEiT-3 to the original VLM Zoo. It should be noted that in addition to the image-text pair data listed in the Table 7 , BEiT-3's pre-training data also includes unimodal image dataset (ImageNet-21K) and text datasets (English Wikipedia, BookCorpus, OpenWebText, CC-News, Stories). The diversity of these VLMs ensures that the experimental results calculated on them can reflect the performance of the VLM model selection algorithm in real-world situations.To cover as wide a distribution of image classification tasks as possible, the LOVM Benchmark collects 23 diverse datasets. These datasets differ from each other in terms of the number of categories, category semantics, image domains, and so on. Table 8 displays the relevant information of each dataset. The diversity of these tasks ensures that the experimental results calculated on them can reflect the performance of the VLM model selection method in real-world situations.In LOVM, our aim is to maximize the rank similarity between the prediction of VLMs' ranking rT = {r T m } M m=1 and VLMs' ground truth ranking r T = {r T m } M m=1 on the target dataset, especially the rank similarity of the top 5 VLMs in rT and r T . This is because we tend to focus only on whether those appropriate models can be chosen. To ensure fair comparability, we follow the evaluation metrics used by LOVM [64] to assess different model selection methods and directly utilize the code provided by LOVM [64] to calculate these metrics: [53] 397 scene und. natural image Country211 [43] 211 geolocation natural image Stanford Cars [25] 196 classification natural image Flowers102 [39] 102 classification natural image CIFAR100 [26] 100 classification natural image DTD [5] 46 classification textural image RESISC45 [4] 45 classification satellite images GTSRB [47] 43 classification natural image Oxford Pets [41] 37 classification natural image VOC2007 [10] 20 classification natural image STL10 [6] 10 classification natural image EuroSAT [19] 10 classification satellite images MNIST [27] 10 classification hand-writing SVHN [38] 10 OCR natural image CLEVR-C [23] 8 object counting natural image CLEVR-D [23] 8 distance est. natural image FER2013 [16] 7 fac. exp. rec. natural image DMLab [62] 6 distance est. synthetic Retinopathy [24] 5 classification retina scan KITTI [15] 4 distance est. natural image PCam [51] 2 classification histopathology Rendered SST2 [43] 2 OCR text imageEQUATION• Kendall's Rank Correlation (τ ) -Kendall's Rank Correlation τ measures the ranking consistency between two ranking lists. We follow LOVM [64] to focus on the VLMs within the intersection of the top 5 VLMs in rT and r T and use τ to evaluate a model selection method's capability.ModelGPT is a method proposed by LOVM [64] . In this section, we introduce the metrics that ModelGPT used in Equation 3.ModelGPT [64] utilizes ChatGPT [40] to generate auxiliary text data by designing prompts to query ChatGPT. This extra text data mainly includes the Captions Dataset and the Synonyms Dataset.Captions Dataset. ModelGPT uses the following prompt to guide LLM to generate realistic and confusing text data corresponding to the user-provided classes. The reason for requiring ChatGPT to generate confusing texts is to increase the classification difficulty of the text data, thereby enhancing its ability to distinguish the performance of different models.Generate long and confusing image captions for the {domain} domain, which will be used to evaluate a Vision-Language Model's {task} performance. Generate 50 captions for {classname}:We show some generated auxiliary text examples. For example, in the category of dog, one of the text samples generated by ChatGPT is "An adorable dog perfect for cuddles and playtime." ModelGPT collects the results from this prompt to form the captions dataset, D cap .Synonyms Dataset. ModelGPT uses synonyms to evaluate VLM's text encoder. For example, we expect an excellent VLM to extract similar embeddings for the words "chair" and "seat". The prompt to guide LLM to generate synonyms is as follows.Please list the superclasses/synonyms for {classname}. For example: chair: [furniture, seat, bench, armchair, sofa] {classname}:ModelGPT collects the results from this prompt to form the synonyms dataset, D syn .ModelGPT uses six metrics for model selection, which can be divided into Text Classification scores and Dataset Granularity scores. Text Classification scores include the Text top-1 accuracy score and Text f1-score. While Granularity scores include the Fisher criterion, Silhouette score, Class Dispersion score and Synonym Consistency score. Here we focus on introducing the various metrics included in the Granularity scores. We refer to the relevant content in LOVM.Fisher Criterion ϕ fisher . The Fisher score measures the closeness of VLM's text classifier to one another. Equation 12shows the calculation process of it where ti is the text classifier of the i-th class derived using the prompt ensemble strategies proposed in [43] , θ(•, •) is a function that calculates the cosine similarity between two vectors, and |C| is the number of classes.EQUATION)Silhouette Score φ sil . The Silhouette Score measures the separation of different-class samples in the caption dataset D cap . To calculate it, ModelGPT averages the cosine similarity of captions to the nearest other class's classifier by:EQUATION)where ti is the text classifier of the i-th class derived using the prompt ensemble strategies proposed in [43] , θ(•, •) is a function that calculates the cosine similarity between two vectors, and |C| is the number of classes. D cap [j] k representing sample k of class j in the caption dataset D cap . There is a total of N such samples for each class.Class Dispersion Score ρ disp . Class Dispersion Score quantifies the degree of same-class tightness or data cone radius, which is calculated using the following Equation:EQUATION)The definitions of all symbols in Equation 14 are consistent with those in Equation 13.Synonym Consistency Score γ syn . Synonym consistency allows us to evaluate the degree of content shift between the VLMs' pre-training and target dataset. The calculation process is shown as follows:EQUATION)The definitions of ti , θ( Vision-Language Models. Vision-Language Models represent a class of multimodal models adept at correlating textual and visual information. These VLMs can comprehend rich semantics and possess strong generalization capabilities. Hence, they are often used in data-limited scenarios [54, 29] . Many VLMs are pre-trained or fine-tuned on extensive text-image pairs using loss functions such as contrastive loss, endowing them with powerful text-image matching capability. Prominent VLMs include CLIP [43] , ALIGN [22] , FLAVA [45] , Florence [61] , and CoCa [60] . These VLMs possess robust zero-shot image classification capabilities [43] , which enables its widespread application in tasks characterized by long-tail distributions or those where collecting substantial training data is challenging, such as medical image analysis. Some work [37, 57] has also shown that by incorporating external knowledge, the zero-shot capabilities of CLIP can be further enhanced. In recent years, the number of open-source VLMs has been increasing [21] . Previous work [64] has pointed out that different VLMs possess varying image classification capabilities. This indicates that the performance of VLMs can vary significantly across different tasks and domains. These models with diverse capabilities constitute a VLM Zoo rich in knowledge. This VLM Zoo enables us to utilize different VLMs for various classification tasks, thereby changing the paradigm of using a single VLM to complete diverse classification tasks. This paper focuses on selecting the most suitable VLM for the target task from the VLM Zoo.Model Selection. The essence of the model selection problem lies in measuring the transferability of the model to the target task. Previous model selection methods [48, 49, 58, 59, 9, 20] evaluate the PTM's transferability by performing a forward pass of the PTM on the target task's data and calculating the metric of transferability based on the forward pass's result. For example, H-Score [2] , NCE [48] , LEEP [49] , LogME [58] estimate transferred log-likelihood, negative conditional entropy, log expectation, marginalized likelihood to obtain proxy metric of transferability, respectively. Some new methods, such as Task2Vec [1] and Model Spider [63] , generate representation vectors for both the models and the tasks and measure the transferability of the model to the task by calculating the similarity between these vectors. However, VLMs are typically used in zero-shot or few-shot scenarios, where we are unable to obtain a large amount of data for the target task in advance, making traditional model selection approaches unsuitable for VLMs. Additionally, previous methods have mainly focused on single-modal models, overlooking the characteristics of VLMs. Therefore, in this paper, we concentrate on designing model selection algorithms that are suitable for scenarios with limited data and take into account the characteristics of VLMs.In this section, we provide some details on the implementation of SWAB, which are not mentioned in the main text due to space constraints.The statistics of classes unrelated to the target class are generally not valuable for reuse. Meanwhile, when the number of classes |C S | in open-source datasets S is large, solving the optimal transport problem in Equation 5 can be time-consuming (as current optimal transport toolkits generally compute via CPU). To reduce the runtime of optimal transport, we can first filter the classes C S . Consider that only statistics of classes relevant to the target dataset are helpful. Therefore, we can filter out the classes in C S that are irrelevant to the target dataset T based on the class-level textual semantic similarity between the open-source datasets' classes and the target dataset's classes. This process is shown in the following Equation:EQUATIONEQUATIONHere S i,: refers to the i-th row of the semantic similarity matrix calculated using Equation 16, which represents the vector formed by the similarity between the i-th class c S i in C S and each class C T = {c T 1 , • • • , c T k T }Partial optimal transport extends the optimal transport framework, enabling the selective transfer of elements from a source to a target distribution, rather than moving all elements. Its optimization problem is defined as in Equation 18. Here mass refers to the total amount of mass actually be transferred. We set mass = 0.9 in our implementation.EQUATIONWe found that when using Equation 8 to bridge the Capability Gap, the transport matrix γ * obtained using partial optimal transport yields better results than the one obtained using the original optimal transport via solving the Equation 5. Therefore, in our implementation, we use the transport matrix derived from partial optimal transport to bridge the capability gap. This also indicates that when estimating VLM's statistics on the target dataset, different types of statistics have different preferences for the estimation methods used. This variability is worth further investigation.When bridging the Modality Gap as described in Section 3.2, we find that applying z-score normalization to the text and image features used in this process yields better results. Therefore, in our implementation, we normalize the features of all text and image samples during the modality bridging process using the following Equation:EQUATIONHere x ∈ R d represents the image sample's or text sample's feature, while µ ∈ R d and σ ∈ R d are calculated using the features of all samples of the same modality within its respective dataset. for each VLM as the VLM's overall prediction ranking on the target dataset. In this section, we provide more experimental results of SWAB.In Section 2.3 and Figure 2 , we analyze whether generated text data can act as good image proxies.Our conclusion is that due to the Modality Gap, text samples cannot directly serve as an effective substitute for images in model evaluation. To demonstrate that our method SWAB can bridge this Modality Gap and thereby make text samples a better substitute for images, we conduct the following experiment.From the Figure 4 , it is evident that the predicted model accuracy calculated using the modified text samples is closer to the true model accuracy compared to that calculated with the original text samples. This suggests that bridging the Modality Gap leads to better image proxies.We use ImageNet as our dataset. First, we employ the method introduced in Section 3.2 to predict the gap vectors for each class of the target dataset based on gap vectors calculated on open-source datasets. Then, we add the corresponding class's predicted gap vectors to the generated text data of ImageNet to bridge the modality gap. Finally, we calculate the zero-shot classification accuracy of different models on these modified text data. To measure the consistency between the predicted Top-1 = .= .Text Samples Modified Text Samples = .= .Figure 4 : Comparison of the consistency metrics between the accuracy calculated using text data before and after bridging the gap and the model's true accuracy. After bridging the modality gap, the text data act as better substitutes for image data to evaluate the model's performance.accuracy and the true image classification accuracy, we calculate the Kendall Rank Correlation (τ , higher is better) and Mean Absolute Error (MAE, lower is better). We compare the consistency metrics of text data and modified text data. It can be observed that the consistency metrics of modified text data are better, which proves our method can reduce the gap between the generated text data and the image data.E.2 Analysis of the Modality Gap in BLIP and BEiT-3We expand the VLM Zoo provided by LOVM by adding various variants of BLIP and BEiT-3 to enhance the diversity of the VLM Zoo. We find that the zero-shot image classification performance of the base models of BLIP and BEiT-3 is poor. Therefore, we use the retrieval version of the models provided officially. These models are obtained by fine-tuning the base models on COCO and Flickr30k through contrastive learning, thereby possessing better zero-shot image classification performance. During the pre-training of BLIP and BEiT-3, they may have used multiple loss functions in addition to the contrastive loss (BLIP), or perhaps they did not use contrastive loss at all (BEiT-3). Therefore, the existence of the Modality Gap in the feature spaces of these two models requires further investigation.We use image samples from the STL-10 dataset and class-related text samples generated by ChatGPT.For each modality, we randomly extract 200 samples and calculate image-to-image, text-to-text, and image-to-text similarities, and plot histograms. We also observe the presence of Modality Gaps in different models through UMAP visualization [36] . Figure 5 and Figure 6 show the experiment result.Through our experiments, we verify that the feature spaces of the BLIP and BEiT-3 retrieval models still exhibit the Modality Gap phenomenon. We have provided experimental results on the VLM Zoo originally provided by LOVM [64] . This VLM Zoo contains 35 VLMs. Table 9 shows the experimental results, and we can see that SWAB achieves the best performance across all evaluation metrics. We present the per-dataset performance comparison between our methods SWAB and ModelGPT on various datasets of LOVM benchmark in Table 10 . ModelGPT 0.67 0.67 0.00 0.00 -0.33 0.00 0.00 0.67 1.00 0.00 0.67 1.00 0.67 0.00 0.00 0.00 0.00 0.00 0.67 0.00 -1.00 0.33 1.00 0.261 SWAB 0.67 0.91 0.00 0.33 0.00 0.00 0.00 0.55 1.00 0.33 0.33 1.00 0.89 0.00 0.00 0.00 0.33 0.00 0.67 0.00 -0.82 0.82 0.33 0.320Throughout this paper, the term "VLM" specifically refers to a pre-trained VLM. 


Decision: The paper tackles the problem of selecting a visual language model (VLM) from a ModelZoo for a target task using only the labels of the dataset. Two challenges are addressed in the language-only VLM selection (LOVM) problem: the modality gap and the capacity gap. The modality gap refers to the difference between the label (or text) domain and the full task, while the capacity gap refers to the generalization ability of the selector across datasets. Results for model selection are presented on a wide range of datasets and are compared with baselines and prior work, such as ModelGPT.

The reviews for the paper are mostly positive (Accept, Borderline Accept, Borderline Reject, and Borderline Accept).

Reviewer qHTS finds the problem to be novel and interesting with practical value. However, they note that the comparisons with ModelGPT are inconsistent. The rebuttal notes that this might be due to differences in random seeds and other experimental details not described in the paper or the code from the authors. Another difference is due to a larger model zoo. This was clarified in the rebuttal, where results using an identical model zoo from the ModelGPT paper were shown, and the method outperformed the baselines. However, the differences in the numbers seemed fairly large despite the authors' best efforts.

Reviewer NUg2 found the paper well-motivated and had questions regarding some experimental settings. However, their final score remained unchanged after the rebuttal, with a borderline accept rating.

Reviewer omtt was the most critical and questioned the applicability of text-only model selection. They also raised the question of how the model might handle tasks beyond classification. While the rebuttal addressed some concerns, the final rating was a borderline reject.

Reviewer mpun found the paper to be novel and noted that this is the first time LOVM has been applied. However, the Area Chair (AC) notes that LOVM has been proposed in prior work. The reviewer also found the approach well-motivated, and the experiments to be extensive. The main weaknesses were addressed in the rebuttal, and the score was raised to a borderline accept.

The AC read the paper, reviews, rebuttal, and discussions, and generally agrees with the positive sentiment about the paper. However, there are some issues the authors should address in the final version. The first is to make the comparison with ModelGPT easier. Perhaps the authors could report results using the identical model zoo and present a standalone evaluation with identical text. These experimental differences should be documented in the paper to facilitate future comparisons. Second, the AC found the related work section of the paper to be lacking. Some works, such as Task2Vec, are not cited, where the terms domain2vec and label2vec are introduced. Label2vec presents a way to represent tasks using only some statistics of the labels, which is related to this paper.

Task2Vec: Task Embedding for Meta-Learning, Achille et al., ICCV’15

Reviews: review: paper_summary: paper_summary: This article focuses on selecting the best model from a visual-language multimodal model zoo for specific downstream tasks without having images of those tasks. It provides a detailed analysis of two challenges faced in this problem — Modality Gap and Capacity Gap. This paper proposes a method called SWAB, which leverages the statistical measures of VLM on open-source tasks based on the category similarity between historical open-source tasks and target tasks, to mitigate the negative impacts of these gaps on Language-Only VLM model selection. Experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.The performance of ModelGPT is slightly inconsistent with the results in its paper. What are the reasons for this？
2.The method of setting the \alpha value in formula 9 is not stated. Is the same \alpha used for all data sets? What is the exact value?
questions: questions: Refer to the weakness.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes SWAB the modificaiton of LOVM to mitigate the negative impact of two gaps of LOVM: the modality gap and capability gap. The resutls show the effectiveness of SWAB.
strength_weakness: strength_weakness: 1. For the modality gap, the SWAB using open-source datasets, could you please provide how many image samples used in the experiment. And could you please provide the accuracy vs number of open-source image samples?
2. The SWAB seems like a learnig-based model, could you please add the ablition study for no learning based SWAB.
questions: questions: 1. Why adding Gaussian noise to the target generated embedding?
2. How to select weighted parameter $\alpha$ in Eq.9
3. The Auxiliary texts is generated by ChatGPT, could you please test the open-source LLM like LLama2?
limitations: limitations: No.
paper_summary: paper_summary: With the popularity of Vision Language Model (VLM) research in recent years, many versions have emerged, forming the VLM Zoo. This paper aims to select the most appropriate pre-trained VLM from the VLM Zoo, relying solely on texts of the target dataset without access to images. Two challenges are analyzed, namely, “Modality Gap” and “Capability Gap”. One VLM selection method (SWAB) is proposed: first calculate optimal transportation matrix to capture the relevance between open-source and target datasets; then use this matrix to transfer useful statistics of VLMs from open-source to target dataset for bridging two gaps and enhancing the VLM’s capacity estimation. Experiments are carried out across some VLMs and classification datasets to validate the effectiveness.
strength_weakness: strength_weakness: [-] Problem rationality. Although each member in the VLM Zoo has their own strengths, relying solely on textual descriptions to find the most suitable VLM (LOVM) seems too demanding. And language is usually compact, which can easily confuse concepts. Why cannot we use one combination of images and texts (or even text-image pairs) for VLM selection, as images are very easy to obtain in real-world scenarios? 

[-] Selection strategy. Given that the members in VLM Zoo have different strengths and abilities, it means that they have somewhat degree of complementarity. One natural idea is to follow ensemble learning, that is, vote on the results of each member, or select the top-k VLMs’ results for complementary combination. Why only choose the most suitable one? Please provide more comparisons and explanations. 

[-] Generalization. This paper carries out experiments on many datasets, but all for image classification. Since VLMs excel in handling various tasks, such as detection, segmentation, captioning, and VQA, the generalization on these tasks is still unclear.
questions: questions: [-] Better visualization. For Fig. 3, there are too many symbols and formulas, increasing difficulty in understanding (the significance of using images is to reduce understanding costs). To this end, the reviewer believes that this paper can be polished better.
limitations: limitations: This paper does not discuss the limitations of the work. However, the reviewer seems to find some limitations, such as the unrealistic assumptions in settings and the ability to only handle classification tasks. Please refer to Weaknesses for more details.
paper_summary: paper_summary: This paper considers a zero-shot image classification strategy by selecting the most appropriate Pre-Trained VLM from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. Two challenges, i.e., the “Modality Gap” across two different modalities, and the "Capability Gap" between the VLM’s overall ranking and its ranking for target dataset, hinder the appropriate selection of the different VLMs. To address the challenges, this paper adopts a transportation matrix to capture the relevance between open-source and target datasets, and also uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset. Extensive experiments on image classification datasets demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The main weakness is that the proposed VLMs selection strategy relies on Open-Source Datasets’ Data, which restricts the practicability of the proposed method. For the zero-shot image classification task, using the aggregated knowledge from multiple VLMs is ok, but using open-source datasets may violate the zero-shot data restrictions, or even lead to benchmark leakage. In addition, the impact of the open-source datasets is not analyzed. For example, if the categories in the open-source datasets are very different from the ones in the target zero-shot classification tasks, whether the performance of the proposed method is still remarkable?
questions: questions: A comprehensive analysis on the impact of the open source datasets may improve the rationality of the proposed method for addressing the zero-shot image classification task.
limitations: limitations: This paper does not include the limitation discussion. Practically, at least the issues raised in the weakness section should be considered if they cannot be addressed appropriately in current version.

Comments: title: Thanks, but inconsistent numbers
Impact of the new members in the model zoo
Response to Rebuttal
comment: Dear Area Chair,

Thank you for your attention to our work. We think that ModelGPT's $R_5$ could indeed be lower than ImageNet. This is because, within LOVM, it utilizes the actual performance of various models on open-source datasets as a training set's label to train a Linear Regression Model  to predict performance or ranking of the VLMs (for details, see Section 3 of LOVM). The inputs of this Linear Regression Model include the current model's score on the ImageNet Baseline and some performance metrics calculated on text samples generated by LLM (such as text-acc1 and text-f1, details in Section 3.2 of LOVM). Therefore, **the performance prediction of the Linear Regression Model for the current model actually depends on the weighted sum of scores on the ImageNet Baseline and the performance metrics calculated on text samples produced by LLM.**

However, as analyzed in our paper regarding the Modality Gap, **due to the existence of a modality gap between image features and text features in the feature spaces of many VLMs, text samples cannot directly serve as a good substitute for image samples**. Therefore, **we cannot expect that the performance metrics calculated directly on text samples generated by LLMs (such as text-acc1 and text-f1) will positively correlate with the actual performance of the current model,** which in turn affects the performance of the linear regression model in LOVM that uses metrics like text-acc1 as inputs. We believe there are two pieces of evidence that can substantiate this idea:

1. In Figure 5 of our paper, we analyzed the consistency between the text top-1 accuracy calculated on text samples and the actual accuracy of the model before and after using our method to bridge the modality gap. The experimental results show that before bridging the modality gap, the consistency between text top-1 accuracy and the model's actual accuracy was poor. However, **our method significantly improves the consistency between the two by bridging the modality gap, demonstrating the effectiveness of our approach.**

2. We observed that in the original paper of LOVM, the improvement in the $R_5$ metric when using ModelGPT compared to the ImageNet Baseline is very slight (0.461 vs 0.452). **This modest increase could likely disappear due to the randomness of the methods mentioned in our previous response** (i.e., this improvement might fall within the error bars, so after changing to a new random seed, this improvement might disappear). This indicates that the introduction of the performance metrics calculated directly on text samples generated by LLMs provides minimal help. 

Thank you very much for your attention and suggestions regarding our work. We hope our responses address your concerns.
Thank you again for the quick responses. The reasons you provided make sense as potential sources of discrepancy between the results. It is unfortunate that the authors of LOVM have not released a reproducible version of their code. However, please note that with your implementation, ModelGPT performs no better than the ImageNet baseline according to the R5 metric.
Dear Area Chair,

Thank you very much for your interest in our work. **First, we want to emphasize that we used the code provided by LOVM to ensure consistency with the evaluation methods of their benchmark.** We think the slight differences between the ModelGPT results we reported and those in the LOVM paper can be attributed to two main reasons:

1. **The use of different random seeds.** As we mentioned in our response to Reviewer NUg2's Q4, in both the LOVM paper and ours, we add Gaussian noise to the text features to simulate the variance in image samples because different images can correspond to the same caption. This process introduces randomness. We noticed that in the code provided by LOVM, no random seed was specified, and they also did not report the random seed used in their experiments. To ensure the reliability of our experimental results, we repeated the experiments using 10 different random seeds (from 1 to 10) and reported the mean and standard deviation of the results. Therefore, the difference in random seeds may lead to slight variations between our results and theirs.

2. **The text generated by the LLM used may differ.** In the GitHub repository provided by LOVM, the specific text collection generated by ChatGPT was not directly provided; instead, they included a Python script to generate the text collection using ChatGPT. In our experiments, we used this provided Python script to generate the text collection with ChatGPT. Considering the inherent randomness in LLM-generated responses, the text collection we used might differ slightly from the one used by LOVM. This variability in LLM-generated responses can also contribute to the slight differences between our results and theirs.

We emphasize that in our experiments, we used the same evaluation methods as LOVM. Furthermore, when comparing ModelGPT with our approach, we ensured that the data and random seeds were completely consistent to guarantee the comparability of the experimental results.

Thank you very much for your attention and suggestions regarding our work. We hope our responses address your concerns.
Thank you very much for your response. We would like to provide the following further clarifications regarding your concerns:

**Regarding Q1**: We believe that text with different semantics can serve as a substitute for images that match its semantics. In fact, several studies have recognized and validated this perspective. For instance, in [1], the authors state that “collecting a large enough test set that covers different image distributions is a fundamental challenge.” Therefore, they use diverse text generated by large language models as a supplement to image training samples to debias vision models and improve performance (see Figure 1 and Section 2.2 in [1] for more details). Based on our work and the successful implementation of this approach in previous studies, we believe that considering the high noise characteristics of image or image-text pairs collected online, as mentioned in numerous papers [2,3,4], the practice of "using text with different semantics as a substitute for images that match its semantics" is both reasonable and valuable.

**Regarding Q2**: Firstly, we emphasize that once the model rankings are obtained, these rankings can enhance the performance of an ensemble (e.g., using the top three models most suitable for the current task in an ensemble will generally outperform an ensemble formed by randomly selecting three models from the model zoo). Secondly, as we mentioned in our rebuttal, using multiple models in an ensemble requires significant computational resources. Therefore, in many cases, a common practice to balance performance and efficiency is to select the best single model. As a result, our paper primarily focuses on the scenario of using a single model.

[1] Zhang et al. Diagnosing and Rectifying Vision Models using Language. ICLR 2023.

[2] Radford et al. Learning transferable visual models from natural language supervision. ICML 2021

[3] Li et al. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS 2021.

[4] Li et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML 2022.
Thanks for the quick response. Adding new models makes sense, but makes the comparison harder. Also ModelGPT reports:
INB+G+C 0.461 (R5) 0.223 (\tau) for top-1 accuracy according to Table 2 in their paper, which are a fair bit higher than your table above. Could you clarify how the results were obtained?
Thank you very much for your acknowledgment and efforts.
Thank you very much for your acknowledgment and efforts.
Thank you very much for your acknowledgment and efforts.
Dear Area Chair,

Thank you very much for your attention to our work. In fact, we initially conducted our experiments on the 35 models provided in the LOVM Benchmark. However, we later received some suggestions from other researchers to include a greater variety of vision-language models (e.g. BLIP and BEIT-3) to enhance the diversity of models in the benchmark, thereby increasing the credibility of the evaluation results. Therefore, we expanded the variety of models based on the LOVM Benchmark. To address your concerns, we are also providing our experimental results on the original 35 models from the benchmark.

| Method                | ModelGPT | SWAB (Ours) |
|-----------------------|----------|-------------|
| $R_5$                  | 0.452    | **0.501**       |
| $\tau$                   | 0.201    | **0.291**       |
| $R_5 + \tau$            | 0.653    | **0.792**       |
Another difference might stem from adding new members in the Model Zoo. Could you quantify the impact of this? For example, what is the performance of your approach using same Model Zoo in the LOVM paper?
Thanks to the authors for their efforts during the rebuttal. I have read all the comments from other reviewers and the authors' responses. Some of my concerns have been resolved, such as Q3. Yet, I stand by my opinion that “make the diverse text from LLMs substitute for diverse images” is a bit idealistic, as compact texts cannot cover pixel-level vision. Besides, for Q2, ranking VLMs does not seem to imply the effectiveness of an ensemble. Thus, I decide to slightly raise my score to "borderline reject".
Dear Reviewer omtt,

We have responded to your comments in detail. As the discussion period will end in less than 10 hours, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors
Thanks for your response and addressing my concerns. I will keep my score.
Thank the authors for the Rebuttal. I read the Rebuttal and other reviewers' comments. Since the authors have addressed my concerns, I would like to raise my score to boardline accept.
Dear Reviewer mpun,

We have responded to your comments in detail. As the discussion period will end in less than one day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors
Dear Reviewer omtt,

We have responded to your comments in detail. As the discussion period will end in less than one day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors
Dear Reviewer NUg2,

We have responded to your comments in detail. As the discussion period will end in less than one day, we would like to ask whether there are any additional concerns or questions we could address.

Thanks very much for your effort!

Best regards,

Authors
Dear reviewer mpun,

We thank you sincerely for your time and effort in reviewing our manuscript and providing valuable suggestions. As the author's response period is nearing its conclusion, we kindly remind you to review our response.

We have provided detailed responses to your questions and hope that they adequately address your concerns. If you need further clarification or have any other questions, please feel free to discuss them with us! We are more than willing to continue our communication with you. We would greatly appreciate it if you would update the rating by taking our responses into account.
Dear reviewer omtt,

We thank you sincerely for your time and effort in reviewing our manuscript and providing valuable suggestions. As the author's response period is nearing its conclusion, we kindly remind you to review our response.

We have provided detailed responses to your questions and hope that they adequately address your concerns. If you need further clarification or have any other questions, please feel free to discuss them with us! We are more than willing to continue our communication with you. We would greatly appreciate it if you would update the rating by taking our responses into account.
We appreciate your response and are happy to address your issue. Here is our reply:

* Q1: Is the parameter $\alpha$ sensitive to different datasets, and what is the best strategy for selecting it? Provide some analysis on this.
* A1: Firstly, we emphasize that we use the same $\alpha$ across all datasets. The reasons for this are: (1) we found that using the same $\alpha$ achieves good performance on most datasets and (2) applying the same $\alpha$ to all datasets is a simple and effective approach. 

    Regarding the strategy for selecting $\alpha$, our experiments revealed that our method's performance is fairly robust to different $\alpha$ values, so we chose an $\alpha$ that performs well on our benchmark. To provide relevant analysis on this point, we varied the $\alpha$ values and demonstrated our method's performance metrics on the LOVM Benchmark.

    | $\alpha$   | 0.2   | 0.3   | 0.4   | 0.5   | 0.6   | 0.7   | 0.8   | ModelGPT |
    |---------|-------|-------|-------|-------|-------|-------|-------| -------|
    | $R_5$     | 0.487 | 0.513 | 0.478 | 0.470 | 0.452 | 0.461 | 0.487 | 0.457 |
    | $\tau$     | 0.301 | 0.206 | 0.264 | 0.296 | 0.330 | 0.290 | 0.264 | 0.197 |
    | $R_5+\tau$   | 0.788 | 0.719 | 0.742 | 0.766 | 0.782 | 0.751 | 0.751 | 0.654 |

    The experimental results show that, for different values of $\alpha$, our approach consistently outperforms the current SoTA method, ModelGPT, demonstrating the robustness of our method to the choice of $\alpha$.

* Q2: Regarding the experiment setting in Q2, is the performance measured on just one specific test dataset, or is it an average across all 23 test datasets?
* A2: It is an average across all 23 test datasets.
Thank you for your detailed response and for addressing my concerns. However, I have a few more questions. Is the parameter 
$\alpha$ sensitive to different datasets, and what is the best strategy for selecting it? Additionally, could you provide some analysis on this? Another question for clarification: Regarding the experiment setting in Q2, is the performance measured on just one specific test dataset, or is it an average across all 23 test datasets?
Thanks for your response and my concerns are solved.
Dear Reviewer mpun,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.
Dear Reviewer omtt,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.
Dear Reviewer NUg2,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.
Dear Reviewer qHTS,

We hope this message finds you well. I am reaching out to kindly request your prompt response to confirm whether our responses adequately address your queries. We sincerely thank you for your time and effort during this discussion period. Your timely feedback is greatly appreciated.