Bridge the Modality and Capability Gaps in Vision-Language Model Selection
Authors: Chao Yi
Yuhang He
De-Chuan Zhan
Han-Jia Ye
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language Model
Model Selection
Zero-Shot Image Classification
Primary area: machine_vision

Abstract: Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zero-shot image classification tasks, a promising strategy is selecting appropriate Pre-Trained VLMs from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the “Modality Gap”—the disparity in VLM’s embeddings across two different modalities, making text a less reliable substitute for images; and the “Capability Gap”— the discrepancy between the VLM’s overall ranking and its ranking for target dataset, hindering direct prediction of a model’s dataset-specific performance from its general performance. We propose VLM Selection With gAp Bridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. It then uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps to obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset’s images. Experiments across various VLMs and image classification datasets validate SWAB’s effectiveness. Code is available at: https://github.com/YCaigogogo/SWAB.

Bridge the Modality and Capability Gaps in Vision-Language Model Selection
Chao Yi, Yu-Hang He, De-Chuan Zhan, Han-Jia Ye
State Key Laboratory for Novel Software Technology. Nanjing University hyic,heyh, zhande, yehjj-llamda anajaanuauaa.......a..... .nj..eu..cc. edu.
Abstract
Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zero- shot image classification tasks, a promising strategy is selecting appropriate Pre- Trained VLMs from the VLM Zoo, relying solely the text data of the target dataset without access to the dataset's images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the "Modality
disparity in VLM's embeddings across two different modalities, making text less reliable substitute for images; and the "Capability Gap"- the discrepancy between the VLM's overall ranking and its ranking for target dataset, hindering direct prediction of model's dataset-specific performance from its general performance. We propose VLM Selection With gAp Bridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. then uses this matrix transfer useful statistics of VL.Ms from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset's 's images. Experiments across various VLMs and image classification datasets validate SWAB'S effectiveness. Code is available at: https: //gi thub. www Coa////aoooooo///A....
Introduction
Vision- DisionLLanguage Models (VLMs) [46, 22, 48, 68] have demonstrated impressive image-text matching ability. One notable application of VLMs is zero-shot image classification [46, 40, 14, 37], where VLMs are leveraged to generate image classifiers using only class names directly. This zero-shot approach has shown considerable success in scenarios with scarce
training images [35, 18).
Despite the success of VLM image classification, the performance according to the datasets and domains [11], making it challenging to use a single model to handle all tasks. Fortunately, many open-source VLMs are available [21], and these VLMs form vast VLM Zoo. With different architectures, pre-training datasets, or training methods, these VLMs have different strengths. The diverse pre-trained VLMs increase likelihood of pinpointing at least one VLM that excels in given target dataset in most cases. To more effectively reuse the VLM Zoo across diverse target tasks and unlock its full potential, we need a model selection method to choose suitable VLMs from the VLM Zoo for the target task. However, in scenarios such as zero-shot image
VLM may vary substantially
Throughout this paper, the term "VLM" specifically refers to pre-trained VLM.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Step 1: User Gives Task Description
Step Use Model Selection Algorithm Select Model
I want VLM to help me classify (Cat, Dog) classes (Natural Image) domain.
Spen-Source Datanels
Image Data
"Lion"
AA slooping
Text Data
Model Search Engine Q
Abs Reil Perf
0.76
Step Use LLM to Generate Auxiliary Texts
Model Selection Algorithm
"Generate 50 captions for (Cat, Dog) separately for the (Natural Image) domain."
VLM
aiT
0.51
T-
Input
Reist-Related Auxiliary Texts
VLM 8
"A cat is catching a mouse"
"A dog is barking stranger"
Model Zoo
Language- Only
Dog running"
Selected Model
Target Datasaat Data
Figure 1: Paradigm of Sanguage-Only VLM Selection (LOVM). Users describe the details of their target tasks in text form, such as class names and image domains. Then, LOVM utilizes this information to generate class-related labeled texts through ChatGPT, These texts serve as substitutes for image samples in subsequent model selection algorithms. The model selection algorithm uses two types of data, including the open-source datasets (which have image and text data) and the text data from the target dataset, to predict the VLM's absolute or relative performance on a target dataset. It then selects the most appropriate VLM based on the predicted performance.
classification, many users might not have labeled images for their target tasks, especially those who not Machine Learning researchers. They prefer to describe their needs in text and use Model Search Engine to find the most suitable model. So one solution is identifving the most suitable VLMs in the zoo for a target datasel without access to the datasel's images. This VLM selection termed as (Language-Onfe VLM Selection" (LOVM) [73], and the paradigm is illustrated in Figure 1.
Two key types of information are available for LOVM. One is the target dataset's text data, i.e., names of the target classes and class-related labeled texts generated by LLMs (Details described in Section B. The other is the open-source datasets, collected in the form of images with their corresponding class names. Based on these data, the goal is to estimate a VLM's zero-shot image classification capability ranking among the VLM challenges stemming from the inherent heterogeneity in models and datasets. The first challenge is the Modality Gap across different modal features extracted by a VLM. Since the visual and textual features extracted by VLMs tend to cluster into two distinct groups and have gap vectors between them [31], using text data as image proxies rank VLMs is inaccurate. The second challenge is the Capability Gap between the VLM's overall ranking and its ranking in the specific dataset. Owing to the VLM's performance variation across different datasets, the VLM's average performance on open-source datasets is hard to reflect its performance on specific target dataset. Thus, selecting a VLM based solely on its general strength may prove to be less effective strategy.
the target dataset. LOVM encounters two
In this paper, we propose VLM Selection With gAp Bridging (SWAB) to address both gaps. The key idea is to reuse VLMs' statistics from open-source datasets to estimate their statistics the target dataset, which mitigates the negative impact of these two gaps. In particular, SWAB first uses optimal transport calculate the transport matrix based on textual similarity between class names of open-source and target datasets. After applying VLMs on open-source datasets to calculate VLMs' statistics, i.e., the class-specific modality gap vectors and performance rankings of different VLMs, S SWAB utilizes these statistics to estimate the same type of statistics on the target dataset. After that, S SWAB uses the estimated gap vectors align the features of texts with the features of images from the corresponding category, which bridges the modality gap. Meanwhile, SWAB'S estimated VLMs' ranking also improves the prediction of their rankings on the target task, bridging the capability gap. The related work is in the appendix C. The main contributions are:
We analyze two key challenges in LOVM the modality gap across VLM's modal features and the capability gap between the VLM's overall ranking and its ranking on the target dataset.
We propose SWAB, which utilizes optimal transport transform useful statisties of VLMs on open-source datasets to the target dataset to bridge two gaps.
Experimental results on LOVM benchmark composed of wide range of VLMs and image classification datasets demonstrate the effectiveness of S SWAB.

---

2 Preliminary
We formally introduce the LOVM setting, a baseline method for LOVM, and analyze the two kinds of gaps in LOVM. We use to represent the Euclidean norm of vector unless otherwise defined.
2.1 Selecting VLMs from Model Zoo
Zero-Shot Image Classification of VLM. Assume there is pre-trained VLM f f (f' (f' consisting of an image encoder f' and text encoder f Given an image classification dataset T with AT class names CT {e]
.... c ........ J. we input the class names CT (probably with templates like "A photo of (class]") into the VLM's text encoder f" to get the image classifiers {t, ر Then, given a test image Di, we use the image encoder f to extract its feature Ø;. Finally, we predict the label via the cosine similarity between the image feature át, and image classifiers { The class with the highest cosine similarity to the image is selected as the predicted class fi- Given a; = f (xx), ł f (c)) Equation describes this zero-shot image classification process:
ú = f(, CT) = argmax of E]T] ||::|| ||ty||
VLM Zoo. In recent years, a large number of (pre-trained) VLMs have emerged. Assume a collection of M VLMs constitute VLM Zoo M [fmm (S fm)) m= The capability of fm is determined by three key factors: the model architecture (e.g., Transformer [57], ConvNeXt [34]), the pre-trained dataset (e.g., LAION-400M [47], MS-COCO [32]), and the training method (e.g., contrastive loss [46], caption loss [67]). Combinations of these factors result in "good and diverse"
VLMs in M. Given dataset T, it is probable to find suitable VLM from the VLM zoo with high zero-shot image classification performance on T.
Language-Onle VLM Selection (LOVM). Rather than using images from the target dataset, LOVM focuses on the zero-shot scenario where only the target dataset's 's text data, such as its class names Cr, are available for VLM selection. Besides, cation datasets S. The set of class names in S is Cs = {cf .... -gg 1. and the D's denote the labelled images in these classes. Given a target task T, the VLM selection method h estimates the zero-shot classification ability of fm based on Cr, Cs. and D's via FlT = h(Sm CT, Cs. D's), where m E [L..... .... TT is the predicted ranking of the m-th VLM fm on T. The higher the ranking, the more probable fm achieves higher zero-shot image classification performance on T. Assuming we can obtain the test image set D/ of the target dataset T with |D/] im- ages, then we can calculate the zero-shot image classification accuracy PT. of fm is calculated by Pm
can obtain some open-source image classifi-
[1; fm (z,, Cr)). fm CT) represents the predicted class with the (EDD)
Σ
|D
same manner as Equation 1. I(.) is the indicator function, which outputs 1 if the condition is satisfied, and 0 otherwise. Based on {pm LM by assigning higher ranking m= we obtain the true ranking of M VLMs TT = [rT,..., rM] models with higher accuracy p. However, in the zero-shot see- nario, we can't 't obtain the test images set DI in advance. Therefore, the goal of LOVM is to make the predicted ranking T = [AT ....... FT] be an accurate estimation of the ground truth ranking T = ITT ...... TTr] so that the best VLM can be selected.
Evaluation of LOVM Methods. We measure the performance of the LOVM algorithm by comparing the ranking similarity between TT and ,T_ Specifically, we calculate the Top-5 Recall Rs (ranges from 0 to 1) and Kendall's 's Rank Correlation r (ranges from -1 to 1). The larger the better.
2.2 Possible Paradigms for LOVM
Non- Non-LLearning-bsee LOVM. There are three main paradigms for LOVM. The first paradigm is to neglect the visual encoder and select VLM solely on texts. In detail, we can utilize ChatGPT [43] to generate auxiliary texts Dr based class names CT of T. More details are described in Section B.I. These class-specific texts act as "image proxies". Then, whether a VLM fm fits T could be measured by transferability metrics, e.g., H-Score [2] and LogME [65], between the VLM's text encoder f and generated text dataset Dr. The second paradigm relies on the general performance
3

---

of certain VLM fm. We use open- datasets to measure a VLM's general performance. If fm achieves high zero-shot classification performance over open-source datasets, then it expected be competitive on T. These methods assume that VLM''s ranking is relatively consistent across tasks.
Learning-based LOVM. The third paradigm is based on the learning process. In detail, the ability of a VLM could be predicted based on a ranker model fR- The input of fR is a vector 8 depicting the dataset-specific representation of fm on T, while the output of fR is the relative/absolute performance P'm E IR of fm on T. The fR could be learned on open-source datasets S [70, 73J. Due to the availability of both class names Cs and images D/s in the open-source dataset S such as ImageNet [8],
JMM we can calculate each VLM's representation {8 J m==1,==1 and true zero-shot image classification accuracy {pmm M [mmmm=1,=== m=1,x=1- Here N refers the number of datasets in S. After constructing the train M,
set, the ranker model I' is learned based on the {8) P mm
N Σ
M
min fR
((fa(s"). m=
(2)
é is a loss function that measures the discrepancy between the prediction and the ground truth, which can be Mean Squared Error Loss and Huber Loss, among others. Given T, the learned fn able to predict the performance {p% mmm1 over {sT 1MMMMM mm via PT = fr(( (T((sT)). Finally, we can get the predicted VLMs' ranking A based on (pT 1 Jm=1 This approach has similarities with meta- learning [12. 3]. Meta-learning attempts to use data from multiple datasets to learn a model adaptable to the target task, while Learning-based LOVM employs data from multiple datasets to learn a ranker model for selecting the suitable model from VLM Zoo for the target task. The representation 8 one of the keys in this paradigm, and ModelGPT [73] calculates values sm via the capability of VLM's text encoder fm
ModelGPT uses generated text data Dy as substitutes for images to calculate some metries, which measures the zero-shot ability of fm on unseen images by the classification ability of fm on Dy:
T m. = Mctric, (fm:
Here Metric, indicates the i-th metrics function such as Top-1 Accuracy and F1-Score. For example, the Top-1 Accuracy sT., could be calculated in a similar manner as Equation 1, with the only difference heing that the test samples were replaced with text samples l; instead of image samples æy:
T m,
Σ (K..))DTT
Im4: CT)).
DT
Besides, ModelGPT uses some metries for assessing the features' quality extracted by the VLM's text encoder fT More details are in the Section B.2. Moreover, the zero-shot classification performance of fm on ImageNet is also included in 8 a general ability measure of fm. ModelGPT implements fn as a simple linear model.
2.3 Analysis of the Two Gaps in LOVM
There are two main challenges that limit the application of the aforementioned paradigms in LOVM. The first is the modality gap across different modalities' features in VLM's feature space, and the second is the capability gap between VLM's overall performance and dataset-specific performance. Modality Gap. As described in Section 2.2, methods like H-Score, LogME, and ModelGPT utilize the ChatGPT generated auxiliary texts Dy as image proxies calculate metries that measure the zero- shot accuracy on the target dataset T. In other words, the zero-shot classification ability across text and image modalities is estimated by the intra-modality classification ability. The latent assumption is that the generated texts and their corresponding images are closely aligned in VLM's feature space. However, this assumption is difficult to meet [31], and instances' features are more likely to cluster according to their modalities. In particular, we define the modality gap vector between the features of an image-text pair (æ;, t4) as fm.i P/ (xx) f (t+). Values the gap vector are generally not close to zero. We name this phenomenon as Modality Gap in LOVM, which makes the scores on πT hard to reveal the true zero image classification capability of a VLM on given dataset.

---

(R) Modality Gap
(b) Capability Gap
of JynageNet:
Model Zoo (M = 43)
Task Zoo (N = 23)
Acc
Acce
2
Imagettet
= 0.21
V
MAE 0.32
Test
Y2 e
VLM
T-- w--
VUM
72 E RM
2 (H.) M
Text Sumples
On
2.34
Oxford Pate
T I
T--
8 F' 8
Ere (max(,)
min (T))
Calculate
M
Test Image Samples
Figure 2: Validation Experiments on the Modality Gap and Capability Gap. (a) Predicted VLMs' zero-shot image classification accuracy based on generated text data vs. VLM's true accuracy based on test images. Each point in the graph represents model. From the result, predicted accuracy poorly aligns with the true accuracy. indicating these text data are ineffective substitutes for image data. (b) We calculate the zero-shot image classification performance rankings of 43 VLMs across 23 datasets. We compute the average standard deviations and the mean value of differences between each VLM's maximum and minimum ranking. The result shows the performance of VLM varies greatly across different datasets.
find that the
We conduct a validation experiment on ImageNet with 43 VLMs. We first generate 50 auxiliary texts per class as Dr and then calculate the predicted Top-1 accuracy via Equation 4. Next, we use test images to calculate the VLM's true Top-1 accuracy. The consistency between the predicted Top-1 accuracy and true zero-shot image classification accuracy Pm,.T is measured by the Kendall Rank Correlation (r, higher is better) and Mean Absolute Error (MAE, lower is better). It can be observed from the left part of Figure 2 that the predicted accuracy derived from auxiliary texts Dr does not closely match the true accuracy, indicating that these generated auxiliary texts in Dy are not effective proxies for images.
To make the auxiliary texts act as better image proxies, one intuitive idea is to estimate the gap vector g for each image-text pair. Then we can add it the feature f (t:) of the text t; eliminate the modality gap, which may lead to more accurate scores sT. in Equation 3. However, the gap vector cannot be calculated directly without the target dataset's images. Furthermore, gap vectors for different classes are diverse, so using a shared vector across all datasets may not be a good choice.
Capability Gap. To select one VLM from the model zoo given target dataset, one direct approach to select the VLM that performs the best on average across multiple datasets. For example, we may first estimate the VLM's zero-shot classification ability open-source datasets and then select the VLM with the highest performance. The key question is whether a VLM's s average ranking on the open-source datasets can reveal its true ranking the target dataset. Our empirical analyses indicate that there exists a discrepancy between the VLM's overall ranking and its ranking on a specific dataset. We name the discrepancy between the VLM's average ability and its specific ability as the Capability Gap, which results from the fact that a VLM's performance fluctuates significantly across various datasets.
To verify the claim, we test 43 VLMs on 23 target datasets provided by [73] and obtain the rankings of each VLM across these datasets. Based on these ranking results, we calculate the average standard deviation and the mean value of the difference between each VLM's maximum and minimum ranking. The experiment process is illustrated in the right part of Figure 2. We find that the mean difference between one VLM's maximum and the minimum ranking is 38.86. Since the total number of VLMs is 43, such a difference demonstrates that the tor-performing VLM in one dataset could likely be among the worst in another.
One solution to bridge such a capability gap is to consider the VLMs' ranking on related dataset. In other words, the ranking of VLMs on datasets from open-source datasets collections that are relevant to the target task may provide more useful insights than general performance ranking across all tasks. The main challenge to figure out which open-source datasets are similar to the target dataset and transform the VLM's ranking these datasets to the target dataset.
5

---

Step Construct the Transport Matrix
Step 3: Bridge the Capability Gap
Open-Source
Classes
Open-Source Tasks' Class-Specific Rank
Target Task's
Text Encoder
Class-Speciff Rank
C5
Transport Matrix
Optimal Transport
mm
Tr Aru
D Pins
r
P
Target Task's Classes
2: Bridge the Modality
VLM's twat embadding cluster
Stop Ensemble Predicted Rankings
Task's a Gap Vectors
Open-Source
Target Taak's Gap Vectors
Text Samples
VLMss image ombedding ciuster
m
a Final Rank Pradiction
Add
6m
6
Rank Model
Weighted Borda Count
.7n5
mmkk
Gap
+8
Modified Test Samplea
P (2)
Figure 3: The workflow of SWAB. SWAB first constructs a transport matrix Y &s xky using optimal transport, based on textual semantic similarity between classes in the open-source datasets Cs = {cf. ... ..h. } and the target dataset's classes Cy = {oT cky J- Using this matrix, SWAB estimates VLM fm's class-specific gap vectors (0m,1." } on the target dataset T from the gap vectors G Xd in the open-source datasets. These estimated gap vectors help modify text data to act as more effective substitutes for image data. The modified text data will then be input into the Ranker Model fn. which predicts VLM's performance .T.(1)) the target dataset. Besides, SWAB also uses the transport matrix y" to predict VLM's performance ranking on the target dataset based on VLM's 'ss class-specific rankings r R open-source datasets. Finally, SWAB combines these two ranking predictions Fm and TT))))) to determine the VLM's final ranking prediction.
Summary. We emphasize two kinds of gaps in LOVM, i.e., the modality gap across features of different modalities generated by VLM, and the capability gap between a VLM's overall ranking and its ranking given specific target dataset. Both two gaps pose obstacles previous model selection methods, such as I LogME and ModelGPT, and degrade their abilities in VLM selection. Moreover, those intuitive approaches to bridge the gaps still face challenges.
3 VLM Selection with Gap Bridging
To mitigate the impact of both gaps on LOVM and integrate non-learning-basse based and learning- based LOVM methods, we propose VLM Selection With gAp Bridging (SWAB). The key idea of SWAB to bridge modality and capability gaps by utilizing class-level statistics of VLMS from open-source datasets. By measuring the textual similarity between the target dataset's class names and those in open-source datasets, we construct bridge matrix. Based it, we estimate the gap vectors between image and text modalities, which rectifies the text-derived scores ModelGPT. addition, we predict the VLM's performance ranking for the target dataset based the bridge matrix and VLM's ranking on the open-source dataset. Both estimated statistics will be used to obtain a more accurate language-only VLM selection. The workflow of SWAB is illustrated in Figure 3.
3.1 Construct the Bridge Matrix Using Optimal Transport
Benefiting from the open-source datasets, some useful class-level statistics, such as modality gap vectors and zero-shot classification accuracy of certain VLM, could be calculated, which can help the performance ranking estimation of VLM on the target dataset. To better utilize these class- statistics for predicting the corresponding statistics of VLM on the target task, we introduce semantic relevance information between open-source datasets' classes and target dataset's classes into the statistics reusing process, which is automatically generated through Optimal Transport [7, 45).
Recall that the sets of class names of the open-source datasets and the target dataset are Cs = {ep k and C7 = {cT))A } respectively. The semantic relevance between two classes could be measured by the textual similarity between their class names. In detail, we use a pre-trained text encoder (e.g., MPNet [49]), which extracts text features for these class names, i.e., (o(cf), ,(((.))

---

between the text feature of the i-th class in the open-source datasets olc))) and that of the j-th class in the target dataset ő(c]) ). The larger the cosine similarity, the more similar the two classes are. After that,
and (o(T)), ... -(( Then, we can calculate the cosine similarity
o(c") d(e!) T((e))
In practice, we exponentiate each element in cost E B using the base to amplify its differences. We then solve the optimal transport problem with the constructed cost matrix to get the transport matrix Y Since optimal transport aims to obtain a transport matrix that minimizes the transmission cost, the transport matrix 7 will reuse more information between semantically similar classes:
we construct the cost matrix in optimal transport via cost.j = le )illooe III
Y"
argmin YER 20
Tii cost s.t. 71 = w 7 = 0; Ni 2 0.
(5)
i.j
The cost matrix quantifies the expense of moving elements between all class pairs, and y x is the transport matrix. OT minimizes the cost indicated by the matrix cost and moves elements from one distribution u to another v. In SWAB, we define u and v as uniformly distributed vector u = 1/ks E & and 1/ky E A This indicates that we treat all classes as equally important. We may also incorporate prior knowledge of class importance to define u and v.
The solution y" of the OT problem in Equation 5 could be solved efficiently [13], and y acts as a bridge matrix between open-source datasets' classes and target dataset's classes. Usually, the smaller cost,,j is, the larger the corresponding element Y obtained by OT, indicating statistics of the i-th class of open-source datasets may help more when we estimate the statisties of the j-th target class.
3.2 Bridge the Modality Gap and Capability Gap
Bridge the Modality Gap. Given the m-th VLM fm in the model zoo, we want to estimate the modality gap ym between the extracted image and text features for the j-th class in the target dataset T to bridge the modality gap. However, in the zero-shot scenario, we can't get the target dataset's images in advance, so we can't 't directly calculate the gap vectors using image-text pairs. To solve this problem, SWAB estimates the target dataset's s gap vectors based on the open-source datasets' gap vectors with y" Given the k-th open- class ci. we can get the set of images D'S, { y4) (z y1) E D'S, y: = <f from the open-source datasets. |D's is the number of
images in D's.. Then, the modality gap vector gh,k for class ch and model fm can be calculated
through g I (E..)))DD
Σ
1 (2:)
(2 Ih is the average difference between II
)
the normalized class text prototype embedding and all normalized image embeddings from the class & In a similar manner, the gap vectors of all open-source classes { ... Im,kk J can be obtained given fm- We use matrix G E TR*sxd,, to represent those ks gap vectors for the m-th VLM in the VLM Zoo, and dm is the dimensionality of features extracted by fm.
The gap vectors {9 .......... the transport matrix y" If two classes are semantically similar, then we can reuse the gap vector from the similar class. We set the predicted gap vector for the j-th target class 9m. as weighted sum of G and the weight comes from y", which is 9M. = |CT((>t,) G Yi is the j-th column of y*, We use scaling factors [CT| to ensure that for each target class, the sum of y equals 1. This scale operation has also been used in previous work [62, 63). After that, we modify the step of ModelGPT in Equation 3, where the metries over the generated auxiliary texts Dr are calculated. We add the gap vector 9h.. to the embeddings of the auxiliary texts D' from the j-th class in the target dataset:
for the target dataset could be estimated based on the G and
tm,s = ff)(tt)) + ST... Vt, D4.
(6)
The modified text embedding i.., serves as better image proxies. In other words, classification metrics on (t:) only reveal the discerning ability of the text encoder of fm. which is far from the (cross-modal) zero-shot classification ability due to the modality gap. By bridging such a
gap with modified text embedding, classification metrics on bm,i are closer to the classification metrics on images with textual classifier. Therefore, we use { in Equation 6 better
7

---

inputs to calculate s in Equation 3. The updated score vectors s are then input to the ranker model fn. which is able to get more accurate VLM's performance prediction P Based on the performance prediction (pTm 1m-1, we can obtain the VLMs' ranking (T = |+T .PM
TT
Ranking (IpT... ). Ranking(-) transforms the accuracies into models' ranking.
Bridge the Capability Gap. Whether the m-th VIM fm fits the target task T could also be estimated by the performance of fm on the open-source datasets related to T. We first calculate the VLM's class-level performance ranking over the whole open-souree datasets. Given the k-th open-source class oj and the corresponding set of images D we calculate the zero-shot classification accuracy p via Equation 1. We then determine each VI.M's ranking on the k-th open-source class c using Equation 7. We calculate VLMs' ranking for other open-source classes in the same way.
F = IT
M,A Ranking ( ([pī, -. ......MMMM))))).) kM.
Next, estimate the ranking of certain VLM fm on the target dataset T. By re-organizing the ranking values in Equation 7, the performance ranking vector of fm on all ks open-source classes is w = [ Im,,.... ... .rm,ks E z If fm ranks high on certain open-source classes related to the classes in the target datasct, the performance ranking of fm on the target dataset may also be high. Thus, we perform weighted sum of ranking values in r and assign larger weights to those classes related to the target dataset. This can be done by using y":
(8) Elements in 77 E IT are the predicted ranking of the m-th VLM fm for ke target classes. Since we only need the relative order of ranks, there is no additional scale factor in Equation 8. After that, we average class-specific ranking values in T and use *T.(2) Mean (+T) to denote the estimated ranking of fm on the target dataset TT In summary, we predict the ranking of models on each class in the target dataset based on their rankings in classes of the open-source datasets, guided by the semantic relevance between the classes. We predict the ranking of models on each category in the target dataset based on their rankings in categories of the open-source dataset, guided by the semantic relevance between the categories. Benefiting from the models' consistently aligned classification performance ranking across similar classes, this approach bridges the capability gap in VLM selection.
T m
Tr Y
3.3 Summary of SWAB
As described in Section 3.2, given a target dataset T and VLM fm. we denote the two performance
ranking predictions obtained by bridging the modality gap and capability gap as FT: TT.(1) and +T.(2) respectively. T ,T.()) is predicted based on ModelGPT with our modified embeddings of the generated auxiliary texts for T. =T)(2) is predicted by the weighted sum of VLM's class-specific ranking values on the open-source datasets. These two predictions, respectively, originate from learning- based and non-learning-base based LOVM methods. We ensemble two predictions together and achieve a more accurate model ranking estimation. We utilize the weighted Borda Count to aggregate two rankings:
AT a TT.01 + (1 a) -T,(2)
We set a = 0.5. Ultimately, SWAB determines the final predicted ranking of the VLMs in the VLM Zoo based on #T ,ens, The pseudo-code of SWAB is listed in Algorithm
4 Experiments
4.1 Evaluation on LOVM Benchmark
Setups. We follow LOVM [73] to use its provided VLM Zoo. In addition, to further enhance the diversity of the VLM Zoo, we add some representative VLMs such as BLIP [28] and BEÎT- [59] to expand the VLM Zoo. The final VLM Zoo contains a total of 43 models, which differ in aspects such as model architecture, pre-training datasets, and training methods. In the appendix, we also provide experimental results of different model selection methods on the 35 models originally offered by LOVM. We evaluate different methods on 23 datasets, i.e. ImageNet [8], Aircraft [36].
8

---

Table I: Results on LOVM Benchmark. We evaluate our method across 23 datasets and 43 pre- trained VLMs. The results are averaged over all datasets. Our SWAB achieves the best results across all metrics. For methods that involve adding random noise to data features, we report the standard deviation of metrics across 10 experiments to mitigate the impact of randomness on result reliability.
Methods
H- H-Score
NCE
LEEP
LogME
INB
Avg Rank
||
ModelGPT
SWAB
(a(t)
0.174
0.235
0.161
0.191
0.443
0.443
0.446::0004
0.504
r(t)
0.000
-0.014
0.014
-0.014
0.267
0,246
0.270= 009
0.318 =0.002
Rs + +t)
0.174
0.221
0.175
0.177
0.710
0.689
||
0.716::0011
0.822
CIFAR100 [26] and so on. We obtain VLM's ground truth ranking based on VLM's Top-1 Accuracy calculated on the target dataset's test image set.
Baseline. We select representative methods for each of the three paradigms mentioned in Section 2.2 as our baselines. For the first paradigm, we use four classic model selection methods: H-Score [2], NCE [55], LEEP [56] and LogME [65]. For the second paradigm, we use the VLM's ranking on ImageNet (INB) and VLM's average ranking (Avg Rank) on classes of the open- datasets in the LOVM Benchmark. For the third paradigm,
compare our method with ModelGPT [73].
Evaluations. We use Top-5 Recall and Kendall's Rank Correlation to measure the similarity between the predicted and the ground truth model rankings to evaluate the LOVM method's 's performance. We also calculate the sum of these two metrics to consider the method's comprehensive capability.
Implementation Details. For fair comparison, SWAB follow ModelGPT [73] to sequentially extract target dataset from each of the 23 datasets in the LOVM Benchmark and treat the remaining datasets as open-source datasets. Besides, SWAB adopts ModelGPT's approach of adding Gaussian noise to corrupt the target dataset's generated text embeddings. Since LOVM does not provide the specific image data for the 23 datasets, we download these datasets ourselves and adopt their standard data splits. Using the templates provided by LOVM, we construct classifiers and recalculate each VI.M's zero-shot image classification accuracy on these datasets. Additionally, we utilize the code provided by LOVM to generate class-related text data using ChatGPT. For H-Score, NCE, LEEP, LogME, INB, and Avg Rank, we follow the practices of previous work and do not add noise to the model's s inputs. To ensure reliable results, we conduct ten repeated experiments using random seeds from 1 to 10 and report the mean value and standard deviation of ModelGPT's performance and SWAB'S performance in Table 1.
Results Analysis. From Table 1, we can draw the following conclusions: (1) Metric-based non- learning model selection methods such as LogME show poor performance on the LOVM Benchmark. This is primarily because such algorithms rely on the target dataset's 's images, thus the modality gap has a greater negative impact on them when using generated text data as a substitute for images. (2) Using open-source datasets is helpful for LOVM. We find that using open-source datasets in non-learning way (e.g. INB, Avg Rank) or a learning way (e.g. ModelGPT) all helps LOVM, since their performance significantly surpasses that of methods not utilizing open-source datasets (e.g. LogME). (3) Despite leveraging more open-source datasets, the performance of Average Rank is worse than INB. This confirms our analysis of the Capability Gap, which suggests discrepancy between the average ranking of a VLM and its ranking on a specific dataset. (4) Our SWAB achieves the best performance across all evaluation metrics. Notably, our final performance of Rs + (0.822) represents a significant improvement of 14.8% over the original SoTA method ModelGPT (0.716).
4.2 Ablation Study
Table 2: Ablation Study of SWAB. SWAB-C, S SWAB-M, and SWAB indicates only bridging the Capability Gap, only bridging the Modality Gap, and bridging both gaps in SWAB
We conduct an ablation study to demonstrate that bridging the Modality Gap and Capability Gap are both essential for SWAB. Table 2 presents our ex- periment results, from which we can observe that SWAB achieves the best performance across all met- rics when both gaps are bridged simultaneously. The ablation study confirms our analysis.
Method
Re(t)
7(1)
Rs +(t)
SWAB-C SWAR-M SWAB
0.29611001 0.4872002
0.783
0.4742000
0.3162209
0.790
0.504=0000
0,318+00002
0.822

---

4.3 Influence of Key Components in SWAB
Will Bridge the Capability Gap Be Beneficial for VLM Selection? We compare the LOVM performance directly using the VLM's average ranking on each class of open-source datasets and weighted-sum ranking based on transport matrix y². The results are shown in Table 3. We can find that utillzzing class relevance to bridge the Capability Gap is beneficial for VLM's Model Selection.
Will Bridge the Modality Gap Be Beneficial for VLM Selection? To eliminate the interference of other factors, we solely utilize the learning- based predicted rankings =T.(1) in SWAB, and the input to the ranker model fm only consists of metrics calculated on the generated text data Dr, which serves as substitutes for images. In this way, the method's performance depends solely the quality of the generated text data Dy. From the Table 4,
find that the generated text data Dy become better substitutes for image data after bridging the Modality Gap.
Table 3: Results of (7)(2) on the LOVM before and after bridging the Capability Gap.
Table 4: Results of Pm (T.(2) before and after bridg- ing the Modality Gap (MG).
Method
Rs(1)
=(t)
Rs + r(t)
Method
Rs(t)
r(f)
Rş +r()(
Average Rank
0.443
0.246
0.689
Before Bridging MG
0.216
0.061
0,277
OT Weighted Rank
0.513
0.217
0.730
After Bridging MG
0.371
0.080
0.451
Which Kind of Gap Vectors Should We Use? When utilizing the gap vectors from open-source datasets, we have two options: (1) Use the dataset-level mean gap vector calculated on the whole dataset's image-text pairs. (2) Use the class-level mean gap vector calculated on the corresponding class's image-text pairs. We hope that the gap vectors are as close to each other as possible so that their mean vector can substitute well for the whole set. Based on this idea, we calculate the statistics of the gap vectors within a dataset and within each class. We calculate three metrics which include: (I) the standard deviation of these gap vectors' magnitude; (2) the mean cosine similarity between these gap vectors and their corresponding mean gap vectors; and (3) the standard deviation of these cosine similarities. These metrics reflect the consistency of the gap vectors. Table 5 shows the results.
Table 5: Results of metrics measuring gap vec- tors' consistency belonging to the same dataset or the same class. M: Magnitude, D: Direction.
Table 6: Results of SWAB-M on the LOVM Benchmark using the datasct-level mean gap vectors and class-level mean gap vectors.
ImageNet D-Mean(1) D-Sid(L)
Gap Vector
Rs(t)
r(t)
Rs + rtt)
Gap Vector
M-Si((L)
Dataset Mean
0.443
0.304
0.747
Dataset Mean
0.04
0.68
0.07
Class Mean
0.474
0.316
0.790
Class Mean
0.03
0.85
0.04
From the Table 5, we can find that the class-level gap vectors tend be more consistent, which inspires us to use the class-level mean gap vectors. We also compare the results of SWAB-M on the LOVM Benchmark using the dataset-level mean gap vectors and the class-level mean gap vectors, respectively. The implementation details are the same as Table 2. Table 6 shows the results, which verifies that using the class-level mean gap vectors is a better choice.
Conclusion
We analyze and address two key challenges in Language-Only VLM Selection (LOVM), which are VLM's modality gap across different modal features and VLM's Capability gap between its overall and dataset-specific rankings. Our key insight that we can reuse the model's useful statistics on open-source datasets to help the model selection on the target dataset. SWAB utilizes a transport matrix between classes of the target dataset and open-source datasets to transfer VLM's class-specific modality gap vectors and class-specific rank from open-source datasets to the target dataset, which mitigates the negative impacts of these two gaps. Experiment results on the LOVM benchmark show the superiority of our method.
10

---

Acknowledgements
This work is partially supported by National Key R&D Program of China (20222D01 14805), NSFC (62376118, 18, 62250069, 620061 12, 61921006), Collaborative Innovation Center of Novel Software Technology and Industrialization, CCI-Tencent Rhino-Bird Open Research Fund (RAGR20240101).
References
[1] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless C. Fowlkes, Stefano Snatto, and Pictro Perona. Task2vec: Task embedding for meta-learning. In ICCV, 2019.
[2] Yajic Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Lconidas J. Guibas. An copomattion-hhorettii approach to transferability in task transfer learning. In ICIP, 2019.
[3] Len-Lun Chao, Han-Jia Ye, De- De-Chuan Zhan, Mark E. Campbell, and Kilian Q. Weinberger. Revisiting meta-learning as supervised learning. CoRR, 2020. URL https: //wxivaaxvvv///a///////e//eaaaa/ www.wwwwwwwwaa......oowoaaaaaa.a htggabbs/2022.05///
[4] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Pruc. IEEE, 2017.
[5] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[6] Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of singlc-layer networks in unsupervised feature learning. In AISTATS, 2011.
[7] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NewrIPS, 2013.
[8] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[9] Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks. In ECCV, 2022.
[10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisser- man. The pascal visual object classes challenge 2007 (voc2007) results. http://ww.ppaeal 2007.
[11] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML, 2022.
[12] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcaming for fast adaptation of deep networks. ICMI., 2017.
[13] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélic Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Jevgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. JMLR, 2021.
[14] Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming- Yang, Hartwig Adam, Laurent Itti. Balaji Lakshminarayanan. and Jiaping Zhao. Improving zero-shot generalization and robustness of modal models. In CVPR, 2023.
[15] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013.
[16] Dumitru Tan Goodfellow, Will Cukierski, and Yoshua Bengio. Challenges in representation learning: Facial expression recognition challenge, 2013.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] Xiaoxuan He, Siming Fu, Xinpeng Ding, Yuchen Cao, and Hualiang Wang. Uniformly distributed category prototype-guid guided lision-language framework for long tail recognition. In ACM MM. 2023.
11

---

[19] Patrick Helber, Benjamin Bischke. Andreas Dengel. and Damian Borth, Burosat: A novel dataset and deep learning benchmark for land and land cover classification. J-STARS, 2019.
[20] Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In JCMI., 2022.
[21] Gabricl Uharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https::/t///tt //doi. org/10 2281/zenodo.. 5143773.
[22] Chao Jia, Yinfci Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In /CML, 2021.
[23] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Fei-Fei Li, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
[24] Kaggle and ByePacs. Kaggle diabetic retinopathy detection, 2015. URL hettps://wwwkkeet wwggle.com/ccccoo.e..ooooooo.. c/ blabetic-retinopatte
ieetection/dat..
[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graiee categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13) 13), 2013.
[26] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[27] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs, 2010.
[28] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML,, 2022.
[29] Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How configure good in-context sequence for visual question answering. In CVPR, 2024.
[30] Weihua Li. Wenyang Liu, Yanbu Guo, Bingyi Wang, and Hua Qing. Deep contextual representation learning for identifying essential proteins via integrating multisource protein features. Chinese Journal of Electronics, 2023.
[31] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Under- standing the modality gap in multi-moda contrastive representation learning. In NewrlPS, 2022.
[32] Tsung-Y Lin, Michael Maire. Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[33] Jian-Dong Liu, Zhi-Hao Tan, and Zhi-Hua Zhou. Towards making learnware specification and market evolvable. In AAAI, 2024.
[34] Zhuang Liu, Hanzi Mao, Chao-Yusa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. convnet for the 2020s. In CVPR, 2022.
[35] Teli Ma. Shijie Geng, Mengmeng Wang. Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple tailed recognition haseline via vision-language model. CoRR, abs/2111. 144455, 2021.
[36] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, Abs/1306.5151, 2013.
[37] Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. In CVPR, 2023.
[38] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: uniform manifold approxi- mation and projection. J. Open Source Softw., 2018.
[391 Deyu Meng and Lina Sun. Some new trends of deep learning research. Chinese Journal of Electronics, 2019.
[40] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
12

---

[41] Yuval Netzer. Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NewrlPS Workshop. 2011.
[42]
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of 2008.
classes. In ICVGIP, 2008
[43] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. NewrIPS, 2022.
[44] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[45] Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends Machine Learning, 2019.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[47] Christoph Schuhmana, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes. Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip- dip-filtered 400 million image-text pair. CoRR, abs/2111.021 2021.
[48] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, 2022.
[49] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tin-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurlPS, pages 16857-16867, 2020.
[50] Johannes Stallkamp, Marc Schlipsing. Jan Salmen, and Christian lgel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN, 2011.
[51] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Handling learwwares developed from heteroge- neous feature spaces without auxiliary data. In IJCAI, 2023.
[52] Peng Tan, Hai-Tian Liu, Zhi-Hao Tan, and Hua Zhou. Handling learnwares from heterogeneous feature spaces with explicit label exploitation. NeurlPS, 2024.
[53] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Towards enabling learwware handle heteroge- neous feature spaces. Mach. Learn., 2024.
[54] Zhi-Hao Tan, Jian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, Xiao- Chuan Zou, Yang Yu, and Zhi-Hua Zhou, Beimingwu: A learnware dock system. KDD, 2024.
[55] Anh Tuan Tran, Cuong Nguyen, and Tal Hassner. Transferability and hardness of supervised classifica- tion tasks. In ICCV, 2019.
[56] Anh Tuan Tran, Cuong V. Nguyen, and Tal Hassner. Leep: new measure evaluate transferability of Iearned representations. In /CML, 2020.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
[58] Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant CNNs for digital pathology, 2018
[59] Wenhui Wang, Hangho Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In CVPR, 2023.
[60] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[61] Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng, Exploring diverse in-context configurations for image captioning. In NeurlPS, 2023,
[62] Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Rectify heterogeneous models with semantic mapping. In ICML, 2018
13

---

[63] Han-Jia Ye, De-Chuan Zhan, Yuan Jiang. and Zhi-Hua Zhou. Heterogeneous few-shot model rectification with semantic mapping. TPAMI, 2021.
[64] Chao Yi, Lu Ren, De-Chuan Zhan, and Han-Jia Ye. Leveraging cross-modal neighbor representation for improved clip classification. In CVPR, 2024.
[65] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long, Logme: Practical assessment of pre-trained models for transfer learning. In ICML, 2021.
[66] Kaichao You, Yong Liu, Ziyang Zhang, Jianmin Wang, Michael I. Jordan, and Mingsheng Long. Ranking and tuning pre-trained models: a new paradigm for exploiting model hubs. JMLR, 2022.
[67] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. TMLR, 2022.
[68] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li. Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng. Luowei Zhou, Pengchuan Zhang. Florence: A new foundation model for computer vision. CoRR, abs/2111. 1432, 2021.
[69] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme. Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski. Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020.
[70] Yi-Kai Zhang, Ting-Ji Ji Huang. Yao- Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. NewrIPS, 2023.
[71] Zhi-Hua Zhou. Learnware: the future of machine learning. Frontiers Compud. Sci., 2016.
[72] Zhi-Hua Zhou and Hhi-Hao Tan. Learwware: small models do hig. Sci. China Inf. Sci., 2024.
[73] Orr Zohar, Shih-Cheng Huang. Kuan-Chieh Wang, and Serena Yeung. Lovm: Language-only vision model selection. In NeurlPS, 2023.
[74] Beiji Zou, Xi Shan, Chengzhang Zhu, Yulan Dai, Kejuan Yue, Yuanqiong Chen, Yalong Xiao, and Jiaer Huang. Deep lcarning and its application in diabetic retinopathy screening. Chinese Journal of Electronics, 2020.
14

---

In the Appendix, we introduce more details about the LOVM Benchmark as well as our extensions to it. Besides, we introduce ModelGPT's implementation and our SWAB's implementation. We also provide more experimental results of SWAB. The structure of the Appendix is as follows:
In section A, we introduce the relevant information of the 43 models and 23 datasets used in our experiments, We also introduce the evaluation metrics used in LOVM Benchmark [73].
In section B, we introduce the metries used in Equation 3.
In section C, we introduce the related work of the paper. In section D, we provide some details on SWAB'S implementation.
In section E, we provide more experimental results of SWAB.
A LOVM Benchmark Details
LOVM Benchmark [73] consists of 35 pre-trained VLMs and 23 datasets, with a total 35 805 evaluations. To further enhance the diversity of the VLM Zoo, add some representative VLMs such as BLIP [28] and BEiT-3 [59] to expand the VLM Zoo. The final VLM Zoo contains a total of 43 models, with total of 43 x 23 989 evaluations. For each evaluation, LOVM provides the VLM's zero-shot image classification accuracy on the corresponding dataset. Therefore, we can get the ground truth performance ranking of 43 VLMS on the 23 datasets.
A.1 VLMs of LOVM Benchmark
To cover as many types of models as possible, the LOVM Benchmark uses OpenCLIP library [21] to get diverse VLMs. These VLMs differ from each other in terms of the model architecture (ResNet [17]. Transformer [57], ConvNext [34]) the pre-trained dataset (OpenAl's Data [46], LAION 2b [47]), the training method (loss pmertiow/yypepparrneeeedaaaa augmentation) and so on. Table 7 displays the relevant information of each VI.M. We further add BLIIP and BEiT-3 to the original VL.M Zoo. should be noted that in addition to the image-text pair data listed in the Table 7, BEiT-3's pre-training data also includes unimodal image dataset (ImageNet-21K) K) and text datasets (English Wikipedia, BookCorpus, WebTex, CC-News, Stories). The diversity of these VLMs ensures that the experimental results calculated on them can reflect the performance of the VLM model selection algorithm in real-world situations.
A.2 Datasets of LOVM Benchmark
To cover wide a distribution of image classification tasks possible, the LOVM Benchmark collects 23 diverse datasets. These datasets differ from each other in terms of the number of categories, category semanties, image domains, and so on. Table 8 displays the relevant information of each dataset. The diversity of these tasks ensures that the experimental results calculated on them can reflect the performance of the VLM model selection method in real-world situations.
A.3 Evaluation Metrics of LOVM Benchmark
In LOVM, our aim is to maximize the rank similarity between the prediction of VLMs' ranking rT {fT I m== and VLMs' ground truth ranking TT {rTTTMM Jm= the target dataset, especially the rank similarity of the top VLMs in fr and rT. This is because we tend to focus only on whether those appropriate models can be chosen. To ensure fair comparability, we follow the evaluation metrics used by LOVM [73] to assess different model selection methods and directly utilize the code provided by LOVM [73] to calculate these metrics:
Top-5 Recall (Rş)) - Top-5 Recall Rs measures the model selection algorithm's 's accuracy in identifying the true top five best-perfominng models within its predicted top five models. The calculation method is shown in Equation 11. Here IND(FE) and IND(r)) indicates the model indices sets of the top VLMs in r.g and ry, respectively. A Top 5 Recall closer to 1 signifies greater accuracy in the predicted rankings.
F IND(FE) n IND(r
(10)
F R₂
(11)
5
15

---

Table 7: The detailed information of 43 models used in the LOVM Benchmark. Some of the
information in the table comes from [73].
Model
Name
Dataset
Name
RN50
RNSO
openai
WIT
RNIO1
openai
RN50x4
RN50x4
openai
WIT
RN50-16
16
openai
RN50x64
RN50x64
openai
WIT
VIT-B-12
VIT-B/32
Iaion-400m_ei
L400m
VIT-B-32
B/32
laion400m_=32
L400m
VII-B-32-quickgelu
VTF-B/32
Ision:400m_e33
L400m
VTT
VIT B/32
openai
WIT
10
VIT-B-32
VIT-B/32
1aion2h <34b, b79k
L.bbb
BTT--32 32
VIT B/32
laion2b _66
1.2b
12
VIT-B-16
VIT-B/16
laion400m_ec3
1.00mm
13
VIT-B-16
VIT-B/16
openai
WIT
14
VII-B-16-240
VII-B/16-240
laion400mm_e32
L400mn
VIT-L-14
VIT-L/14
Iaion400m_ee _31
L400m
16
ViF-L-14
VIT-L/14
aaion4000m_c32
L400m
VITL-114
WIT-L/14
Iiionhh _0____322___1____
18
VIF-L-14
VIT-L/14
openai
WIT
VILL
VIT-L/14-336
openai
WIT
20
VIT-G-14
VIT-G/14
1aion2h 110h b42k
_
1.2h-a
VIT-G-14
VIT-G/14
laion2b ___4333bb_bb b88k
L2b-a
22
VIT-H. 14
VIT-H/14
laion2h <32h b79k
1.2b-b
coca VTT-B-32
CoCa-ViTE VTT-B22
laion2b _lb3__b00kk
L2b-c
VYT-B-32
CoCa- CoCa-ViT-B/S3 VTT-B/32
mscoco
finetuued_ piecunee_aiin22o_s33_.o0
L2b-c coco
coca VIT-L-14
CoCa-ViT-L/I4
laion2b _13k_b90kk
L2b-c
26
VIT-L-14
CoC3-ViT-L/14
mscoco
finetuned_ laco:edd__ioo________ st3___sssbb_boeeee
L2b-c +
convnext base base
Con-NEXT-B
laion400ml==133
L400m-cc
CONTNEXX-BWW
laion2h <13b b82k
1.2b-l
convmext base
COw-NEXT-BW
bhionb_ss3bb_bat b82k _uggee
L2b-e
base
CON-NEXX.BW
laion aain_aesthetii__gg _1., h82k
1.2b-F
convnext base w
Com-NEXT-BW-520
laion cei__aasshttccceee s13b b82k
L2b-f
32
convnext hase
CowTTEXX18
laion aesthetic 313b b82k augreg
1.2b-g
33
convnext_large large_ree_____
Con-NEXT-LD
lnion2h, s26b (1166__10000 b102k augreg
L2b-h
convnext_aagee large 320
Cou-NEXX-DDD320
laion2b b_33k__11_______
L2b-i
35
convnext largedd_ge -20
Cow-NEXT-LD-522
laiun2b 22____9bbbbl31. soup
L2b-j
pet.reetiilll.... settivval_base_cooo bas__baasee________eee__ee__eeoe
BLIP retrieval base
COCO=>G-CCCCCI2MM--
BLIP-Dataset
37
BLIP retrieeerll base b30k
BLIP retrieval base_130k
BLIP Dulasett
large enco
retrieval large cccco
COCO-VVH-CCCCCI2M--ST
39
BLIP retrieealee Jarge_[30k
BLIP pettieeal large_I306
BLIP Dulaset
hase, cocn
retrieval hase coco
CCI2M-CCMM+SRR--COCOHY
41
BEIT-3 retrreeeeevvlll.. base
BEYT-3 "e__eerreeva__beee base_f30k
CCI WWW.CCCM-SS---OOOO--G-------
BETT- Dataset-f
BETT-3 retrieveeevvll...ee Jarge_ coco
BEiT-3_rettivve ret.rrttivaalleee Jargo coco
CCI2MHCCM-sBBU-COOONNGG
BEiT-Dataset
43
BEST-33_errieval__ Laa___larre___0
BEIT-3rretrieval Sestrieval_larg__eeee f30k
CC 2MM-CCM-BBU-CCO0OOGG---
BETT- -att-Dataset-F
Table 8: Detailed information of 23 tasks used the LOVM Benchmark. This table comes from [73].
Dutaset
Classes
Task
Domain
Imagenet [8]
1000
classification
natural image
SUN397 [60]
397
scene und.
natural image
Country211 [46]
211
geolocation
natural image
Stanford Cars [25]
196
classification
natural image
Flowers 102 [42]
102
classification
natural image
CIFAR100 [26]
100
classification
natural image
DTD [5]
46
classification
textural image satellite images natural image natural image
RESISC45 [4]
45
classification
GTSRB [50]
43
classification
Oxford Pets (44)
classification
VOC2007 [10]
20
classification
natural image natural image satellite images
STL10 [6]
10
classification
BuroSAT [19]
classification
MNIST [27]
classification
hand-writing
SVHN [41]
10
OCR object counting distance est. fac. exp. rec. distance est. classification
natural image
CLEVR-C [23]
8
natural image
CLEVR-D [23]
8
natural image
FER2013 [16]
7
natural image
DMI ab [69]
6
synthetic
Retinopathy [24]
5
retina scan natural image
KITTI [15]
4
distance est. classification OCR
PCam [58]
histopathology text image
Rendered SST2 [46]
16

---

Kendall's Rank Correlation (r) Kendall's Rank Correlation 7 measures the ranking consistency between two ranking lists. We follow LOVM [73] to focus on the VLMs within the intersection of the top 5 VLMS in rr and rTy and use to evaluate a model selection method's capability.
B ModelGPT Details
ModelGPT is a method proposed by LOVM [73]. In this section, we introduce the metrics that ModelGPT used in Equation 3.
B.1 The Generation Process of Auxiliary Text Samples
ModelGPT [73] utilizes ChatGPT [43] to generate auxiliary text data by designing prompts to query ChatGPT. This extra text data mainly includes the Captions Dataset and the Synonyms Dataset.
Captions Dataset.
ModelGPT uses the following prompt guide LLM generate realistic and confusing text data corresponding to the user-provided classes. The reason for requiring ChatGPT to generate confusing texts is increase the classification difficulty of the text data, thereby enhancing its ability to distinguish the performance of different models.
Generate long and confusing image captions for the (domain) domain, which will be used to evaluate Vision-Language Model's (task) performance. Generate 50 captions for [classname):
We show some generated auxiliary text examples. For example, the category of dog, one of the text samples generated by ChatGPT is "nn adorable dog perfect for cuddles and playtime." ModelGPT collects the results from this prompt to form the captions dataset, cap
Synonyms Dataset. ModelGPT uses synonyms to evaluate VLM's text encoder. For example, we expect an excellent VLM to extract similar embeddings for the words "chair" and "seat". The prompt to guide LLM to generate synonyms is as follows.
Please list the superclaases/synonyys for [classname). For example:
chair: [furniture, seat, bench, armchair, sofa] [classname):
ModelGPT collects the results from this prompt to form the synonyms dataset, D
B.2 Text-Derived Scores
ModelGPT uses six metrics for model selection, which can be divided into Text Classification scores and Dataset Granularity scores. Text Classification scores include the Text top-1 accuracy score and Text fl -core. While Granularity scores include the Fisher criterion, Silhouette score, Class Dispersion score and Synonym Consistency score. Here we focus on introducing the various metrics included in the Granularity scores. We refer to the relevant content in LOVM.
Fisher Criterion dissber- The Fisher score measures the closeness of VLM's s text classifier one another. Equation 12 shows the calculation process of it where é, is the text classifier of the i-th class derived using the prompt ensemble strategies proposed in [46], ((, .) is function that calculates the cosine similarity between two vectors, and |C| is the number of classes.
1 |C|
Ørisher
max, i [0(E, f/)]
(12)
Silhouette Score Psl- The Silhouette Score measures the separation of different-class samples in the caption dataset Deap To calculate it, ModelGPT averages the cosine similarity of captions to the nearest other class's classifier by:
1
Pai
max;
O Gike Gle.
(13)
N

---

where f; is the text classifier of the i-th class derived using the prompt ensemble strategies proposed in [46], -(:-) is function that calculates the cosine similarity between two vectors, and [C] the number of classes. Dep[j]; representing sample & of class j in the caption dataset Deu, There is total of N such samples for each class.
Class Dispersion Score Pdisp- Class Dispersion Score quantifies the degree of same-class tightness or data cone radius, which is calculated using the following Equation:
IC
N
1
Plisp
CN
0 [1),
(14)
The definitions of all symbols in Equation 14 are consistent with those in Equation 13.
Synonym Consistency Score Thya: Synonym consisteney allows us evaluate the degree of content shift between the VLMs' pre-training and target dataset. The calculation process is shown as follows:
ICl N
1
Y'syn
Σ
O(D** (... E.)
(15)
C
The definitions of f,, ((:-). |C] and N in Equation 15 are consistent with those in Equation 13. [ representing sample l of class in the synonym dataset Dayn
C Related Work
Vision-Language Models. Deep learning has achieved competitive performance [39, 74, 30] and the vision-language model is new research hotspot of it. Vision-Language Models represent class of multimodal models adept at correlating textual and visual information. These VLMs can comprehend rich semantics and possess strong generalization capabilities. Hence, they are often used in data-limited scenarios [61, 29]. Many VLMs are pre-traines or fine-tuned on extensive text-image pairs using loss functions such as contrastive loss, endowing them with powerful text-image matching capability. Prominent VLMs include CLIP [46], ALIGN [22]. FLAVA [48], Florence [68], and CoCa [67]. These VLMs possess robust zero-shot image classification capabilities [46], which enables its widespread application in tasks characterized by long-tail distributions or those where collecting substantial training data is challenging, such medical image analysis. Some work [40, 64] has also shown that by incorporating external knowledge, the zero-shot capabilities of CLIP can be further enhanced. In recent years, the number of open-source VLMs has been increasing [21]. Previous work [73] has pointed out that different VLMS possess varying image classification capabilities. This indicates that the performance of VLMs can vary significantly across different tasks and domains. These models with diverse capabilities constitute a VLM Zoo rich in knowledge. This VLM Zoo enables us to utilize different VLMs for various classification tasks, thereby changing the paradigm of using single VLM to complete diverse classification tasks. This paper focuses on selecting the most suitable VLM for the target task from the VLM Zoo.
Model Selection. In recent years, how to select the most suitable model for the target task from the model zoo has received widespread attention. For example, a series of works learwware [53, 52, 71, 72, 33, 51, 54] have attempted to solve this problem. The essence of the model selection problem lies in measuring the transferability of the model the target task. Previous model selection methods [55, 56, 65, 66, 9, 20] evaluate the PTM's transferability by performing a forward pass of the PTT on the target task's data and calculating the metric of transferability based the forward pass's 's result. For example, H-Score [2), NCE [55], LEEP [56), LogME [65] estimate transferred ligelikel, negative conditional entropy, log expectation, marginalized likelihood to obtain proxy metric of transfcrability, respectively. Some new methods, such as Task2Vec [1] and Model Spider [70], generate representation vectors for both the models and the tasks and measure the transferability of the model to the task by calculating the similarity between these vectors. However, VLMs are typically used in zero-shot or few-shot scenarios, where we are unable obtain a large amount of data for the target task in advance, making traditional model selection approaches unsuitable for VLMs. Additionally, previous methods have mainly focused on single-modal models, overlooking the characteristics of VIMSs. Therefore, in this paper, we concentrate on designing model selection algorithms that are suitable for scenarios with limited data and take into account the characteristics of VLMs.
18

---

D Implementation Details of SWAB
In this section, we provide some details on the implementation of SWAB, which are not mentioned in the main text due to space constraints.
D.1 Filtering the Open-Source Tasks' Classes
The statistics of classes unrelated to the target class generally not valuable for reuse. Meanwhile, when the number of classes |Cs] in open-source datasets S is large, solving the optimal transport problem in Equation 5 can be time-consuming (as current optimal transport toolkits generally compute via CPU). To reduce the runtime of optimal transport, we can first filter the classes Cs- Consider that only statistics of classes relevant to the target dataset are helpful. Therefore, we can filter out the classes in C's that are irrelevant to the target dataset T based on the class-level textual semantic similarity between the open-source datasets' classes and the target dataset's classes. This process shown in the following Equation:
( (c))
S = | [[(())
(16)
= max(S,.) }}.
(17)
Here St, refers to the i-th row of the semantic similarity matrix calculated using Equation 16, which represents the vector formed by the similarity between the i-th class cf in Cs and each class CT = {cT ... o } of the target task. À is threshold and we set = 0.5. k's refers to the number of classes in the filtered set C's- Then use the filter classes C's to calculate the transport matrix 7 E R xky and continue with the following steps.
D.2 Using Partial Optimal Transport for Bridging the Capability Gap
Partial optimal transport extends the optimal transport framework, enabling the selective transfer of elements from a source to target distribution, rather than moving all elements. Its optimization problem defined as in Equation 18. Here mass refers to the total amount of mass actually be transferred. We set mass = 0.9 in our implementation.
Y* argmin Σ i.
Ki cost.
*a x
(18)
s.t. 71 S u: 7 S Mi 2 0; 1 17 mass < min [|/u||2,110/1
We found that when using Equation bridge the Capability Gap, the transport matrix y obtained using partial optimal transport yields better results than the one obtained using the original optimal transport via solving the Equation 5. Therefore, in our implementation, we use the transport matrix derived from partial optimal transport to bridge the capability gap. This also indicates that when estimating VLM's statisties on the target dataset, different types of statistics have different preferences for the estimation methods used. This variability is worth further investigation.
D.3 Data Normalization in Bridging the Modality Gap.
When bridging the Modality Gap described in Section 3.2, we find that applying 7-score normal- ization to the text and image features used in this process yields better results. Therefore, in our implementation, normalize the features of all text and image samples during the modality bridging process using the following Equation:
(19)
Here x E TRd represents the image sample's 's or text sample's feature, while μ Rdd and a E IRd are calculated using the features of all samples of the same modality within its respective dataset.
19

---

Algorithm SWAB
1: Input: Target dataset's class names CT, open-source datasets' class names Cs. open-source datasets' images D
2: Use ChatGPT to generate auxiliary text data Ds and Dy based on Cs and Cr.
3: Calculate VLM's class-level zero-shot image classification rankings (1mm)) (n,im=1iim) and class-
level gap vectors { - mi
4: for kş datasets do
5: Calculate textual similarity between the current dataset's class names and other open-source
datasets' class names construct cost matrix.
6: Solve Optimal Transport Problem based on the cost matrix to get transport matrix Y",
7: Use > and other open-source datasets' class-level gap vectors to predict the current dataset's class-level gap vectors, Add the predicted class-level gap vectors to the corresponding text data's 's feature of Ds to get modified text data.
8: end for
9: Calculate textual similarity between open-soures datasets' class names Css and target dataset's
class names Cy to construct cost matrix.
10: Solve Optimal Transport Problem based on the cost matrix to get transport matrix 7".
mm mii,iikT of the target 11: Use " and (g m= Mii=kg to predict the class-level vectors (0 J m=
m=1,i=1
dataset. Add (9 maaimmm1l m=1,-=1 to corresponding text data's feature of Dy to get modified text data.
MiMMi
12: Use open-source datasets' modified text data and VLMs' ground truth zero-shot image classifica- tion performance to train the ranker model fm-
Use the ranker model fm to predict VLMs' rankings (TTT.(1 p==M on the target dataset based the target dataset's modified text data.
13:
Jm=1
m=1,ie1 Use 7 and (m,,)) m=M,i=ks to predict the VLMMs class-level zero-shot image classification rankings mm,)m=m of the target dataset.
m=M,ieky
15: Calculate the average prediction rankings across all classes in the target dataset (fT T.()) m= of (AT (fm,) fm.iim==,,e
Mitkk for each VLM as the VLM's overall prediction ranking on the target dataset.
Ensemble (fT.11p (1 and (+T,(2) m=m to get final predicted rankings (AT ens mammm m=M
[
D.4 Pseudo Code of SWAB
Algorithm shows the pseudo-code of SWAB.
E More Experiment Results
In this section, we provide more experimental results of SWAB.
E.1
Bridging the Modality Gap Leads to Better Image Proxies
In Section 2.3 and Figure 2, we analyze whether generated text data Our conclusion that due to the Modality Gap, text samples cannot directly serve as an effective substitute for images in model evaluation. To demonstrate that our method SWAB can bridge this Modality Gap and thereby make text samples a better substitute for images, we conduct the following experiment.
good image proxies.
From the Figure 4, evident that the predicted model accuracy calculated using the modified text samples is closer to the true model accuracy compared to that calculated with the original text samples. This suggests that bridging the Modality Gap leads to better image proxies.
We use ImageNet as our dataset. First, we employ the method introduced in Section 3.2 to predict gap vectors for each class of the target dataset based on gap vectors calculated on open-source datasets. Then, we add the corresponding class's predicted gap vectors to the generated text data of ImageNet to bridge the modality gap. Finally, we calculate the zero-shot classification accuracy of different models these modified text data. To measure the consistency between the predicted Top-1
20

---

Text Samples
Modified Text Samples
ImageNet 50-shol Text Samples Per Class
ImageNet////// 50-shot Text Samples Per Class
Ground-Trhh Accuray
Ground Sruts Accraayy
t= 0.51
MAE 0:05
0.4
0.21
84-
MAE 0.32
0.2
02
0.5 0.6
Prodicted Accuracy
Predicted Acturacy
Figure 4: Comparison of the consistency metrics between the accuracy calculated using text data before and after bridging the gap and the model's accuracy. After bridging the modality gap, the text data act as better substitutes for image data to evaluate the model's performance.
accuracy and the true image classification accuracy, we calculate the Kendall Rank Correlation (r, higher is better) and Mean Absolute Error (MAE, lower is better). We compare the consistency metrics of text data and modified text data. It can be observed that the consistency metrics of modified text data are better, which proves method can reduce the gap between the generated text data and the image data.
E.2 Analysis of the Modality Gap in BLIP and BETT-3
We expand the VLM Zoo provided by LOVM by adding various variants of BLIP and BEiT-3 to enhance the diversity of the VLM Zoo. We find that the zero-shot image classification performance of the base models of BLIP and BEiT-3 is poor. Therefore, models provided officially. These models are obtained by fine-tuning the base models on COCO and Flickr30k through contrastive learning, thereby possessing better zero-shot image classification performance. During the pre-training of BLIP and BEÎT-3, they may have used multiple loss functions in addition to the contrastive loss (BLIP), perhaps they did Therefore, the existence of the Modality Gap in the feature spaces of these two models requires further investigation.
the retrieval version of the
contrastive loss all (BEiT-3).
We use image samples from the STL-10 dataset and class-related text samples generated by ChatGPT. For each modality, we randomly extract 200 samples and calculate image-to-imane text-to-text, and image-to image-to-tex similarities, and plot histograms. We also observe the presence of Modality Gaps in different models through UMAP visualization [38]. Figure 5 and Figure 6 show the experiment result. Through our experiments, we verify that the feature spaces of the BLIP and BEÎT-3 retrieval models still exhibit the Modality Gap phenomenon.
PLPPE YING
Figure 5: The distribution of image-to-image (i2i) cosine similarity, text-to-text (t2t) cosine similarity, and image-to-text (12t) cosine similarity values for different BEÎT-3 and BLIP models.
21

---

Figure 6: UMAP visualization of image sample features and text sample features from different
BEiT-3 and BLIP models.
E.3 Experiment Result LOVM's original VLM Zoo
We have provided experimental results on the VLM Zoo originally provided by LOVM [73]. This VLM Zoo contains 35 VLMs. Table 9 shows the experimental results, and we can see that SWAB achieves the best performance across evaluation metrics.
Table 9: Results on LOVM'S original VLM Zoo. We evaluate our method across 23 datasets and 35 pre-trained VLMs. The results are averaged over all datasets.
Methods
H-Score
NCE
LEEP
LogME
INB
Avg Rank
ModelGPT
SWAB
Rs(t)
0.183
0.235
0.139
0.200
0.443
0.443
0.446+0.004
=0.005
r(t)
0.000
0.000
0.014
-0.014
0.267
0.261
0.27210.010
0.310
Rg r(t)
0.183
0.235
0.153
0.186
0.710
0.704
0.718+0.013
0.801
E.4 Per-Dataset Experiment Result
We present the per-dataset performance comparison between our methods SWAB and ModelGPT on various datasets of LOVM benchmark in Table 10.
Table 10: LOVM Benchmark (top-1 accuracy). We compare the per-dataset experiment result (fixing the random seed) between ModelGPT and SWAB.
Sunford Cans
CLEVR- DIEVR-DIST DIST.
CLEVR-COUNT
Rendered SST2
CIFARIOO
Countr211
Reiinopuhty
DMLab
BUnSAT
3ER2013
Floerrs 102
ImageNet
MNIST
Oxfond Pess
RESISCAS
SUN397
SVHN
VOC2000
Mean
SWAB
0.80 0.80 0.00
0.60
0.10
0.20
0.00
0.80
0.60
0.60
0.80
0.60
1.00
0.00
0.40
0.00
0.50
0.20
1.00
0.20
0.60
0.60
0.60
0.504
ModelGPT
0.67 0.67 0.00 0.00
.0.33 0.00
0.00 0.67 1.00
0.00
0.67
1.00
0.67 0.00
0.00
0.00
0.00
0.00
0.67
0.00
11.00
0.33
1.00
0.261
0.320
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer the bullet point summary at the end of the Introduction.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification: We did not find significant drawbacks in the method.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work does not involve theoretical result.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide our experiment details in Section 4.1 and Appendix
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open-source our code, models, and datasets in the GitHub repository.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide these details in Section 4.1.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: To ensure the reliability of our results, we conducted multiple experiments for results with randomness and reported the mean performance and standard deviation Table 1.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: We will place the relevant information in the GitHub repository.
9. Code Of Ethics
23

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:// http::///earpps...ppbii///iiiiii http:///ll..//cccccccccooooooooa
Question:
Answer: [Yes]
Justification: We have read and fully comply with the Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work does not involve negative social impacts.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not involve these situations.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We ensure this requirement met.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not involve assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing and research with human subjects. 15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human
Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve this.
24


Reviews:
[{"review_id": "0nD6uQ7oh3", "reviewer": "Reviewer_qHTS", "summary": "This article focuses on selecting the best model from a visual-language multimodal model zoo for specific downstream tasks without having images of those tasks. It provides a detailed analysis of two challenges faced in this problem — Modality Gap and Capacity Gap. This paper proposes a method called SWAB, which leverages the statistical measures of VLM on open-source tasks based on the category similarity between historical open-source tasks and target tasks, to mitigate the negative impacts of these gaps on Language-Only VLM model selection. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "Refer to the weakness.", "limitations": "N/A", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The problem studied in this paper is novel and interesting, with significant practical value. In recent years, the number of open-source VLMs has been increasing, and their zero-shot capabilities have been improving. For practical downstream users, especially those who lack computational resources, have limited funding to label extensive evaluation datasets, or are non-machine learning practitioners with little experience in setting up and evaluating various VLMs, the Language-Only Vision Model selection (LOVM) problem studied in this article is important.\n2.This paper presents a clear and well-motivated problem statement. The authors provide a detailed analysis of the negative impacts of the Modality Gap and Capacity Gap on Language-Only Vision Model (LOVM) selection through explanations and experimental validation. They demonstrate that we cannot directly use text samples related to categories in the target dataset as substitutes for test image samples, nor can we use the average performance of VLMs on open-source datasets as proxies for their performance on specific tasks. This problem statement is compelling and well-supported.\n3.The method presented in the paper is novel and interesting. It leverages the category similarity between target tasks and open-source tasks to estimate the corresponding gap vectors and accuracy/ranking of VLMs on the target tasks, based on their gap vectors and accuracies/rankings on the open-source tasks. This idea is grounded in a reasonable assumption: the statistics of VLMs on similar categories are likely to be similar. By doing so, the method effectively utilizes the rich information contained in category similarity to make targeted use of the models' statistics on historical tasks.\n4.The evaluation approach in the article is comprehensive and detailed. The benchmark includes 23 commonly used image classification datasets and 43 widely recognized VLMs (such as CLIP, CoCa, BLIP, BEiT, among others, with different pre-training methods and model architectures), making the experimental results highly convincing. Additionally, the article compares its method with various baseline approaches, including the current state-of-the-art method ModelGPT, and demonstrates its significant superiority over these methods. The article also conducts thorough ablation studies to validate the effectiveness of each component of the proposed method.", "weaknesses": "1.The performance of ModelGPT is slightly inconsistent with the results in its paper. What are the reasons for this？\n2.The method of setting the \\alpha value in formula 9 is not stated. Is the same \\alpha used for all data sets? What is the exact value?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers' recognition of our work. Below are our responses to the relevant questions:\n* Q1: The performance of ModelGPT is slightly inconsistent with the results in its paper.\n* A1: This is because we conducted multiple repeated experiments using different random seeds. ModelGPT added Gaussian noise to perturb the features of the text samples. To enhance the reliability of the results, we used 10 different random seeds for repeated experiments and reported the mean and standard deviation of the results. Since the original ModelGPT paper did not perform repeated experiments, our results show slight differences compared to their paper.\n\n* Q2: The method of setting the $\\alpha$ value in formula 9.\n* A2: $\\alpha$ is a hyperparameter. We used the same alpha value of 0.45 for all experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BMTyKsoDWJ", "reviewer": "Reviewer_NUg2", "summary": "The paper proposes SWAB the modificaiton of LOVM to mitigate the negative impact of two gaps of LOVM: the modality gap and capability gap. The resutls show the effectiveness of SWAB.", "questions": "1. Why adding Gaussian noise to the target generated embedding?\n2. How to select weighted parameter $\\alpha$ in Eq.9\n3. The Auxiliary texts is generated by ChatGPT, could you please test the open-source LLM like LLama2?", "limitations": "No.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is clear and structure is easy to follow.", "weaknesses": "1. For the modality gap, the SWAB using open-source datasets, could you please provide how many image samples used in the experiment. And could you please provide the accuracy vs number of open-source image samples?\n2. The SWAB seems like a learnig-based model, could you please add the ablition study for no learning based SWAB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' comments. Below are our responses:\n* Q1: How many image samples were used in the experiment.\n* A1: We followed LOVM's approach [1] to use 23 datasets.  Each of the 23 datasets was used as a test dataset in turn, with the remaining 22 datasets serving as open-source datasets. Therefore, the size of the open-source datasets varied for different tasks. The largest open-source dataset contained 497,123 images, the smallest contained 389,080 images, and the average number of images was 476,189. \n\n    It is important to emphasize that the previous SoTA method, ModelGPT [1], also used the same open-source datasets as we did. \n\n* Q2: Provide the accuracy vs number of open-source image samples.\n* A2: We conducted additional experiments by randomly sampling images from the original open-source datasets at different ratios to create new open-source datasets and applied our method based on them. We use the same sampling ratio for each category. We repeated the experiment 5 times and show the averaged the results. We found that the variation among different experimental results was small. The results are as follows:\n    | Data Ratio | 40%   | 60%   | 80%   | 100%  |\n    |------------|-------|-------|-------|-------|\n    | $R_5$       | 0.508 | 0.519 | 0.525 | 0.534 |\n    | $\\tau$       | 0.247 | 0.256 | 0.253 | 0.260 |\n    | $R_5$ + $\\tau$  | 0.755 | 0.775 | 0.778 | 0.794 |\n\n    The result shows that our method is robust to the number of data in the open-source datasets used. This is because the VLM's average gap vector of each category and  the VLM's ranking of each category are robust to data volume. As shown in Table 5 in our paper, gap vectors for different samples within the same category are consistent. Additionally, reducing the dataset size has minimal impact on model rankings.\n\n* Q3: Add the ablation study for no learning based SWAB.\n* A3: Firstly, we emphasize that our method focuses on bridging the modality gap and capacity gap in VLM selection. In achieving these goals, we do not introduce any learning process. In SWAB, only the Rank Model in the third step depicted in Figure 3 of our paper requires training. And this Rank Model is introduced by the ModelGPT we follow. We use our  SWAB to optimize the ModelGPT's capabilities by bridging the modality gap.\n\n    Regarding the reviewers' concerns about the improvements of our method over non-learning based methods, the corresponding results are shown in Table 3 of our paper's main text and Figure 5 in the appendix.  The results shown in both table 3 and figure 5 do not use the rank model, and therefore, no training process is introduced:\n    * In Table 3 of the main text, We demonstrated the performance improvement in selecting the VLM by using the weighted average of VLM's ranking on categories of open-source dataset with SWAB to bridge the capacity gap, compared to using the average VLM's ranking of various categories of open-source data directly. \n    *  In Figure 5 of the appendix, we separately show the top-1 classification accuracy calculated on text samples before and after using SWAB to bridge the modality gap between text and image data. This demonstrates a better rank consistency between the text top-1 classification accuracy of VLMs and actual classification accuracy of VLMs after using our SWAB.\n\n* Q4: Why adding Gaussian noise to the target generated embedding?\n* A4: We added Gaussian noise to simulate the variance in image samples because different images can correspond to the same caption. For example, in the text \"A dog running on the grass,\" the dog in the images can vary in breed. We add Gaussian noise to the features of text samples to simulate the diversity of images. We found that adding Gaussian noise improves the performance of our method. Some existing works [1, 2] have also adopted similar approaches.\n\n* Q5: How to select weighted parameter in Eq.9.\n* A5: $\\alpha$ is a hyperparameter. We used the same alpha value of 0.45 for all experiments.\n\n* Q6: Test the open-source LLM like Llama2 to generate the Auxiliary texts.\n* A6: We used Llama 2 to generate the corresponding text and found that our method performs excellently with texts generated by different LLMs. We attribute this to the fact that generating category-specific diverse texts is a relatively simple task for LLMs, and common LLMs possess this capability.\n| Metrics | $R_5$  | $\\tau$ | $R_5 + \\tau$ |\n|---------|------|------|----------|\n| Llama2  | 0.527| 0.254| 0.781    |\n\n[1] Zohar, Orr, et al. \"Lovm: Language-only vision model selection.\" NeurIPS 2023.\n\n[2] Zhang, Yuhui, et al. \"Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data.\" ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bqsn2ZC6cZ", "reviewer": "Reviewer_omtt", "summary": "With the popularity of Vision Language Model (VLM) research in recent years, many versions have emerged, forming the VLM Zoo. This paper aims to select the most appropriate pre-trained VLM from the VLM Zoo, relying solely on texts of the target dataset without access to images. Two challenges are analyzed, namely, “Modality Gap” and “Capability Gap”. One VLM selection method (SWAB) is proposed: first calculate optimal transportation matrix to capture the relevance between open-source and target datasets; then use this matrix to transfer useful statistics of VLMs from open-source to target dataset for bridging two gaps and enhancing the VLM’s capacity estimation. Experiments are carried out across some VLMs and classification datasets to validate the effectiveness.", "questions": "[-] Better visualization. For Fig. 3, there are too many symbols and formulas, increasing difficulty in understanding (the significance of using images is to reduce understanding costs). To this end, the reviewer believes that this paper can be polished better.", "limitations": "This paper does not discuss the limitations of the work. However, the reviewer seems to find some limitations, such as the unrealistic assumptions in settings and the ability to only handle classification tasks. Please refer to Weaknesses for more details.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "[+] Due to the differences in training data, architecture and pipelines, existing VLMs do have their own strengths in terms of capabilities. Thus, studying how to better utilize members in VLM Zoo under different scenarios is somewhat valuable.\n\n[+] The paper has rich symbols and formalizations, and the space utilization is also reasonable.\n\n[+] Some experiments are conducted to demonstrate the effectiveness of components and designs on a diverse set of datasets of image classification.", "weaknesses": "[-] Problem rationality. Although each member in the VLM Zoo has their own strengths, relying solely on textual descriptions to find the most suitable VLM (LOVM) seems too demanding. And language is usually compact, which can easily confuse concepts. Why cannot we use one combination of images and texts (or even text-image pairs) for VLM selection, as images are very easy to obtain in real-world scenarios? \n\n[-] Selection strategy. Given that the members in VLM Zoo have different strengths and abilities, it means that they have somewhat degree of complementarity. One natural idea is to follow ensemble learning, that is, vote on the results of each member, or select the top-k VLMs’ results for complementary combination. Why only choose the most suitable one? Please provide more comparisons and explanations. \n\n[-] Generalization. This paper carries out experiments on many datasets, but all for image classification. Since VLMs excel in handling various tasks, such as detection, segmentation, captioning, and VQA, the generalization on these tasks is still unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's comments. Below are our responses:\n* Q1: Why not use combination of images and texts (or even text-image pairs) for VLM selection？\n* A1: \n\n    1. Firstly, we want to emphasize that in our paper, \"Language-Only\" means that for the target new task's dataset, we can only obtain the text descriptions (e.g. category names) of the target task. And we can't obtain the images of the target task. This has been explained in lines 29-31 of our paper. **But for the open-source datasets, we did use the image-text pairs mentioned by the reviewer.**\n \n    2. Secondly, we follow the setting proposed by LOVM [1] in NeurIPS 2023, rather than the setting we devised ourselves. This setting has already been peer-reviewed and validated. \n   \n    3. Thirdly, benefiting from image-text pre-training on large-scale data, VLMs represented by CLIP have strong zero-shot capabilities. Therefore, users naturally expect to obtain a model with strong zero-shot capabilities without providing image data. And this aligns with our \"Language-Only\" setting. This highlights the rationality and practical significance of the setting in our paper.\n\n  4. Finally, we want to emphasize that generating diverse text with LLMs is simpler and more cost-effective than collecting precisely annotated diverse images. To obtain accurate evaluation results, the test image samples need to be sufficiently diverse and accurately labeled. However, collecting such diverse and precisely labeled image samples is quite challenging and may require significant human and financial resources. Images or image-text pairs scraped from the web can be extremely noisy, as pointed out in the CLIP[2], ALBEF[3], and BLIP[4] papers. And as mentioned in the introduction of our paper and by reviewer qHTS, collecting these images is even more difficult for some downstream users, such as non-machine learning practitioners. What they want is to input their task description into a search box, and have a model selection algorithm return the suitable VLMs for their task.\n\n  **Compared to collecting images, generating diverse, high-quality text with LLMs is relatively easy. This opinion is also mentioned in some work[5]. Our paper further analyzes the critical issue of \"how to make the diverse text generated by LLMs better substitute for diverse images\" and provides a solution, thereby demonstrating significant value and importance of our work.**\n\n* Q2: Why only choose the most suitable VLM? \n* A2: \n\n    1. First, we want to emphasize that our method focuses on model selection, specifically providing an accurate estimate of the performance ranking of VLMs in the VLM Zoo for a target task. This performance ranking estimate helps users make better use of the VLM Zoo for their target tasks, whether selecting the single best model or choosing the Top-K models for an ensemble. In fact, the reviewer's question is concerned with \"how to use the VLM Zoo based on the obtained ranking list,\" while our method focuses on \"how to accurately estimate the performance ranking of VLMs in the VLM Zoo for the target task.\" These are different points of focus. \n\n    2. Secondly, our article prioritizes selecting a single best model mainly because the inference cost of a single model is smaller compared to an ensemble of multiple models, aligning better with the reality that most downstream users lack sufficient computational resources. However, users can still use the performance ranking list provided by our method to select the top-K models for ensemble. We reiterate that the ranking list estimates provided by our method can universally assist various uses of the VLM Zoo, including using a single model or an ensemble.\n\n* Q3: Why only focus on the image classification task?\n* A3: \n\n    1. Firstly, as mentioned in LOVM[1], the reason we do not primarily focus on tasks such as segmentation and object detection is that these tasks usually require the introduction of additional model structures and training processes on top of the original VLM pre-trained model. This leads to additional noise when evaluating VLM performance (for example, using different model fine-tuning methods may result in completely different VLM performance rankings, making it difficult to determine the true ranking of VLMs on the target task). Therefore, our experiments mainly focus on zero-shot image classification tasks.\n\n    2. Secondly, our SWAB is not specifically designed for image classification tasks. For the Modality Gap, our method aims to reduce the distribution differences between different modal data. For the Capacity Gap, our goal is to use the model's ranking in related historical tasks to predict its ranking in new target tasks. These designs are not tied to image classification and can be extended to other zero-shot tasks. For example, in zero-shot image-text retrieval tasks, we can use our SWAB method to better utilize text samples as substitutes for image samples for model evaluation.\n\n    3. Thirdly,  zero-shot image classification tasks are highly representative applications for VLMs and are among the most important and valuable tasks for VLMs. We believe that the excellent model selection results of our method in this task already demonstrate the significance and effectiveness of our approach.\n\n [1] Zohar et al. Lovm: Language-only vision model selection. NeurIPS 2023.\n\n [2] Radford et al. Learning transferable visual models from natural language supervision. ICML 2021\n\n [3] Li et al. Align before fuse: Vision and language representation learning with momentum distillation. NeurIPS 2021.\n\n [4] Li et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML 2022.\n\n [5] Zhang et al. Diagnosing and Rectifying Vision Models using Language. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iy2Y4ZI0e5", "reviewer": "Reviewer_mpun", "summary": "This paper considers a zero-shot image classification strategy by selecting the most appropriate Pre-Trained VLM from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. Two challenges, i.e., the “Modality Gap” across two different modalities, and the \"Capability Gap\" between the VLM’s overall ranking and its ranking for target dataset, hinder the appropriate selection of the different VLMs. To address the challenges, this paper adopts a transportation matrix to capture the relevance between open-source and target datasets, and also uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset. Extensive experiments on image classification datasets demonstrate the effectiveness of the proposed method.", "questions": "A comprehensive analysis on the impact of the open source datasets may improve the rationality of the proposed method for addressing the zero-shot image classification task.", "limitations": "This paper does not include the limitation discussion. Practically, at least the issues raised in the weakness section should be considered if they cannot be addressed appropriately in current version.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: \nThe main novelty of this paper is to utilize Language-Only VLM Selection to address the zero-shot image classification. This is the first time that the Language-Only VLM Selection is applied in the zero-shot scenario where the missing of image data makes it difficult to directly select the VLMs.\nQuality: \nThis paper has well motivated methods and extensive experiments.\nClarity: \nThe paper is well written with clear description and readable figures.\nSignificance: \nThe two key issues of modality gap and capability gap are considered in this paper. In addition, optimal transport-based methods are proposed to alleviate the issues and the experimental results on multiple VLMs and image classification datasets show the effectiveness of them.", "weaknesses": "The main weakness is that the proposed VLMs selection strategy relies on Open-Source Datasets’ Data, which restricts the practicability of the proposed method. For the zero-shot image classification task, using the aggregated knowledge from multiple VLMs is ok, but using open-source datasets may violate the zero-shot data restrictions, or even lead to benchmark leakage. In addition, the impact of the open-source datasets is not analyzed. For example, if the categories in the open-source datasets are very different from the ones in the target zero-shot classification tasks, whether the performance of the proposed method is still remarkable?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's comments. Below are our responses:\n\n* Q1：Use open-source datasets may violate zero-shot data restrictions.\n* A1: \n\n    1. Firstly, we want to emphasize that our method focuses on how to use open-source datasets to help us select the appropriate models from the VLM Zoo for the current task, rather than using open-source datasets to enhance the zero-shot image classification capabilities of pre-trained VLMs. For the pre-trained VLMs we selected, we directly complete the image classification tasks on the target datasets without using any additional data or undergoing any extra training process.  **Therefore, we do not introduce additional data when the VLM performs image classification tasks, and there is no issue with data leakage.**\n\n    2. Secondly, the open-source datasets we use are all common and widely used. And since the pre-traning phase of VLMs have also used a lot of open-source data, it is likely that VLM's pre-training data has already included the data similar to the open-source datasets we use. Therefore, we believe that our use of common open-source datasets is reasonable.  It is important to emphasize that the previous SoTA method, ModelGPT[1], used the same open-source datasets as we did, ensuring that our comparison with their results is fair. Moreover, we believe that in the era of large models, a developmental trend is that we should make the most of the data and resources that are easily accessible to us.\n\n* Q2: Whether the performance of the proposed method is still remarkable if the categories in the open-source datasets are very different from the ones in the target zero-shot classification tasks.\n* A2: \n\n    The key insight of our method is to leverage the category relatedness to reuse the model's performance evaluation results from previous tasks. Therefore, for a completely new dataset that has no correlation with the open-source datasets, we might not expect it to achieve outstanding performance, just as different algorithms always have their suitable application scenarios. **However, we think this issue will not cause a drastic deterioration in our method's performance for the following reasons:**\n\n    * For most tasks, it is generally possible to find tasks that have a certain degree of correlation with them. This correlation can be at the object category level (e.g., tigers and leopards are both large felines) or at the attribute level (e.g., we found that in our dataset, the accuracy of VLM in recognizing the \"striped\" pattern in the DTD [2] dataset is positively correlated with its accuracy in recognizing \"tiger\" and \"zebra\" in Imagenet [3]. This is because the skin patterns of zebras and tigers are striped).\n\n    * For some new datasets that indeed have no correlation with open-source datasets, we can first measure their degree of correlation with the open-source datasets by calculating the text similarity between their class names and those of the open-source datasets. Then, we can set a threshold. Then for open-source tasks with a correlation score lower than the threshold, we do not use our SWAB method, and the method degrades to the original SoTA ModelGPT [1] method. For open-source tasks with a correlation score higher than the threshold, we use our SWAB method to leverage the open-source datasets to help us better select the model.\n\n    * As VLMs are evaluated on an increasing number of tasks, their performance evaluation results on previous tasks are expanded, further enhancing the capability of our method. This creates a virtuous cycle.\n\n[1] Zohar, et al. \"Lovm: Language-only vision model selection.\" NeurIPS 2023.\n\n[2] Cimpoi, Mircea, et al. \"Describing textures in the wild.\" CVPR 2014.\n\n[3] Deng, Jia, et al. \"Imagenet: A large-scale hierarchical image database.\" CVPR 2009.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our deepest gratitude to the reviewers for the meticulous examination of the paper and their insightful and valuable comments. We acknowledge that most reviewers observed the shining point, saying the proposed method is novel (qHTS, mpun), and interesting (qHTS, mpun), well-motivated (qHTS,NUg2,omtt,mpun), effective(qHTS,mpun). They also think our experiment results is comprehensive to indicate good validation of the proposed method(qHTS,mpun), and the question we studied is valuable and important (qHTS,omtt,mpun). 

Please check the answer to specific comments. Due to the rebuttal limitation, each response is limited to 6000 characters. **We have to control the response content due to this limitation, but we are more than happy to discuss this paper and any further questions with all reviewers in the next week.**

Meta-review: The paper tackles the problem of selecting a visual language model (VLM) from a ModelZoo for a target task using only the labels of the dataset. Two challenges are addressed in the language-only VLM selection (LOVM) problem: the modality gap and the capacity gap. The modality gap refers to the difference between the label (or text) domain and the full task, while the capacity gap refers to the generalization ability of the selector across datasets. Results for model selection are presented on a wide range of datasets and are compared with baselines and prior work, such as ModelGPT.

The reviews for the paper are mostly positive (Accept, Borderline Accept, Borderline Reject, and Borderline Accept).

Reviewer qHTS finds the problem to be novel and interesting with practical value. However, they note that the comparisons with ModelGPT are inconsistent. The rebuttal notes that this might be due to differences in random seeds and other experimental details not described in the paper or the code from the authors. Another difference is due to a larger model zoo. This was clarified in the rebuttal, where results using an identical model zoo from the ModelGPT paper were shown, and the method outperformed the baselines. However, the differences in the numbers seemed fairly large despite the authors' best efforts.

Reviewer NUg2 found the paper well-motivated and had questions regarding some experimental settings. However, their final score remained unchanged after the rebuttal, with a borderline accept rating.

Reviewer omtt was the most critical and questioned the applicability of text-only model selection. They also raised the question of how the model might handle tasks beyond classification. While the rebuttal addressed some concerns, the final rating was a borderline reject.

Reviewer mpun found the paper to be novel and noted that this is the first time LOVM has been applied. However, the Area Chair (AC) notes that LOVM has been proposed in prior work. The reviewer also found the approach well-motivated, and the experiments to be extensive. The main weaknesses were addressed in the rebuttal, and the score was raised to a borderline accept.

The AC read the paper, reviews, rebuttal, and discussions, and generally agrees with the positive sentiment about the paper. However, there are some issues the authors should address in the final version. The first is to make the comparison with ModelGPT easier. Perhaps the authors could report results using the identical model zoo and present a standalone evaluation with identical text. These experimental differences should be documented in the paper to facilitate future comparisons. Second, the AC found the related work section of the paper to be lacking. Some works, such as Task2Vec, are not cited, where the terms domain2vec and label2vec are introduced. Label2vec presents a way to represent tasks using only some statistics of the labels, which is related to this paper.

Task2Vec: Task Embedding for Meta-Learning, Achille et al., ICCV’15
Decision: Accept (poster)