Title: Bridge the Modality and Capability Gaps in Vision-Language Model Selection
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision-Language Model

Model Selection

Zero-Shot Image Classification
Abstract: Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zero-shot image classification tasks, a promising strategy is selecting appropriate Pre-Trained VLMs from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the “Modality Gap”—the disparity in VLM’s embeddings across two different modalities, making text a less reliable substitute for images; and the “Capability Gap”— the discrepancy between the VLM’s overall ranking and its ranking for target dataset, hindering direct prediction of a model’s dataset-specific performance from its general performance. We propose VLM Selection With gAp Bridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. It then uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps to obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset’s images. Experiments across various VLMs and image classification datasets validate SWAB’s effectiveness. Code is available at: https://github.com/YCaigogogo/SWAB.

Bridge the Modality and Capability Gaps in Vision-Language Model Selection

Abstract

Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. The expanding variety of Pre-Trained VLMs enhances the likelihood of identifying a suitable VLM for specific tasks. To better reuse the VLM resource and fully leverage its potential on different zeroshot image classification tasks, a promising strategy is selecting appropriate PreTrained VLMs from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset's images. In this paper, we analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection: the 'Modality Gap'-the disparity in VLM's embeddings across two different modalities, making text a less reliable substitute for images; and the 'Capability Gap'- the discrepancy between the VLM's overall ranking and its ranking for target dataset, hindering direct prediction of a model's dataset-specific performance from its general performance. We propose VLM S election W ith g A p B ridging (SWAB) to mitigate the negative impact of two gaps. SWAB first adopts optimal transport to capture the relevance between open-source and target datasets with a transportation matrix. It then uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset for bridging two gaps. By bridging two gaps to obtain better substitutes for test images, SWAB can accurately predict the performance ranking of different VLMs on the target task without the need for the dataset's images. Experiments across various VLMs and image classification datasets validate SWAB's effectiveness. Code is available at: https://github.com/YCaigogogo/SWAB .

1 Introduction

Vision-Language Models (VLMs) [46, 22, 48, 68] have demonstrated impressive image-text matching ability. One notable application of VLMs is zero-shot image classification [46, 40, 14, 37], where VLMs are leveraged to generate image classifiers using only class names directly. This zero-shot approach has shown considerable success in scenarios with scarce or no training images [35, 18].

Despite the success of VLM in image classification, the performance of a VLM may vary substantially according to the datasets and domains [11], making it challenging to use a single model to handle all tasks. Fortunately, many open-source VLMs are available [21], and these VLMs form a vast VLM Zoo. With different architectures, pre-training datasets, or training methods, these VLMs have different strengths. The diverse pre-trained VLMs increase the likelihood of pinpointing at least one VLM that excels in a given target dataset in most cases. 1 To more effectively reuse the VLM Zoo across diverse target tasks and unlock its full potential, we need a model selection method to choose suitable VLMs from the VLM Zoo for the target task. However, in scenarios such as zero-shot image

Language-Only

Target Dataset's Data

VLM 1

VLM 2

VLM 3

0.76

0.84

0.51

Abs / Rel Perf

Algorithm

Open-Source Datasets' Data

T

V

T

V

T

V

'Dog'

'A Dog is running'

'Lion'

'A Lion is sleeping'

Image Data

Text Data

Step 1: User Gives Task Description

Step 3: Use Model Selection Algorithm to Select Model

I want a VLM to help me classify {Cat, Dog}

classes in {Natural Image} domain.

Model Search Engine

Step 2: Use LLM to Generate Auxiliary Texts

'Generate 50 captions for {Cat, Dog} separately

for the {Natural Image} domain.'

Input

'A cat is catching a mouse'

'A dog is barking at a stranger'

Class-Related Auxiliary Texts

classification, many users might not have labeled images for their target tasks, especially those who are not Machine Learning researchers. They prefer to describe their needs in text and use a Model Search Engine to find the most suitable model. So one solution is identifying the most suitable VLMs in the zoo for a target dataset without access to the dataset's images . This VLM selection is termed as 'Language-Only VLM Selection' (LOVM) [73], and the paradigm is illustrated in Figure 1.

Two key types of information are available for LOVM. One is the target dataset's text data, i.e. , names of the target classes and class-related labeled texts generated by LLMs (Details described in Section B.1). The other is the open-source datasets, collected in the form of images with their corresponding class names. Based on these data, the goal is to estimate a VLM's zero-shot image classification capability ranking among the VLM zoo on the target dataset. LOVM encounters two challenges stemming from the inherent heterogeneity in models and datasets. The first challenge is the Modality Gap across different modal features extracted by a VLM. Since the visual and textual features extracted by VLMs tend to cluster into two distinct groups and have gap vectors between them [31], using text data as image proxies to rank VLMs is inaccurate. The second challenge is the Capability Gap between the VLM's overall ranking and its ranking in the specific dataset. Owing to the VLM's performance variation across different datasets, the VLM's average performance on open-source datasets is hard to reflect its performance on a specific target dataset. Thus, selecting a VLM based solely on its general strength may prove to be a less effective strategy.

In this paper, we propose VLM S election W ith g A p B ridging (SWAB) to address both gaps. The key idea is to reuse VLMs' statistics from open-source datasets to estimate their statistics on the target dataset, which mitigates the negative impact of these two gaps. In particular, SWAB first uses optimal transport to calculate the transport matrix based on textual similarity between class names of open-source and target datasets. After applying VLMs on open-source datasets to calculate VLMs' statistics, i.e. , the class-specific modality gap vectors and performance rankings of different VLMs, SWAB utilizes these statistics to estimate the same type of statistics on the target dataset. After that, SWAB uses the estimated gap vectors to align the features of texts with the features of images from the corresponding category, which bridges the modality gap. Meanwhile, SWAB's estimated VLMs' ranking also improves the prediction of their rankings on the target task, bridging the capability gap. The related work is in the appendix C. The main contributions are:

We analyze two key challenges in LOVM - the modality gap across VLM's modal features and the capability gap between the VLM's overall ranking and its ranking on the target dataset.

We propose SWAB, which utilizes optimal transport to transform useful statistics of VLMs on open-source datasets to the target dataset to bridge two gaps.

Experimental results on a LOVM benchmark composed of a wide range of VLMs and image classification datasets demonstrate the effectiveness of SWAB.

2 Preliminary

We formally introduce the LOVM setting, a baseline method for LOVM, and analyze the two kinds of gaps in LOVM. We use ∥ · ∥ to represent the Euclidean norm of a vector unless otherwise defined.

2.1 Selecting VLMs from a Model Zoo

Zero-Shot Image Classification of VLM. Assume there is a pre-trained VLM f = ( f I , f T ) consisting of an image encoder f I and a text encoder f T . Given an image classification dataset T with k T class names C T = { c T 1 , · · · , c T k T } , we input the class names C T (probably with templates like 'A photo of {class}') into the VLM's text encoder f T to get the image classifiers { ˆ t j } k T j =1 . Then, given a test image x i , we use the image encoder f I to extract its feature ˆ x i . Finally, we predict the label via the cosine similarity between the image feature ˆ x i and image classifiers { ˆ t j } k T j =1 . The class with the highest cosine similarity to the image is selected as the predicted class ˆ y i . Given ˆ x i = f I ( x i ) , ˆ t j = f T ( c T j ) , Equation 1 describes this zero-shot image classification process:

VLM Zoo. In recent years, a large number of (pre-trained) VLMs have emerged. Assume a collection of M VLMs constitute a VLM Zoo M = { f m = ( f I m , f T m )} M m =1 . The capability of f m is determined by three key factors: the model architecture ( e.g. , Transformer [57], ConvNeXt [34]), the pre-trained dataset ( e.g. , LAION-400M [47], MS-COCO [32]), and the training method ( e.g. , contrastive loss [46], caption loss [67]). Combinations of these factors result in 'good and diverse' VLMs in M . Given a dataset T , it is probable to find a suitable VLM from the VLM zoo with high zero-shot image classification performance on T .

Language-Only VLM Selection (LOVM). Rather than using images from the target dataset, LOVM focuses on the zero-shot scenario where only the target dataset's text data, such as its class names C T , are available for VLM selection. Besides, we can obtain some open-source image classification datasets S . The set of class names in S is C S = { c S 1 , · · · , c S k S } , and the D I S denote the labelled images in these classes. Given a target task T , the VLM selection method h estimates the zero-shot classification ability of f m based on C T , C S , and D I S via ˆ r T m = h ( f m | C T , C S , D I S ) , where m ∈ [1 , · · · , M ] . ˆ r T m is the predicted ranking of the m -th VLM f m on T . The higher the ranking, the more probable f m achieves higher zero-shot image classification performance on T . Assuming we can obtain the test image set D I T of the target dataset T with | D I T | images, then we can calculate the zero-shot image classification accuracy p T m of f m is calculated by p T m = 1 | D I T | ∑ ( x i ,y i ) ∈ D I T I ( y i = f m ( x i , C T )) . f m ( x i , C T ) represents the predicted class with the same manner as Equation 1. I ( · ) is the indicator function, which outputs 1 if the condition is satisfied, and 0 otherwise. Based on { p T m } M m =1 , we obtain the true ranking of M VLMs r T = [ r T 1 , . . . , r T M ] by assigning higher ranking r to models with higher accuracy p . However, in the zero-shot scenario, we can't obtain the test images set D I T in advance. Therefore, the goal of LOVM is to make the predicted ranking ˆ r T = [ˆ r T 1 , · · · , ˆ r T M ] be an accurate estimation of the ground truth ranking r T = [ r T 1 , . . . , r T M ] so that the best VLM can be selected.

Evaluation of LOVM Methods. We measure the performance of the LOVM algorithm by comparing the ranking similarity between r T and ˆ r T . Specifically, we calculate the Top-5 Recall R 5 (ranges from 0 to 1) and Kendall's Rank Correlation τ (ranges from -1 to 1). The larger the better.

2.2 Possible Paradigms for LOVM

Non-Learning-based LOVM . There are three main paradigms for LOVM. The first paradigm is to neglect the visual encoder and select VLM solely on texts. In detail, we can utilize ChatGPT [43] to generate auxiliary texts ˜ D T based on class names C T of T . More details are described in Section B.1. These class-specific texts act as 'image proxies'. Then, whether a VLM f m fits T could be measured by transferability metrics, e.g. , H-Score [2] and LogME [65], between the VLM's text encoder f T m and generated text dataset ˜ D T . The second paradigm relies on the general performance

of a certain VLM f m . We use open-source datasets to measure a VLM's general performance. If f m achieves high zero-shot classification performance over open-source datasets, then it is expected to be competitive on T . These methods assume that a VLM's ranking is relatively consistent across tasks.

Learning-based LOVM . The third paradigm is based on the learning process. In detail, the ability of a VLM could be predicted based on a ranker model f R . The input of f R is a vector s T m , depicting the dataset-specific representation of f m on T , while the output of f R is the relative/absolute performance ˆ p T m ∈ R of f m on T . The f R could be learned on open-source datasets S [70, 73]. Due to the availability of both class names C S and images D I S in the open-source dataset S such as ImageNet [8], we can calculate each VLM's representation { s n m } M,N m =1 ,n =1 and true zero-shot image classification accuracy { p n m } M,N m =1 ,n =1 . Here N refers to the number of datasets in S . After constructing the train set, the ranker model f R is learned based on the { s n m , p n m } M,N m =1 ,n =1 :

ℓ is a loss function that measures the discrepancy between the prediction and the ground truth, which can be Mean Squared Error Loss and Huber Loss, among others. Given T , the learned f R is able to predict the performance { ˆ p T m } M m =1 over { s T m } M m =1 via ˆ p T m = f R ( s T m ) . Finally, we can get the predicted VLMs' ranking ˆ r based on { ˆ p T m } M m =1 . This approach has similarities with metalearning [12, 3]. Meta-learning attempts to use data from multiple datasets to learn a model adaptable to the target task, while Learning-based LOVM employs data from multiple datasets to learn a ranker model for selecting the suitable model from a VLM Zoo for the target task. The representation s T m is one of the keys in this paradigm, and ModelGPT [73] calculates values s T m via the capability of a VLM's text encoder f T m .

ModelGPT uses generated text data ˜ D T as substitutes for images to calculate some metrics, which measures the zero-shot ability of f m on unseen images by the classification ability of f m on ˜ D T :

Here Metric i indicates the i -th metrics function such as Top-1 Accuracy and F1-Score. For example, the Top-1 Accuracy s T m, 1 could be calculated in a similar manner as Equation 1, with the only difference being that the test samples were replaced with text samples t i instead of image samples x i :

Besides, ModelGPT uses some metrics for assessing the features' quality extracted by the VLM's text encoder f T m . More details are in the Section B.2. Moreover, the zero-shot classification performance of f m on ImageNet is also included in s T m as a general ability measure of f m . ModelGPT implements f R as a simple linear model.

2.3 Analysis of the Two Gaps in LOVM

There are two main challenges that limit the application of the aforementioned paradigms in LOVM. The first is the modality gap across different modalities' features in VLM's feature space, and the second is the capability gap between VLM's overall performance and dataset-specific performance. Modality Gap. As described in Section 2.2, methods like H-Score, LogME, and ModelGPT utilize the ChatGPT generated auxiliary texts ˜ D T as image proxies to calculate metrics that measure the zeroshot accuracy on the target dataset T . In other words, the zero-shot classification ability across text and image modalities is estimated by the intra-modality classification ability. The latent assumption is that the generated texts and their corresponding images are closely aligned in VLM's feature space. However, this assumption is difficult to meet [31], and instances' features are more likely to cluster according to their modalities. In particular, we define the modality gap vector g between the features of an image-text pair ( x i , t i ) as g m,i := f I m ( x i ) -f T m ( t i ) . Values in the gap vector are generally not close to zero. We name this phenomenon as Modality Gap in LOVM, which makes the scores on ˜ D T hard to reveal the true zero-shot image classification capability of a VLM on a given dataset.

GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc0c)GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:d835)GLYPH(cmap:d835)

(a) Modality Gap

(b) Capability Gap

(

GLYPH(cmap:d835) = 43

)

VLM

1

VLM

2

.  .  .

VLM

M

Task Zoo

(

GLYPH(cmap:d835) = 23

)

ImageNet

.  .  .

Oxford Pets

Test

On

2

5

..

9

1

7

..

4

.  .  .  .

4

8

..

3

Model Rank

GLYPH(cmap:d835)1

∈ GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)2

∈ GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

M ∈ GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)=1

M

std(GLYPH(cmap:d835)

GLYPH(cmap:d835))

GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:d835)GLYPH(cmap:dfd0)

GLYPH(cmap:d835)=1

M

(max(GLYPH(cmap:d835)

GLYPH(cmap:d835)

) - min(GLYPH(cmap:d835)

GLYPH(cmap:d835)))

GLYPH(cmap:d835)

= GLYPH(cmap:d835)GLYPH(cmap:dfd5).

GLYPH(cmap:dfd5)GLYPH(cmap:dfd6)

T

V

T

V

T

V

VLM

Acc

Acc

Task 1

Task 2

VLM

1

2

3

3

2

1

Text Samples

Test Image Samples

Calculate

Test Acc

We conduct a validation experiment on ImageNet with 43 VLMs. We first generate 50 auxiliary texts per class as ˜ D T and then calculate the predicted Top-1 accuracy via Equation 4. Next, we use test images to calculate the VLM's true Top-1 accuracy. The consistency between the predicted Top-1 accuracy and true zero-shot image classification accuracy p m, T is measured by the Kendall Rank Correlation ( τ , higher is better) and Mean Absolute Error (MAE, lower is better). It can be observed from the left part of Figure 2 that the predicted accuracy derived from auxiliary texts ˜ D T does not closely match the true accuracy, indicating that these generated auxiliary texts in ˜ D T are not effective proxies for images.

To make the auxiliary texts act as better image proxies, one intuitive idea is to estimate the gap vector g for each image-text pair. Then we can add it to the feature f T m ( t i ) of the text t i to eliminate the modality gap, which may lead to more accurate scores s T m,i in Equation 3. However, the gap vector cannot be calculated directly without the target dataset's images. Furthermore, gap vectors for different classes are diverse, so using a shared vector across all datasets may not be a good choice.

Capability Gap. To select one VLM from the model zoo given a target dataset, one direct approach is to select the VLM that performs the best on average across multiple datasets. For example, we may first estimate the VLM's zero-shot classification ability on open-source datasets and then select the VLM with the highest performance. The key question is whether a VLM's average ranking on the open-source datasets can reveal its true ranking on the target dataset. Our empirical analyses indicate that there exists a discrepancy between the VLM's overall ranking and its ranking on a specific dataset. We name the discrepancy between the VLM's average ability and its specific ability as the Capability Gap , which results from the fact that a VLM's performance fluctuates significantly across various datasets.

To verify the claim, we test 43 VLMs on 23 target datasets provided by [73] and obtain the rankings of each VLM across these datasets. Based on these ranking results, we calculate the average standard deviation and the mean value of the difference between each VLM's maximum and minimum ranking. The experiment process is illustrated in the right part of Figure 2. We find that the mean difference between one VLM's maximum and the minimum ranking is 38.86. Since the total number of VLMs is 43, such a difference demonstrates that the top-performing VLM in one dataset could likely be among the worst in another.

One solution to bridge such a capability gap is to consider the VLMs' ranking on a related dataset. In other words, the ranking of VLMs on datasets from open-source datasets collections that are relevant to the target task may provide more useful insights than a general performance ranking across all tasks. The main challenge is to figure out which open-source datasets are similar to the target dataset and transform the VLM's ranking on these datasets to the target dataset.

Optimal

Transport

GLYPH(cmap:d835)

Text Encoder

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Step 1: Construct the Transport Matrix

Open-Source

Tasks' Classes

Target

Task's Classes

GLYPH(cmap:d835)

∗

Transport Matrix

Step 2: Bridge the Modality Gap

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

=

GLYPH(cmap:d835)

GLYPH(cmap:d835)

⋮

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

(GLYPH(cmap:d835)

∗

)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),2

GLYPH(cmap:d835)

Rank

Model

`

Text Samples

Open-Source

Task's Gap Vectors

Target Task's

Gap Vectors

Modified Text Samples

Add

Gap

Step 3: Bridge the Capability Gap

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

=

GLYPH(cmap:d835)

GLYPH(cmap:d835)

⋮

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

=

GLYPH(cmap:d835)

GLYPH(cmap:d835)

⋮

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),(2)

Open-Source Tasks'

Class-Specific Rank

Target Task's

Class-Specific Rank

GLYPH(cmap:d835)=1

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Step 4: Ensemble Predicted Rankings

Weighted

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),GLYPH(cmap:d835)GLYPH(cmap:d835)GLYPH(cmap:d835)

VLM's Final

Rank Prediction

VLM's text embedding cluster

VLM's image embedding cluster

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),2

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),(1)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835),(1)

GLYPH(cmap:d835),(2)

(GLYPH(cmap:d835)

∗

)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

...

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Summary. We emphasize two kinds of gaps in LOVM, i.e. , the modality gap across features of different modalities generated by a VLM, and the capability gap between a VLM's overall ranking and its ranking given a specific target dataset. Both two gaps pose obstacles to previous model selection methods, such as LogME and ModelGPT, and degrade their abilities in VLM selection. Moreover, those intuitive approaches to bridge the gaps still face challenges.

3 VLMSelection with Gap Bridging

To mitigate the impact of both gaps on LOVM and integrate non-learning-based and learning-based LOVM methods, we propose VLM S election W ith g A p B ridging (SWAB). The key idea of SWAB is to bridge modality and capability gaps by utilizing class-level statistics of VLMs from open-source datasets. By measuring the textual similarity between the target dataset's class names and those in open-source datasets, we construct a bridge matrix. Based on it, we estimate the gap vectors between image and text modalities, which rectifies the text-derived scores in ModelGPT. In addition, we predict the VLM's performance ranking for the target dataset based on the bridge matrix and VLM's ranking on the open-source dataset. Both estimated statistics will be used to obtain a more accurate language-only VLM selection. The workflow of SWAB is illustrated in Figure 3.

3.1 Construct the Bridge Matrix Using Optimal Transport

Benefiting from the open-source datasets, some useful class-level statistics, such as modality gap vectors and zero-shot classification accuracy of a certain VLM, could be calculated, which can help the performance ranking estimation of a VLM on the target dataset. To better utilize these class-level statistics for predicting the corresponding statistics of a VLM on the target task, we introduce semantic relevance information between open-source datasets' classes and target dataset's classes into the statistics reusing process, which is automatically generated through Optimal Transport [7, 45].

Recall that the sets of class names of the open-source datasets and the target dataset are C S = { c S i } k S i =1 and C T = { c T i } k T i =1 , respectively. The semantic relevance between two classes could be measured by the textual similarity between their class names. In detail, we use a pre-trained text encoder ϕ ( e.g. , MPNet [49]), which extracts text features for these class names, i.e. , { ϕ ( c S 1 ) , · · · , ϕ ( c S k S ) }

and { ϕ ( c T 1 ) , · · · , ϕ ( c T k T ) } . Then, we can calculate the cosine similarity ϕ ( c S i ) ⊤ ϕ ( c T j ) ∥ ϕ ( c S i ) ∥·∥ ϕ ( c T j ) ∥ between the text feature of the i -th class in the open-source datasets ϕ ( c S i ) and that of the j -th class in the target dataset ϕ ( c T j ) . The larger the cosine similarity, the more similar the two classes are. After that, we construct the cost matrix in optimal transport via cost ij = 1 -ϕ ( c S i ) ⊤ ϕ ( c T j ) ∥ ϕ ( c S i ) ∥·∥ ϕ ( c T j ) ∥ . In practice, we exponentiate each element in cost ∈ R k S × k T ≥ 0 using the base e to amplify its differences. We then solve the optimal transport problem with the constructed cost matrix to get the transport matrix γ ∗ . Since optimal transport aims to obtain a transport matrix that minimizes the transmission cost, the transport matrix γ ∗ will reuse more information between semantically similar classes:

The cost matrix quantifies the expense of moving elements between all class pairs, and γ ∗ ∈ R k S × k T is the transport matrix. OT minimizes the cost indicated by the matrix cost and moves elements from one distribution u to another v . In SWAB, we define u and v as uniformly distributed vector u = 1 /k S ∈ R k S and v = 1 /k T ∈ R k T . This indicates that we treat all classes as equally important. We may also incorporate prior knowledge of class importance to define u and v .

The solution γ ∗ of the OT problem in Equation 5 could be solved efficiently [13], and γ ∗ acts as a bridge matrix between open-source datasets' classes and target dataset's classes. Usually, the smaller cost i,j is, the larger the corresponding element γ ∗ i,j obtained by OT, indicating statistics of the i -th class of open-source datasets may help more when we estimate the statistics of the j -th target class.

3.2 Bridge the Modality Gap and Capability Gap

Bridge the Modality Gap. Given the m -th VLM f m in the model zoo, we want to estimate the modality gap g T m,j between the extracted image and text features for the j -th class in the target dataset T to bridge the modality gap. However, in the zero-shot scenario, we can't get the target dataset's images in advance, so we can't directly calculate the gap vectors using image-text pairs. To solve this problem, SWAB estimates the target dataset's gap vectors based on the open-source datasets' gap vectors with γ ∗ . Given the k -th open-source class c S k , we can get the set of images D I S k = { ( x i , y i ) | ( x i , y i ) ∈ D I S , y i = c S k } from the open-source datasets. | D I S k | is the number of images in D I S k . Then, the modality gap vector g S m,k for class c S k and model f m can be calculated

the normalized class text prototype embedding and all normalized image embeddings from the class c S k . In a similar manner, the gap vectors of all open-source classes { g S m, 1 , · · · , g S m,k S } can be obtained given f m . We use a matrix G S m ∈ R k S × d m to represent those k S gap vectors for the m -th VLM in the VLM Zoo, and d m is the dimensionality of features extracted by f m .

The gap vectors { g T m, 1 , · · · , g T m,k T } for the target dataset could be estimated based on the G S m and the transport matrix γ ∗ . If two classes are semantically similar, then we can reuse the gap vector from the similar class. We set the predicted gap vector for the j -th target class ˆ g T m,j as a weighted sum of G S m , and the weight comes from γ ∗ , which is ˆ g T m,j = | C T | ( γ ∗ : ,j ) ⊤ G S m . γ ∗ : ,j is the j -th column of γ ∗ . We use scaling factors | C T | to ensure that for each target class, the sum of γ ∗ : ,j equals 1. This scale operation has also been used in previous work [62, 63]. After that, we modify the step of ModelGPT in Equation 3, where the metrics over the generated auxiliary texts ˜ D T are calculated. We add the gap vector ˆ g T m,j to the embeddings of the auxiliary texts ˜ D j T from the j -th class in the target dataset:

The modified text embedding ˜ t m,i serves as better image proxies. In other words, classification metrics on f T m ( t i ) only reveal the discerning ability of the text encoder of f m , which is far from the (cross-modal) zero-shot classification ability due to the modality gap. By bridging such a gap with modified text embedding, classification metrics on ˜ t m,i are closer to the classification metrics on images with textual classifier . Therefore, we use { ˜ t m, 1 , · · · } in Equation 6 as better

inputs to calculate s T m in Equation 3. The updated score vectors s T m are then input to the ranker model f R , which is able to get more accurate VLM's performance prediction ˆ p T m . Based on the performance prediction { ˆ p T m } M m =1 , we can obtain the VLMs' ranking ˆ r T , (1) = [ˆ r T , (1) 1 , · · · , ˆ r T , (1) M ] = Ranking ([ ˆ p T 1 , · · · , p T M ]) . Ranking ( · ) transforms the accuracies into models' ranking.

Bridge the Capability Gap. Whether the m -th VLM f m fits the target task T could also be estimated by the performance of f m on the open-source datasets related to T . We first calculate the VLM's class-level performance ranking over the whole open-source datasets. Given the k -th open-source class c S k and the corresponding set of images D I S k , we calculate the zero-shot classification accuracy p S m,k via Equation 1. We then determine each VLM's ranking on the k -th open-source class c S k using Equation 7. We calculate VLMs' ranking for other open-source classes in the same way.

Next, we estimate the ranking of a certain VLM f m on the target dataset T . By re-organizing the ranking values in Equation 7, the performance ranking vector of f m on all k S open-source classes is r S m = [ r S m, 1 , · · · , r S m,k S ] ∈ Z k S + . If f m ranks high on certain open-source classes related to the classes in the target dataset, the performance ranking of f m on the target dataset may also be high. Thus, we perform a weighted sum of ranking values in r S m and assign larger weights to those classes related to the target dataset. This can be done by using γ ∗ :

Elements in ˆ r T m ∈ R k T are the predicted ranking of the m-th VLM f m for k T target classes. Since we only need the relative order of ranks, there is no additional scale factor in Equation 8. After that, we average class-specific ranking values in ˆ r T m , and use ˆ r T , (2) m = Mean ( ˆ r T m ) to denote the estimated ranking of f m on the target dataset T . In summary, we predict the ranking of models on each class in the target dataset based on their rankings in classes of the open-source datasets, guided by the semantic relevance between the classes. We predict the ranking of models on each category in the target dataset based on their rankings in categories of the open-source dataset, guided by the semantic relevance between the categories. Benefiting from the models' consistently aligned classification performance ranking across similar classes, this approach bridges the capability gap in VLM selection.

3.3 Summary of SWAB

As is described in Section 3.2, given a target dataset T and a VLM f m , we denote the two performance ranking predictions obtained by bridging the modality gap and capability gap as ˆ r T , (1) m and ˆ r T , (2) m , respectively. ˆ r T , (1) m is predicted based on ModelGPT with our modified embeddings of the generated auxiliary texts for T . ˆ r T , (2) m is predicted by the weighted sum of VLM's class-specific ranking values on the open-source datasets. These two predictions, respectively, originate from learning-based and non-learning-based LOVM methods. We ensemble two predictions together and achieve a more accurate model ranking estimation. We utilize the weighted Borda Count to aggregate two rankings:

We set α = 0 . 5 . Ultimately, SWAB determines the final predicted ranking of the VLMs in the VLM Zoo based on ˆ r T , ens m . The pseudo-code of SWAB is listed in Algorithm 1.

4 Experiments

4.1 Evaluation on LOVM Benchmark

Setups. We follow LOVM [73] to use its provided VLM Zoo. In addition, to further enhance the diversity of the VLM Zoo, we add some representative VLMs such as BLIP [28] and BEiT3 [59] to expand the VLM Zoo. The final VLM Zoo contains a total of 43 models, which differ in aspects such as model architecture, pre-training datasets, and training methods. In the appendix, we also provide experimental results of different model selection methods on the 35 models originally offered by LOVM. We evaluate different methods on 23 datasets, i.e. ImageNet [8], Aircraft [36],

CIFAR100 [26] and so on. We obtain VLM's ground truth ranking based on VLM's Top-1 Accuracy calculated on the target dataset's test image set.

Baseline. We select representative methods for each of the three paradigms mentioned in Section 2.2 as our baselines. For the first paradigm, we use four classic model selection methods: H-Score [2], NCE [55], LEEP [56] and LogME [65]. For the second paradigm, we use the VLM's ranking on ImageNet (INB) and VLM's average ranking (Avg Rank) on classes of the open-source datasets in the LOVM Benchmark. For the third paradigm, we compare our method with ModelGPT [73].

Evaluations. We use Top-5 Recall and Kendall's Rank Correlation to measure the similarity between the predicted and the ground truth model rankings to evaluate the LOVM method's performance. We also calculate the sum of these two metrics to consider the method's comprehensive capability.

Implementation Details. For a fair comparison, SWAB follow ModelGPT [73] to sequentially extract a target dataset from each of the 23 datasets in the LOVM Benchmark and treat the remaining datasets as open-source datasets. Besides, SWAB adopts ModelGPT's approach of adding Gaussian noise to corrupt the target dataset's generated text embeddings. Since LOVM does not provide the specific image data for the 23 datasets, we download these datasets ourselves and adopt their standard data splits. Using the templates provided by LOVM, we construct classifiers and recalculate each VLM's zero-shot image classification accuracy on these datasets. Additionally, we utilize the code provided by LOVM to generate class-related text data using ChatGPT. For H-Score, NCE, LEEP, LogME, INB, and Avg Rank, we follow the practices of previous work and do not add noise to the model's inputs. To ensure reliable results, we conduct ten repeated experiments using random seeds from 1 to 10 and report the mean value and standard deviation of ModelGPT's performance and SWAB's performance in Table 1.

Results Analysis. From Table 1, we can draw the following conclusions: (1) Metric-based nonlearning model selection methods such as LogME show poor performance on the LOVM Benchmark. This is primarily because such algorithms rely on the target dataset's images, thus the modality gap has a greater negative impact on them when using generated text data as a substitute for images. (2) Using open-source datasets is helpful for LOVM. We find that using open-source datasets in a non-learning way ( e.g. INB, Avg Rank) or a learning way ( e.g. ModelGPT) all helps LOVM, since their performance significantly surpasses that of methods not utilizing open-source datasets ( e.g. LogME). (3) Despite leveraging more open-source datasets, the performance of Average Rank is worse than INB. This confirms our analysis of the Capability Gap, which suggests a discrepancy between the average ranking of a VLM and its ranking on a specific dataset. (4) Our SWAB achieves the best performance across all evaluation metrics . Notably, our final performance of R 5 + τ (0.822) represents a significant improvement of 14.8% over the original SoTA method ModelGPT (0.716).

4.2 Ablation Study

We conduct an ablation study to demonstrate that bridging the Modality Gap and Capability Gap are both essential for SWAB. Table 2 presents our experiment results, from which we can observe that SWAB achieves the best performance across all metrics when both gaps are bridged simultaneously. The ablation study confirms our analysis.

4.3 Influence of Key Components in SWAB

Will Bridge the Capability Gap Be Beneficial for VLM Selection? We compare the LOVM performance directly using the VLM's average ranking on each class of open-source datasets and weighted-sum ranking based on transport matrix γ ∗ . The results are shown in Table 3. We can find that utilizing class relevance to bridge the Capability Gap is beneficial for VLM's Model Selection.

Will Bridge the Modality Gap Be Beneficial for VLM Selection? To eliminate the interference of other factors, we solely utilize the learning-based predicted rankings ˆ r T , (1) m in SWAB, and the input to the ranker model f m only consists of metrics calculated on the generated text data ˜ D T , which serves as substitutes for images. In this way, the method's performance depends solely on the quality of the generated text data ˜ D T . From the Table 4, we can find that the generated text data ˜ D T become better substitutes for image data after bridging the Modality Gap.

T , (2)

Which Kind of Gap Vectors Should We Use? When utilizing the gap vectors from open-source datasets, we have two options: (1) Use the dataset-level mean gap vector calculated on the whole dataset's image-text pairs. (2) Use the class-level mean gap vector calculated on the corresponding class's image-text pairs. We hope that the gap vectors are as close to each other as possible so that their mean vector can substitute well for the whole set. Based on this idea, we calculate the statistics of the gap vectors within a dataset and within each class. We calculate three metrics which include: (1) the standard deviation of these gap vectors' magnitude; (2) the mean cosine similarity between these gap vectors and their corresponding mean gap vectors; and (3) the standard deviation of these cosine similarities. These metrics reflect the consistency of the gap vectors. Table 5 shows the results.

From the Table 5, we can find that the class-level gap vectors tend to be more consistent, which inspires us to use the class-level mean gap vectors. We also compare the results of SWAB-M on the LOVM Benchmark using the dataset-level mean gap vectors and the class-level mean gap vectors, respectively. The implementation details are the same as Table 2. Table 6 shows the results, which verifies that using the class-level mean gap vectors is a better choice.

5 Conclusion

We analyze and address two key challenges in Language-Only VLM Selection (LOVM), which are VLM's modality gap across different modal features and VLM's Capability gap between its overall and dataset-specific rankings. Our key insight is that we can reuse the model's useful statistics on open-source datasets to help the model selection on the target dataset. SWAB utilizes a transport matrix between classes of the target dataset and open-source datasets to transfer VLM's class-specific modality gap vectors and class-specific rank from open-source datasets to the target dataset, which mitigates the negative impacts of these two gaps. Experiment results on the LOVM benchmark show the superiority of our method.

References

Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In ICCV , 2019.

Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas J. Guibas. An information-theoretic approach to transferability in task transfer learning. In ICIP , 2019.

Wei-Lun Chao, Han-Jia Ye, De-Chuan Zhan, Mark E. Campbell, and Kilian Q. Weinberger. Revisiting meta-learning as supervised learning. CoRR , 2020. URL https://arxiv.org/abs/2002.00573 .

Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE , 2017.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR , 2014.

Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS , 2011.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NeurIPS , 2013.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks. In ECCV , 2022.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge 2007 (voc2007) results. http://www.pascalnetwork.org/challenges/VOC/voc2007/workshop/index.html, 2007.

Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In ICML , 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , 2017.

Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. JMLR , 2021.

Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming-Hsuan Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi-modal models. In CVPR , 2023.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR , 2013.

Dumitru Ian Goodfellow, Will Cukierski, and Yoshua Bengio. Challenges in representation learning: Facial expression recognition challenge, 2013.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Xiaoxuan He, Siming Fu, Xinpeng Ding, Yuchen Cao, and Hualiang Wang. Uniformly distributed category prototype-guided vision-language framework for long-tail recognition. In ACM MM , 2023.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. J-STARS , 2019.

Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In ICML , 2022.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021. URL https://doi.org/10.5281/zenodo.5143773 .

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Fei-Fei Li, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR , 2017.

Kaggle and EyePacs. Kaggle diabetic retinopathy detection, 2015. URL https://www.kaggle.com/c/ diabetic-retinopathy-detection/data .

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) , 2013.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs , 2010.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

Li Li, Jiawei Peng, Huiyi Chen, Chongyang Gao, and Xu Yang. How to configure good in-context sequence for visual question answering. In CVPR , 2024.

Weihua Li, Wenyang Liu, Yanbu Guo, Bingyi Wang, and Hua Qing. Deep contextual representation learning for identifying essential proteins via integrating multisource protein features. Chinese Journal of Electronics , 2023.

Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurIPS , 2022.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Jian-Dong Liu, Zhi-Hao Tan, and Zhi-Hua Zhou. Towards making learnware specification and market evolvable. In AAAI , 2024.

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR , 2022.

Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A simple long-tailed recognition baseline via vision-language model. CoRR , abs/2111.14745, 2021.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR , abs/1306.5151, 2013.

Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. In CVPR , 2023.

Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: uniform manifold approximation and projection. J. Open Source Softw. , 2018.

Deyu Meng and Lina Sun. Some new trends of deep learning research. Chinese Journal of Electronics , 2019.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR , 2023.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop , 2011.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP , 2008.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. NeurIPS , 2022.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR , 2012.

Gabriel Peyré and Marco Cuturi. Computational optimal transport. Foundations and Trends in Machine Learning , 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pair. CoRR , abs/2111.02114, 2021.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR , 2022.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurIPS , pages 16857-16867, 2020.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In IJCNN , 2011.

Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Handling learnwares developed from heterogeneous feature spaces without auxiliary data. In IJCAI , 2023.

Peng Tan, Hai-Tian Liu, Zhi-Hao Tan, and Zhi-Hua Zhou. Handling learnwares from heterogeneous feature spaces with explicit label exploitation. NeurIPS , 2024.

Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Towards enabling learnware to handle heterogeneous feature spaces. Mach. Learn. , 2024.

Zhi-Hao Tan, Jian-Dong Liu, Xiao-Dong Bi, Peng Tan, Qin-Cheng Zheng, Hai-Tian Liu, Yi Xie, XiaoChuan Zou, Yang Yu, and Zhi-Hua Zhou. Beimingwu: A learnware dock system. In KDD , 2024.

Anh Tuan Tran, Cuong V. Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In ICCV , 2019.

Anh Tuan Tran, Cuong V. Nguyen, and Tal Hassner. Leep: A new measure to evaluate transferability of learned representations. In ICML , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant CNNs for digital pathology, 2018.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In CVPR , 2023.

Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR , 2010.

Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. In NeurIPS , 2023.

Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Rectify heterogeneous models with semantic mapping. In ICML , 2018.

Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zhi-Hua Zhou. Heterogeneous few-shot model rectification with semantic mapping. TPAMI , 2021.

Chao Yi, Lu Ren, De-Chuan Zhan, and Han-Jia Ye. Leveraging cross-modal neighbor representation for improved clip classification. In CVPR , 2024.

Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In ICML , 2021.

Kaichao You, Yong Liu, Ziyang Zhang, Jianmin Wang, Michael I. Jordan, and Mingsheng Long. Ranking and tuning pre-trained models: a new paradigm for exploiting model hubs. JMLR , 2022.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR , 2022.

Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. CoRR , abs/2111.11432, 2021.

Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020.

Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. NeurIPS , 2023.

Zhi-Hua Zhou. Learnware: on the future of machine learning. Frontiers Comput. Sci. , 2016.

Zhi-Hua Zhou and Zhi-Hao Tan. Learnware: small models do big. Sci. China Inf. Sci. , 2024.

Orr Zohar, Shih-Cheng Huang, Kuan-Chieh Wang, and Serena Yeung. Lovm: Language-only vision model selection. In NeurIPS , 2023.

Beiji Zou, Xi Shan, Chengzhang Zhu, Yulan Dai, Kejuan Yue, Yuanqiong Chen, Yalong Xiao, and Jiaer Huang. Deep learning and its application in diabetic retinopathy screening. Chinese Journal of Electronics , 2020.

In the Appendix, we introduce more details about the LOVM Benchmark as well as our extensions to it. Besides, we introduce ModelGPT's implementation and our SWAB's implementation. We also provide more experimental results of SWAB. The structure of the Appendix is as follows:

In section A, we introduce the relevant information of the 43 models and 23 datasets used in our experiments. We also introduce the evaluation metrics used in LOVM Benchmark [73].

In section B, we introduce the metrics used in Equation 3.

In section C, we introduce the related work of the paper.

In section D, we provide some details on SWAB's implementation.

In section E, we provide more experimental results of SWAB.

A LOVMBenchmark Details

LOVMBenchmark [73] consists of 35 pre-trained VLMs and 23 datasets, with a total of 35 × 23 = 805 evaluations. To further enhance the diversity of the VLM Zoo, we add some representative VLMs such as BLIP [28] and BEiT-3 [59] to expand the VLM Zoo. The final VLM Zoo contains a total of 43 models, with a total of 43 × 23 = 989 evaluations. For each evaluation, LOVM provides the VLM's zero-shot image classification accuracy on the corresponding dataset. Therefore, we can get the ground truth performance ranking of 43 VLMs on the 23 datasets.

A.1 VLMs of LOVM Benchmark

To cover as many types of models as possible, the LOVM Benchmark uses OpenCLIP library [21] to get diverse VLMs. These VLMs differ from each other in terms of the model architecture (ResNet [17], Transformer [57], ConvNext [34]), the pre-trained dataset (OpenAI's Data [46], LAION 2b [47]), the training method (loss function/hyperparameter/data augmentation) and so on. Table 7 displays the relevant information of each VLM. We further add BLIP and BEiT-3 to the original VLM Zoo. It should be noted that in addition to the image-text pair data listed in the Table 7, BEiT-3's pre-training data also includes unimodal image dataset (ImageNet-21K) and text datasets (English Wikipedia, BookCorpus, OpenWebText, CC-News, Stories). The diversity of these VLMs ensures that the experimental results calculated on them can reflect the performance of the VLM model selection algorithm in real-world situations.

A.2 Datasets of LOVM Benchmark

To cover as wide a distribution of image classification tasks as possible, the LOVM Benchmark collects 23 diverse datasets. These datasets differ from each other in terms of the number of categories, category semantics, image domains, and so on. Table 8 displays the relevant information of each dataset. The diversity of these tasks ensures that the experimental results calculated on them can reflect the performance of the VLM model selection method in real-world situations.

A.3 Evaluation Metrics of LOVM Benchmark

In LOVM, our aim is to maximize the rank similarity between the prediction of VLMs' ranking ˆ r T = { ˆ r T m } M m =1 and VLMs' ground truth ranking r T = { r T m } M m =1 on the target dataset, especially the rank similarity of the top 5 VLMs in ˆ r T and r T . This is because we tend to focus only on whether those appropriate models can be chosen. To ensure fair comparability, we follow the evaluation metrics used by LOVM [73] to assess different model selection methods and directly utilize the code provided by LOVM [73] to calculate these metrics:

Top-5 Recall ( R 5 ) - Top-5 Recall R 5 measures the model selection algorithm's accuracy in identifying the true top five best-performing models within its predicted top five models. The calculation method is shown in Equation 11. Here IND (ˆ r 5 T ) and IND ( r 5 T ) indicates the model indices sets of the top 5 VLMs in ˆ r T and r T , respectively. A Top 5 Recall closer to 1 signifies greater accuracy in the predicted rankings.

Kendall's Rank Correlation ( τ ) - Kendall's Rank Correlation τ measures the ranking consistency between two ranking lists. We follow LOVM [73] to focus on the VLMs within the intersection of the top 5 VLMs in ˆ r T and r T and use τ to evaluate a model selection method's capability.

B ModelGPT Details

ModelGPT is a method proposed by LOVM [73]. In this section, we introduce the metrics that ModelGPT used in Equation 3.

B.1 The Generation Process of Auxiliary Text Samples

ModelGPT [73] utilizes ChatGPT [43] to generate auxiliary text data by designing prompts to query ChatGPT. This extra text data mainly includes the Captions Dataset and the Synonyms Dataset.

Captions Dataset. ModelGPT uses the following prompt to guide LLM to generate realistic and confusing text data corresponding to the user-provided classes. The reason for requiring ChatGPT to generate confusing texts is to increase the classification difficulty of the text data, thereby enhancing its ability to distinguish the performance of different models.

Generate long and confusing image captions for the {domain} domain, which will be used to evaluate a Vision-Language Model's {task} performance. Generate 50 captions for {classname}:

We show some generated auxiliary text examples. For example, in the category of dog, one of the text samples generated by ChatGPT is "An adorable dog perfect for cuddles and playtime." ModelGPT collects the results from this prompt to form the captions dataset, D cap .

Synonyms Dataset. ModelGPT uses synonyms to evaluate VLM's text encoder. For example, we expect an excellent VLM to extract similar embeddings for the words 'chair' and 'seat'. The prompt to guide LLM to generate synonyms is as follows.

Please list the superclasses/synonyms for {classname}. For example: chair: [furniture, seat, bench, armchair, sofa] {classname}:

ModelGPT collects the results from this prompt to form the synonyms dataset, D syn .

B.2 Text-Derived Scores

ModelGPT uses six metrics for model selection, which can be divided into Text Classification scores and Dataset Granularity scores . Text Classification scores include the Text top-1 accuracy score and Text f1-score . While Granularity scores include the Fisher criterion , Silhouette score , Class Dispersion score and Synonym Consistency score . Here we focus on introducing the various metrics included in the Granularity scores . We refer to the relevant content in LOVM.

Fisher Criterion ϕ fisher. The Fisher score measures the closeness of VLM's text classifier to one another. Equation 12 shows the calculation process of it where ˆ t i is the text classifier of the i -th class derived using the prompt ensemble strategies proposed in [46], θ ( · , · ) is a function that calculates the cosine similarity between two vectors, and | C | is the number of classes.

̸

Silhouette Score φ sil. The Silhouette Score measures the separation of different-class samples in the caption dataset D cap . To calculate it, ModelGPT averages the cosine similarity of captions to the nearest other class's classifier by:

̸

where ˆ t i is the text classifier of the i -th class derived using the prompt ensemble strategies proposed in [46], θ ( · , · ) is a function that calculates the cosine similarity between two vectors, and | C | is the number of classes. D cap [ j ] k representing sample k of class j in the caption dataset D cap . There is a total of N such samples for each class.

Class Dispersion Score ρ disp. Class Dispersion Score quantifies the degree of same-class tightness or data cone radius, which is calculated using the following Equation:

The definitions of all symbols in Equation 14 are consistent with those in Equation 13.

Synonym Consistency Score γ syn. Synonym consistency allows us to evaluate the degree of content shift between the VLMs' pre-training and target dataset. The calculation process is shown as follows:

The definitions of ˆ t i , θ ( · , · ) , | C | and N in Equation 15 are consistent with those in Equation 13. D syn [ i ] k representing sample k of class i in the synonym dataset D syn .

C Related Work

Vision-Language Models. Deep learning has achieved competitive performance [39, 74, 30] and the vision-language model is a new research hotspot of it. Vision-Language Models represent a class of multimodal models adept at correlating textual and visual information. These VLMs can comprehend rich semantics and possess strong generalization capabilities. Hence, they are often used in data-limited scenarios [61, 29]. Many VLMs are pre-trained or fine-tuned on extensive text-image pairs using loss functions such as contrastive loss, endowing them with powerful text-image matching capability. Prominent VLMs include CLIP [46], ALIGN [22], FLAVA [48], Florence [68], and CoCa [67]. These VLMs possess robust zero-shot image classification capabilities [46], which enables its widespread application in tasks characterized by long-tail distributions or those where collecting substantial training data is challenging, such as medical image analysis. Some work [40, 64] has also shown that by incorporating external knowledge, the zero-shot capabilities of CLIP can be further enhanced. In recent years, the number of open-source VLMs has been increasing [21]. Previous work [73] has pointed out that different VLMs possess varying image classification capabilities. This indicates that the performance of VLMs can vary significantly across different tasks and domains. These models with diverse capabilities constitute a VLM Zoo rich in knowledge. This VLM Zoo enables us to utilize different VLMs for various classification tasks, thereby changing the paradigm of using a single VLM to complete diverse classification tasks. This paper focuses on selecting the most suitable VLM for the target task from the VLM Zoo.

Model Selection. In recent years, how to select the most suitable model for the target task from the model zoo has received widespread attention. For example, a series of works on learnware [53, 52, 71, 72, 33, 51, 54] have attempted to solve this problem. The essence of the model selection problem lies in measuring the transferability of the model to the target task. Previous model selection methods [55, 56, 65, 66, 9, 20] evaluate the PTM's transferability by performing a forward pass of the PTM on the target task's data and calculating the metric of transferability based on the forward pass's result. For example, H-Score [2], NCE [55], LEEP [56], LogME [65] estimate transferred log-likelihood, negative conditional entropy, log expectation, marginalized likelihood to obtain proxy metric of transferability, respectively. Some new methods, such as Task2Vec [1] and Model Spider [70], generate representation vectors for both the models and the tasks and measure the transferability of the model to the task by calculating the similarity between these vectors. However, VLMs are typically used in zero-shot or few-shot scenarios, where we are unable to obtain a large amount of data for the target task in advance, making traditional model selection approaches unsuitable for VLMs. Additionally, previous methods have mainly focused on single-modal models, overlooking the characteristics of VLMs. Therefore, in this paper, we concentrate on designing model selection algorithms that are suitable for scenarios with limited data and take into account the characteristics of VLMs.

D Implementation Details of SWAB

In this section, we provide some details on the implementation of SWAB, which are not mentioned in the main text due to space constraints.

D.1 Filtering the Open-Source Tasks' Classes

The statistics of classes unrelated to the target class are generally not valuable for reuse. Meanwhile, when the number of classes | C S | in open-source datasets S is large, solving the optimal transport problem in Equation 5 can be time-consuming (as current optimal transport toolkits generally compute via CPU). To reduce the runtime of optimal transport, we can first filter the classes C S . Consider that only statistics of classes relevant to the target dataset are helpful. Therefore, we can filter out the classes in C S that are irrelevant to the target dataset T based on the class-level textual semantic similarity between the open-source datasets' classes and the target dataset's classes. This process is shown in the following Equation:

Here S i, : refers to the i -th row of the semantic similarity matrix calculated using Equation 16, which represents the vector formed by the similarity between the i -th class c S i in C S and each class C T = { c T 1 , · · · , c T k T } of the target task. λ is a threshold and we set λ = 0 . 5 . k ′ S refers to the number of classes in the filtered set C ′ S . Then we use the filter classes C ′ S to calculate the transport matrix γ ∗ ∈ R k ′ S × k T and continue with the following steps.

D.2 Using Partial Optimal Transport for Bridging the Capability Gap

Partial optimal transport extends the optimal transport framework, enabling the selective transfer of elements from a source to a target distribution, rather than moving all elements. Its optimization problem is defined as in Equation 18. Here mass refers to the total amount of mass actually be transferred. We set mass = 0 . 9 in our implementation.

We found that when using Equation 8 to bridge the Capability Gap, the transport matrix γ ∗ obtained using partial optimal transport yields better results than the one obtained using the original optimal transport via solving the Equation 5. Therefore, in our implementation, we use the transport matrix derived from partial optimal transport to bridge the capability gap. This also indicates that when estimating VLM's statistics on the target dataset, different types of statistics have different preferences for the estimation methods used. This variability is worth further investigation.

D.3 Data Normalization in Bridging the Modality Gap.

When bridging the Modality Gap as described in Section 3.2, we find that applying z-score normalization to the text and image features used in this process yields better results. Therefore, in our implementation, we normalize the features of all text and image samples during the modality bridging process using the following Equation:

Here x ∈ R d represents the image sample's or text sample's feature, while µ ∈ R d and σ ∈ R d are calculated using the features of all samples of the same modality within its respective dataset.

Algorithm 1 SWAB

1: Input: Target dataset's class names C T , open-source datasets' class names C S , open-source datasets' images D I S .

2: Use ChatGPT to generate auxiliary text data ˜ D S and ˜ D T based on C S and C T .

3: Calculate VLM's class-level zero-shot image classification rankings { r S m,i } m = M,i = k S m =1 ,i =1 and classlevel gap vectors { g S m,i } m = M,i = k S m =1 ,i =1 .

4: for k S datasets do

5: Calculate textual similarity between the current dataset's class names and other open-source datasets' class names to construct a cost matrix.

6: Solve Optimal Transport Problem based on the cost matrix to get transport matrix γ ∗ .

7: Use γ ∗ and other open-source datasets' class-level gap vectors to predict the current dataset's class-level gap vectors. Add the predicted class-level gap vectors to the corresponding text data's feature of ˜ D S to get modified text data.

8: end for

9: Calculate textual similarity between open-source datasets' class names C S and target dataset's class names C T to construct cost matrix.

10: Solve Optimal Transport Problem based on the cost matrix to get transport matrix γ ∗ .

11: Use γ ∗ and { g S m,i } m = M,i = k S m =1 ,i =1 to predict the class-level vectors { ˆ g T m,i } m = M,i = k T m =1 ,i =1 of the target dataset. Add { ˆ g T m,i } m = M,i = k T m =1 ,i =1 to corresponding text data's feature of ˜ D T to get modified text data.

12: Use open-source datasets' modified text data and VLMs' ground truth zero-shot image classification performance to train the ranker model f m .

13: Use the ranker model f m to predict VLMs' rankings { ˆ r T , (1) m } m = M m =1 on the target dataset based on the target dataset's modified text data.

14: Use γ ∗ and { r S m,i } m = M,i = k S m =1 ,i =1 to predict the VLMs' class-level zero-shot image classification rankings { ˆ r T m,i } m = M,i = k T m =1 ,i =1 of the target dataset.

15: Calculate the average prediction rankings across all classes in the target dataset { ˆ r T , (2) m } m = M m =1 of { ˆ r T m,i } m = M,i = k T m =1 ,i =1 for each VLM as the VLM's overall prediction ranking on the target dataset.

16: Ensemble { ˆ r T , (1) m } m = M m =1 and { ˆ r T , (2) m } m = M m =1 to get final predicted rankings { ˆ r T ,ens m } m = M m =1 .

D.4 Pseudo Code of SWAB

Algorithm 1 shows the pseudo-code of SWAB.

E More Experiment Results

In this section, we provide more experimental results of SWAB.

E.1 Bridging the Modality Gap Leads to Better Image Proxies

In Section 2.3 and Figure 2, we analyze whether generated text data can act as good image proxies. Our conclusion is that due to the Modality Gap, text samples cannot directly serve as an effective substitute for images in model evaluation. To demonstrate that our method SWAB can bridge this Modality Gap and thereby make text samples a better substitute for images, we conduct the following experiment.

From the Figure 4, it is evident that the predicted model accuracy calculated using the modified text samples is closer to the true model accuracy compared to that calculated with the original text samples. This suggests that bridging the Modality Gap leads to better image proxies.

We use ImageNet as our dataset. First, we employ the method introduced in Section 3.2 to predict the gap vectors for each class of the target dataset based on gap vectors calculated on open-source datasets. Then, we add the corresponding class's predicted gap vectors to the generated text data of ImageNet to bridge the modality gap. Finally, we calculate the zero-shot classification accuracy of different models on these modified text data. To measure the consistency between the predicted Top-1

GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:d835)GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc0c)GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:d835)GLYPH(cmap:d835)

Text Samples

Modified Text Samples

GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:dfd1)GLYPH(cmap:d835)

GLYPH(cmap:d835)GLYPH(cmap:dc0c)GLYPH(cmap:d835)

= GLYPH(cmap:d835).

GLYPH(cmap:d835)GLYPH(cmap:dfd1)

ImageNet/50-shotTextSamplesPerClass

ImageNet/50-shotTextSamplesPerClass

0.8

0.8

Ground-TruthAccuracy

0.7

Ground-Truth Accuracy

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.2

0.3

0.4

0.5

0.6

0.7

0.8

PredictedAccuracy

PredictedAccuracy

accuracy and the true image classification accuracy, we calculate the Kendall Rank Correlation ( τ , higher is better) and Mean Absolute Error (MAE, lower is better). We compare the consistency metrics of text data and modified text data. It can be observed that the consistency metrics of modified text data are better, which proves our method can reduce the gap between the generated text data and the image data.

E.2 Analysis of the Modality Gap in BLIP and BEiT-3

We expand the VLM Zoo provided by LOVM by adding various variants of BLIP and BEiT-3 to enhance the diversity of the VLM Zoo. We find that the zero-shot image classification performance of the base models of BLIP and BEiT-3 is poor. Therefore, we use the retrieval version of the models provided officially. These models are obtained by fine-tuning the base models on COCO and Flickr30k through contrastive learning, thereby possessing better zero-shot image classification performance. During the pre-training of BLIP and BEiT-3, they may have used multiple loss functions in addition to the contrastive loss (BLIP), or perhaps they did not use contrastive loss at all (BEiT-3). Therefore, the existence of the Modality Gap in the feature spaces of these two models requires further investigation.

We use image samples from the STL-10 dataset and class-related text samples generated by ChatGPT. For each modality, we randomly extract 200 samples and calculate image-to-image, text-to-text, and image-to-text similarities, and plot histograms. We also observe the presence of Modality Gaps in different models through UMAP visualization [38]. Figure 5 and Figure 6 show the experiment result. Through our experiments, we verify that the feature spaces of the BLIP and BEiT-3 retrieval models still exhibit the Modality Gap phenomenon.

('BEIT3', 'retrieval_base_coco')

("BEIT3', 'retrieval_base_f30k')

('BEIT3', 'retrieval_large_coco')

("BEIT3', 'retrieval_large_f30k')

800

i2i

t2t

i2t

800

700

i2i

t2t

i2t

800

700

t2t

i2i

i2t

800

700

t2t

i2i

i2t

600

009

600

600

500

400

400

300

300

300

200

200

200

200

100

100

100

0-

0.2

0.4

0.6

0.8

1.0

0

0.2

0.4

0.6

0.8

1.0

0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Similarity Value

Similarity Value

Similarity Value

Similarity Value

('BLIP', 'retrieval_base_coco')

("BLIP", 'retrieval_base_f30k')

(BLIP", 'retrieval_large_coco')

("BLIP', 'retrieval_large_f30k*)

800

700

i2i

t2t

i2t

800

i2i

t2t

i2t

800

t2t

i2i

i2t

800

00

t2t

i2t

i2i

600

600

600

600

Aouanbay

400

300

300

200

200

200

200

100

100

0

0.0

0.2

Similarity Value

0.4

0.6

0.8

1.0

0

0.2

0.4

Similarty Value

0.6

0.8

1.0

0

0.0

0.2

Similarity Value

0.4

0.6

0.8

1.0

0

0.2

0.4

Similarity Value

0.6

8'0

1.0

14

12

(*BEIT3', 'retrieval_base_coco′)

12.5

10.0

7.5

('BEIT3', *retrieval_base_f30k')

12

10

('BEIT3', 'retrieval_large_coco')

18

16

14

('BEIT3’, *retrieval_large_f30k')

10

8

6

2.5

0.0

5.0

12

10

8

2.5

5.0

2

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

5

0

5

10

15

2

0

2

4

6

B

2.5

0.0

2.55.07.510.012.515.017.5

('BLIP', 'retrieval_base_coco")

("BLIP', 'retrieval_base_f30k")

('BLIP', 'retrieval_large_coco')

(*BLIP', 'retrieval_large_f30k’)

14

12

10

10

10

12

8

10

8

4

0

2

10

12

14

10

12

10

2.5

0.0

2.5

5.0

7.5

10.0

12.5

15.0

E.3 Experiment Result on LOVM's original VLM Zoo

We have provided experimental results on the VLM Zoo originally provided by LOVM [73]. This VLM Zoo contains 35 VLMs. Table 9 shows the experimental results, and we can see that SWAB achieves the best performance across all evaluation metrics.

E.4 Per-Dataset Experiment Result

We present the per-dataset performance comparison between our methods SWAB and ModelGPT on various datasets of LOVM benchmark in Table 10.

Table 10: LOVMBenchmark(top-1 accuracy) .We compare the per-dataset experiment result (fixing the random seed) between ModelGPT and SWAB.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to the bullet point summary at the end of the Introduction.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: We did not find significant drawbacks in the method.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our work does not involve theoretical result.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide our experiment details in Section 4.1 and Appendix D.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will open-source our code, models, and datasets in the GitHub repository.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide these details in Section 4.1.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: To ensure the reliability of our results, we conducted multiple experiments for results with randomness and reported the mean performance and standard deviation in Table 1.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We will place the relevant information in the GitHub repository.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read and fully comply with the Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: Our work does not involve negative social impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not involve these situations.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We ensure this requirement is met.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not involve new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing and research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve this.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a zero-shot image classification strategy by selecting the most appropriate Pre-Trained VLM from the VLM Zoo, relying solely on the text data of the target dataset without access to the dataset’s images. Two challenges, i.e., the “Modality Gap” across two different modalities, and the \"Capability Gap\" between the VLM’s overall ranking and its ranking for target dataset, hinder the appropriate selection of the different VLMs. To address the challenges, this paper adopts a transportation matrix to capture the relevance between open-source and target datasets, and also uses this matrix to transfer useful statistics of VLMs from open-source datasets to the target dataset. Extensive experiments on image classification datasets demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\nOriginality: \nThe main novelty of this paper is to utilize Language-Only VLM Selection to address the zero-shot image classification. This is the first time that the Language-Only VLM Selection is applied in the zero-shot scenario where the missing of image data makes it difficult to directly select the VLMs.\nQuality: \nThis paper has well motivated methods and extensive experiments.\nClarity: \nThe paper is well written with clear description and readable figures.\nSignificance: \nThe two key issues of modality gap and capability gap are considered in this paper. In addition, optimal transport-based methods are proposed to alleviate the issues and the experimental results on multiple VLMs and image classification datasets show the effectiveness of them.\n\nWEAKNESSES:\nThe main weakness is that the proposed VLMs selection strategy relies on Open-Source Datasets’ Data, which restricts the practicability of the proposed method. For the zero-shot image classification task, using the aggregated knowledge from multiple VLMs is ok, but using open-source datasets may violate the zero-shot data restrictions, or even lead to benchmark leakage. In addition, the impact of the open-source datasets is not analyzed. For example, if the categories in the open-source datasets are very different from the ones in the target zero-shot classification tasks, whether the performance of the proposed method is still remarkable?\n\nQUESTIONS:\nA comprehensive analysis on the impact of the open source datasets may improve the rationality of the proposed method for addressing the zero-shot image classification task.\n\nLIMITATIONS:\nThis paper does not include the limitation discussion. Practically, at least the issues raised in the weakness section should be considered if they cannot be addressed appropriately in current version.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWith the popularity of Vision Language Model (VLM) research in recent years, many versions have emerged, forming the VLM Zoo. This paper aims to select the most appropriate pre-trained VLM from the VLM Zoo, relying solely on texts of the target dataset without access to images. Two challenges are analyzed, namely, “Modality Gap” and “Capability Gap”. One VLM selection method (SWAB) is proposed: first calculate optimal transportation matrix to capture the relevance between open-source and target datasets; then use this matrix to transfer useful statistics of VLMs from open-source to target dataset for bridging two gaps and enhancing the VLM’s capacity estimation. Experiments are carried out across some VLMs and classification datasets to validate the effectiveness.\n\nSTRENGTHS:\n[+] Due to the differences in training data, architecture and pipelines, existing VLMs do have their own strengths in terms of capabilities. Thus, studying how to better utilize members in VLM Zoo under different scenarios is somewhat valuable.\n\n[+] The paper has rich symbols and formalizations, and the space utilization is also reasonable.\n\n[+] Some experiments are conducted to demonstrate the effectiveness of components and designs on a diverse set of datasets of image classification.\n\nWEAKNESSES:\n[-] Problem rationality. Although each member in the VLM Zoo has their own strengths, relying solely on textual descriptions to find the most suitable VLM (LOVM) seems too demanding. And language is usually compact, which can easily confuse concepts. Why cannot we use one combination of images and texts (or even text-image pairs) for VLM selection, as images are very easy to obtain in real-world scenarios? \n\n[-] Selection strategy. Given that the members in VLM Zoo have different strengths and abilities, it means that they have somewhat degree of complementarity. One natural idea is to follow ensemble learning, that is, vote on the results of each member, or select the top-k VLMs’ results for complementary combination. Why only choose the most suitable one? Please provide more comparisons and explanations. \n\n[-] Generalization. This paper carries out experiments on many datasets, but all for image classification. Since VLMs excel in handling various tasks, such as detection, segmentation, captioning, and VQA, the generalization on these tasks is still unclear.\n\nQUESTIONS:\n[-] Better visualization. For Fig. 3, there are too many symbols and formulas, increasing difficulty in understanding (the significance of using images is to reduce understanding costs). To this end, the reviewer believes that this paper can be polished better.\n\nLIMITATIONS:\nThis paper does not discuss the limitations of the work. However, the reviewer seems to find some limitations, such as the unrealistic assumptions in settings and the ability to only handle classification tasks. Please refer to Weaknesses for more details.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper proposes SWAB the modificaiton of LOVM to mitigate the negative impact of two gaps of LOVM: the modality gap and capability gap. The resutls show the effectiveness of SWAB.

STRENGTHS:
1. The motivation of this paper is clear and structure is easy to follow.

WEAKNESSES:
1. For the modality gap, the SWAB using open-source datasets, could you please provide how many image samples used in the experiment. And could you please provide the accuracy vs number of open-source image samples?
2. The SWAB seems like a learnig-based model, could you please add the ablition study for no learning based SWAB.

QUESTIONS:
1. Why adding Gaussian noise to the target generated embedding?
2. How to select weighted parameter $\alpha$ in Eq.9
3. The Auxiliary texts is generated by ChatGPT, could you please test the open-source LLM like LLama2?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article focuses on selecting the best model from a visual-language multimodal model zoo for specific downstream tasks without having images of those tasks. It provides a detailed analysis of two challenges faced in this problem — Modality Gap and Capacity Gap. This paper proposes a method called SWAB, which leverages the statistical measures of VLM on open-source tasks based on the category similarity between historical open-source tasks and target tasks, to mitigate the negative impacts of these gaps on Language-Only VLM model selection. Experimental results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1.The problem studied in this paper is novel and interesting, with significant practical value. In recent years, the number of open-source VLMs has been increasing, and their zero-shot capabilities have been improving. For practical downstream users, especially those who lack computational resources, have limited funding to label extensive evaluation datasets, or are non-machine learning practitioners with little experience in setting up and evaluating various VLMs, the Language-Only Vision Model selection (LOVM) problem studied in this article is important.\n2.This paper presents a clear and well-motivated problem statement. The authors provide a detailed analysis of the negative impacts of the Modality Gap and Capacity Gap on Language-Only Vision Model (LOVM) selection through explanations and experimental validation. They demonstrate that we cannot directly use text samples related to categories in the target dataset as substitutes for test image samples, nor can we use the average performance of VLMs on open-source datasets as proxies for their performance on specific tasks. This problem statement is compelling and well-supported.\n3.The method presented in the paper is novel and interesting. It leverages the category similarity between target tasks and open-source tasks to estimate the corresponding gap vectors and accuracy/ranking of VLMs on the target tasks, based on their gap vectors and accuracies/rankings on the open-source tasks. This idea is grounded in a reasonable assumption: the statistics of VLMs on similar categories are likely to be similar. By doing so, the method effectively utilizes the rich information contained in category similarity to make targeted use of the models' statistics on historical tasks.\n4.The evaluation approach in the article is comprehensive and detailed. The benchmark includes 23 commonly used image classification datasets and 43 widely recognized VLMs (such as CLIP, CoCa, BLIP, BEiT, among others, with different pre-training methods and model architectures), making the experimental results highly convincing. Additionally, the article compares its method with various baseline approaches, including the current state-of-the-art method ModelGPT, and demonstrates its significant superiority over these methods. The article also conducts thorough ablation studies to validate the effectiveness of each component of the proposed method.\n\nWEAKNESSES:\n1.The performance of ModelGPT is slightly inconsistent with the results in its paper. What are the reasons for this？\n2.The method of setting the \\alpha value in formula 9 is not stated. Is the same \\alpha used for all data sets? What is the exact value?\n\nQUESTIONS:\nRefer to the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}