Vision-Language Model Selection and Reuse for Downstream Adaptation
Authors: Hao-Zhe Tan
Zhi Zhou
Yu-feng Li
Lan-Zhe Guo
Venue: ICML 2025 poster (2025)
Keywords: Vision-Langage Model; Model Selection; Model Reuse
Primary area: general_machine_learning->representation_learning

Abstract: Pre-trained Vision-Language Models (VLMs) are becoming increasingly popular across various visual tasks, and several open-sourced VLM variants have been released. However, selecting the best-performing pre-trained VLM for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data limitations. To address this problem, this paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called **M**odel **L**abel **L**earning (**MLL**). The proposal contains three key modules: *model labeling*, which assigns labels to each VLM to describe their specialty and utility; *model selection*, which matches the requirements of the target task with model labels; and *model reuse*, which applies selected VLMs to the target task in an ensemble manner. The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent and the ability could grow with the number of candidate VLMs. We also introduce a new benchmark for evaluating VLM selection methods, including 49 VLMs and 17 target task datasets. Experimental results clearly demonstrate the effectiveness of the proposed method for selecting and reusing VLMs.

Vision-Language Model Selection and Reuse for Downstream Adaptation
Hao-Zhe Tan 12
Zhi Zhou
Yu-Feng Li Lan-Zhe Guo 12
Abstract
diction capabilities on visual tasks. However, though VLM shows impressive general ability, as highlighted in Radford et al. (2021), VLMs often fall short of supervised expert models in many downstream tasks. To address this limita- tion, numerous studies (Dosovitskiy et al., 2021; Yu et al., 2022; Fang et al., 2023) enhanced the zero-shot performance of VLMs by studying model architectures, pre-training datasets, and training/iinettningg methods. This effort has led to the development of many open-source pre-trained VLMs with diverse structures and parameters, contributing to VLM model hubs like open-clip (Ilharco et al., 2021), which currently hosts more than 100 pre-trained VL.Ms.
Pre-trained Vision-Laanuaaee Models (VL.M)) are becoming increasingly popular across various vi- sual tasks, and several open-sourced VLM vari- ants have been released. However, selecting the best-performing pre-trained VLM for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data lim- itations. To address this problem, this paper proposes novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL). The proposal contains three key modules: model labeling, which assigns labels to each VLM to describe their specialty and utility; model selection, which matches the requirements of the target task with model labels; and model reuse, which applies selected VLMs to the tar- get task in an ensemble manner. The proposal is highly computationally efficient and growable since the model labeling process is completed tar- get task independent and the ability could grow with the number of candidate VLMSs We also in- troduce a new benchmark for evaluating VLM se- lection methods, including 49 VLMs and 17 target task datasets. Experimental results clearly demon- strate the effectiveness of the proposed method for selecting and reusing VLMs.
With the increasing availability of open-source VLMs, se- lecting and reusing pre-trained model has become a practi- cal alternative to training one from scratch, which demands extensive training data and computing resources (Li et al., 2021), struggling with catastrophic forgetting during the incremental refinement of the trained model in open en- vironments (Guo et al., 2025). Consequently, a critical problem naturally arises: how to select a VLM to reuse for specific downstream tasks. Although we can directly utilize the best-performing model on universal dataset such as ImageNet, previous work (Fang et al., 2022) has shown that the performance of VLMs can vary greatly de- pending on dataset domain. For example, we evaluate the performance of various pre-trained VLMs in the open-clip library across several downstream tasks (I()) and within different classes of a specific task (1))). Figure I(a) reveals that each VLM demonstrates distinet strengths in zero-shot visual tasks, with single model outperforming all others across every task. Interestingly, models that perform worse on general tasks can sometimes surpass stronger models in specific downstream tasks. Furthermore, even in the same task, different VL.MS exhibit varying levels of performance across specific classes, as illustrated in Figure 1(b).
1. Introduction
Vision- Language Models (VLMss,, such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), which are pre-trained on large-scale image-text datasets, have recently attracted significant attention due to their remarkable zero-shot pre-
Therefore, it is important design VLM selection methods, and it would be better if could achieve more fine fine-grained selection, i.e., select different VLMs to handle different classes. The direct way to select model is to evaluate all candidate models' performance on the target task. How- ever, is unrealistic due to time and computational resource limitations. Additionally, previous works on model selec- tion (Tran al.. 2019; You et al., 2021) primarily focus modal models, making them unsuitable for VLM
Natioonl Key Laboratory for Novel Software Technology, Nanjing University, China "School of Intelligence Science and Technology, Nanjing University, China "School of Artificial Intel- ligenee. Nanjing University, China Correspondence to: Yu-Feng Li <liyf@nju.eduu.n>> Lan-Zhe Guo <guolz nju.edu.cn>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Vision-Languant Model Selection and Reuse for Downstream Adaptation
Accuracy downstream CLEVR-D
score per FER2013
MNIST
LIK2013
Surpriae
Disgust
PCan
DMLab
Fear
Renderal SST2
UCF101
Neutral
Happy
(a) Accuracy of VLMs on 7 specific downstream tasks.
(b) F1 score of VLMs per class in FER2013.
Figure 1. The spider charts measure 49 models' capabilities across 7 downstream tasks and classes within task, showing that the best- pest-performing models vary across downstream tasks and classes, highlighting the importance of model selection for VLM. The evaluated 49 models align with those the model hub, discussed Section 5.1.
selection since they only handle either image or text output and cannot incorporate data from the other modality. Zohar al. (2023) is the first study to focus on VLM selection, proposing to evaluate VLM performance using textual in- formation. However, the selection strategy heavily depends on the models' ground-truth performance on large-scale datasets, such as ImageNet. When models excel on large- on specific tasks, selection
data and computationally efficient in the model selection process. Moreover, the proposal highly growable since the capability could grow with the number of candidate mod- els in the model hub and the model labels also scalable since more semantic nodes can be added continually. To facilitate related research, we further introduce compre- hensive benchmark for evaluating VLM selection methods. The benchmark includes 49 pre-trained VLMs and 17 tar- get datasets as downstream tasks. The ground-truc truth model ranking for each target task is provided for evaluation. We construct semantic graph that contains more than 9000 commonly used visual concepts to pre-test VLMs. The ex- periments demonstrate the effectiveness of our approach in both selecting and reusing VLMs, while also validating the scalability of the model hub based on our proposal.
scale datasets but strategy effectiveness drops, as shown in our experiments.
To this end, we introduce a novel paradigm select and reuse VLMs called Model Label Learning (MLL). The core idea is to organize candidate pre- pre-trained VLMs into model hub and describe the specialty and utility of each VLM as the model's label in some manner. When faced with a new downstream task, we can match the task requirements with the model labels to select and reuse models. Specifically, the proposal contains three key interconnected modules: model labeling, model selection, and model reuse. In the model labeling process, we construct semantic graph with com- monly occurring visual concepts and representative samples, and each model undergoes pre-testing on the semantic graph to generate its model label, which describes its capability on these semantic classes. In the model selection process, we generate caption descriptions for both the nodes in the semantic graph and the categories be classified in the target task to compare their similarity. This enables us to evaluate the model's performance on the target classes by aligning the matched semantic nodes with the model labels. In the model reuse process, we apply an ensemble strategy that combines the predictions of the selected models in single class and chooses the highest confidence in all classes as the final prediction on the target task.
In summary, our contributions are as follows:
We highlight that the performance of pre- trained VLM varies across different downstream tasks, even among classes within the same task. To address this, we first propose an important yet rarely studied problem called VLM selection and reuse, which will promote the deployment of VLM in more real tasks.
Problem:
We propose a novel paradigm called Model Label Learning, which includes the processes of model labeling, selection, and reuse. This paradigm is both time- and data-efficient, and highly scalable. can give birth new VLM model hubs, which can simplifying user selection and reuse of VLMs for their tasks.
Method:
We introduce new benchmark for eval- uating pre-trained VLM selection and reuse methods, advancing research in this field. Experimental results validate the effectiveness and scalability of our pro- posal for selecting and reusing VLMS.
Evaluation:
The model labeling process is completed immediately when the candidate VLM is added to the model hub, therefore, it is target task independent, which means the proposal is both
2

---

Vision- Vision-Language Model Selection and Reuse for Downstream Adaptation
2. Related Work
constructing specifications describe the capabilities of the model, closely aligning with our idea of model labeling. Compared with previous selection methods, learnware en- ables scalable and efficient model selection across diverse architectures and input types within a unified framework, improving as the system expands. Model specification is central to the learnware paradigm, describing the function- ality of each model. Recent works (Tan et al., 2024) on learnware paradigm are built on Reduced Kernel Mean Em- bedding (RKME) (Wu et al., 2021), which maps training data distributions to points in Reproducing Kernel Hilbert Space (RKHS) and identifies models by calculating the RKHS distance between RKME specifications. Further- more, Guo et al. (2023) enhanced RKME for heterogeneous label spaces, while Tan et al. (2023) addressed challenges in heterogeneous feature spaces. Besides, Zhou et al. (2023) extended the learnware paradigm to the domain of the diffu- sion models selection. However, learnware requires training data to construct specifications. Considering the scale of VLM pre-trained datasets, it is unrealistic to construct spec- ifications for Iearnware to select models due to limited time and computational resources.
Vision-Language Model. In recent years, there have been sigmificant advances in the field of Vision-Language Models (VLMs), including notable models such as CLIP (Radford et al., 2021), BLIP (Li et al., 2022), etc. These models leverage large-scale datasets containing image-text pairs, such as WIT (Srinivasan et al., 2021), to align visual and text features within a shared embedding space, which has led to impressive capabilities in feature extraction, particularly in the realm of zero-shot visual tasks. Tremendous works (Dosovitskiy et al., 2021; Yu et al., 2022; Fang et al., 2023) attempted to improve the zero-shot capabilities of VLMs by focusing on model architecture, pre-training datasets, and waining/fine-tuningg tuning methods, which lead to the emergence of numerous open-source pre-trained VLMs. As a result, several VLM model hubs are constructed, such as open-clip (Ihharco et al., 2021) and HuggingFace (Wolf et al., 2020), which provide access to numerous VLMs. However, these model hubs lack effective model selection mechanisms; users cannot directly select suitable models for their tasks, they can only select models based on some quantitative indicators, such as download volume, popularity, etc.
3. Preliminaries
Model Selection. As pre-trained models become increas- ingly diverse, how to select appropriate pre-trained models to tackle specific tasks has become a significant challenge. Many researchers have started to focus on this aspect. For example, Negative Conditional Entropy (NCE) (Tran et al., 2019) proposes an information-theerrti quantity learn the transferability and hardness between classification tasks; LEEP (Nguyen et al., 2020) utilizes source prediction prob- abilities instead of hard labels compared with NCE; LogME (You et al., 2021) estimates the correlation between source model features and the target outputs by maximum evidence; MetaGL (Park et al., 2023) solves the model selection prob- lem on graph data by introducing a meta-learming method: EMMS (Meng et al., 2023) uses weighted linear regression to estimate the transferability of candidate models; Model Spider (Zhang et al.. 2024) uses a re-ranking mechanism to enhance the model co-embedding. Although these methods achieve well werf-perfo in different settings, most of them focus on single-modal which cannot be directly used for VLM selection. Moreover, the training data for VLM is inaccessible, which introduces more challenges. Model selection for VLM is still relatively new topic. LOVM (Zo- har et al., 2023) uses a text dataset to describe the prediction task to train a linear model to predict the performance of the VLM. However, this method can only exploit text informa- tion and becomes less effective when there is a domain shift between downstream tasks and training tasks.
3.1. Zero-Shot Vision Task of VLM
Pre-trained VLMs for zero-shot visual tasks are built using two encoders: image encoder and text encoder. The image encoder is used to transform an image into a vector embed- ding, which presents its feature. The text encoder tokenizes the text input and generates a embedding representation by the text token. Let X' + RR" denotes the image encoder and T : y + R" denotes the text encoder, where X e X is the image input, Y E ) is the text input, and n is the dimension of the shared multi-modal embedding space of text emheddings and image embeddings.
In a particular downstream task T, there are Cr classes Cr. For a image x E X, we obtain the image
Yr = ] embeddings I(xx) given by the image encoder I and the text embeddings T④y) of class y produced by the text encoder T. Then, the prediction ÿ of image can be obtained as
exp (sim (()) /T) exp (sim (I(x), (v)))
9 1= =arg max SEYT
(I)
y EYE
where sim (..)) denotes cosine similarity, represents the temperature determined by VLMs.
3.2. Problem Setup
Assume the model hub has M pre-trained VLMs {fm = [Im, Tm HIM where Im denotes the image en- coder of the VLM fm and Tm denotes the text encoder of
Learnware. Learnware (Zhou & Tan, 2022) is a novel paradigm that explores more effective model selection by
3

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
Ranking for Each Class
Ensemble Predictors
Model Hub
Models
Model Labels
V1
V1
M
S
V/
M2
S₂
S Ensemble
Selected Models Each Class
M
Pre- Ros-Testod
S
4 Select Mudel Labels and
V/
Similar Nodes
Label conoralizationn
Similar Nodes Lists
M
S
V/3
@ Prediction for Downstream Task By Ensemble Predictors,
Select Similar Nodes for Each Class
Task Classes
Node Caption
2 Semantic
① Task Generation
Limbedding
Nodes Match
Task Captions
Task Data
pappo
rptz
tably
persze
Semantic Graph
Embedding Space
Downstream Task
Model Labeling
Model Selection
Model Reuse
The framework of MLL paradigm. Models added to the hub first undergo pre-testing phase, during which they are assigned labels that describe their specific functionalities in the labeling module. When a downstream task is presented, the system selects relevant models in the selection module and ensembles them address the task.
Figure 2.
the VLM fm. There are two stages our setting: sub- mission stage for model developers to upload models to model hub and identification stage for users to select suitable models for their tasks from the model hub.
appropriate VLMs from the model hub to address users' downstream tasks based on their requirements and the model labels generated in the submission stage?
In submission stage, model developers submit VLM f. to the model hub, and the model hub assigns label S., to the model to describe its specialty and utility. It is particularly emphasized that uploaded models are anonymous, meaning we do not have access to their training data.
4. Our Approach
As illustrated in Figure 2, the MLL paradigm consists of three key modules: model labeling, model selection, and model reuse. In the model labeling process, MLL constructs semantic graph G with commonly occurring visual con- cepts and representative samples as the evaluation datasets. When models are submitted to the model hub, they are pre- tested the semantic graph and assigned labels Sm. which describe their capability on these semantic classes. In the model selection process, generate caption descriptions for both the nodes in the semantic graph and the categories in the target task to compare their similarity. This enables us to evaluate the model's performance on the target classes by aligning the matched semantic nodes with the model labels. In the model reuse process, we apply an ensemble strategy that combines the selected models' predictions on single class and chooses the highest confidence across all classes as the final prediction.
In identification stage, user attempts to select VLMs from the model hub for the zero-shot downstream task T, by up- loading general information about the task, such classes, domain type, and task type, to describe their requirements. We subsequently utilize this information to select and reuse suitable VLMs, based on the model labels which have been established in the submission stage.
Two main problems in our settings are:
I) In submission stage, how can we design label fully characterize the capabilities of the submitted VLM?
2) In identification stage, how can we select and reuse

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
4.1. Model Labeling
following is example of prompt used generate caption of the class cat.
To thoroughly characterize the capabilities of the model,
we initiate the process by constructing a Semantic Graph g evaluation datasets utiliizigg the WordNet (Miller, 1995) synsets. Firstly, we represent each synset in WordNet as a corresponding node within the semantic graph and es- tablish links between nodes based on their relationships of hypernyms and hyponyms. Subsequently, to capture the world image distribution associated with each node, we randomly select images x, from sample datasets (detailed in Section 5.1) to serve representations for each node v. Due to the limited information in synset name, we also need obtain the caption dataset Dg = {d E Vo} for label generalization where Ve denotes the set of nodes in Semantic Graph G, d, denotes the caption node v. We use "(synset name} which is {synset definition))" as the caption for each node, where "(synset name)" and "(synset d correspond to the synset name and definition of a synset. Utilizing the constructed semantic graph, we generate label Sm for each VLM fm in the model hub that accurately reflects its capabilities.
Generate long detailed caption for the natural picture of cat in the image classification. e.g., "The natural picture of cat, which is... Generate long caption for cat within 50 words.
where natural picture and image classification can be re- placed with the domain and task descriptions, while cat can be substituted with the specific class name for target task.
Then, we can use language model to generate embeddings of graph captions Dg and target task captions Dr. By com- paring the cosine similarity between the embeddings, we can select the top k nodes for each class based on similarity and construct transfer matrix Z (Zey) E R Selecttt |x|Y", where / represents all selected nodes. Additionally, Evy represents the similarity of captions between graph node e and task class y if v among the top k nodes that exhibit the highest similarity with task class y. Otherwise, it will be set to O. Subsequently, the precision Pm,e for each model fm at the graph nodes is defined as follows.
m, sim (I), Tm (d.)),, X,
8% = momm x E X
(3)
Pm,,
1
Σ ZEX,
I
arg max Sm, VEVacletted
1x
Sm = (sm v Vo}
(4)
where Ian (-), Tm) denotes the image encoder and the text encoder of model fm
By leveraging the transfer matrix Z, we can further derive the precision prediction Pm.y for each class y within the downstream task as described below.
Specifically, the constructed semantic graph allows for the seamless addition new nodes and the incremental updat- ing of model labels based on existing foundations. As the nodes in the semantic graph are expanded, its ability to re- flect the performance capabilities of the models is enhanced. Once have obtained labels for each model, them for effective model selection.
Pm..
Pma. Zuy
(6)
VVV Seleated
When a model excels in specific class, it may incorrectly handle data not belonging to that class. Consequently, we need to select models that perform well on specific classes while also maintaining good overall performance. Thus, we introduce a weight parameter a to balance class perfor- mance with overall performance. Then, the reuse metric for model fm in class y is defined as:
utilize
4.2. Model Selection
In the model selection module, given a downstream task T' with CT classes Yr- ( in order to utilize the obtained model labels S,,, we need to match the down- stream task classes Yr with the semantic graph nodes Vg. However, can not match well using original class names. Inspired by previous work (Zohar et al., 2023), we construct expanded captions for both the downstream task classes and the semantic graph nodes. Large Language Models (Ope- nAI, 2023) have made significant advancements, facilitating the generation of text data. Assuming general information about the downstream tasks, such as task types and target domain, is accessible, we use GPT-4 with specific prompts to generate descriptions for each class as shown below, cre- ating the caption dataset Dr for downstream task T. The
1-a
Σ YEYr
Tm.y Pmmy
|Y
Pmay
4.3. Model Reuse
better utilize the selection and harness the capabilities of models in the model hub, we introduce a specific count k of models reuse for each class y, we select up highest-score model to form the ensemble predictor F} {fm fm top & ((rmg) } During testing, for the
5

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
Algorithm 1 Model Selection & Reuse
4.4. Discussion
Input: Model hub M, model labels {Sm) semantic graph G, semantic graph caption dataset Dg, count ls of reused
Our proposal achieves higher accuracy, efficiency, and scal- ability. In terms of accuracy, the proposal elucidates the functionalities of VLMs by labeling models with semantic graph that covers the most common visual concepts and rep- resentative samples to describe different data distributions, enabling more accurate identification of suitable models for users' target tasks. For efficiency, the proposal generates model labels when the pre-trained model is uploaded the model hub, thus, it is highly efficient in the model selection process, without the need to run the candidate models on the target dataset. Regarding scalability, the concepts in the semantic graph can be continually added, thus, the model labels are scalable flexibility. Moreover, as the number of VLMs in the model hub increases, our proposal identifies higher-quality models, leading improved performance on zero-shot downstream visual tasks.
models pre-class, target task T = (X,Y
Output: Task prediction {0}
1: Construct caption dataset Dy for target task T.
2: Match similar nodes /Seeected in Vg with Y by captions
De and Dr.
3: Construct transfer matrix Z IR
V X based
caption similarity of /Selected and Y.
4: for fm E FM do
5: Calculate reuse metric Tm.y for each class in Y by
Eq. 6.7.
6: end for
7: for y E Y do
8: Select k models ensemble predictor F
[Jm I fm E top-k (mmw)mm)
9: Calculate prediction ý for x by Eq. 9, 10).
10: end for
5. Experiments
11: Return Task prediction {1))
5.1. MLL Benchmark
data E X of the downstream task, ensemble predictor Fi infers the confidence p (z)) of class y:
To evaluate the capabilities of the MLL paradigm in zero- shot visual tasks with VLMs, we need to obtain a set of sampling datasets for constructing semantic graph G, along with another set dedicated to downstream target tasks. For this study, we select 49 VL.ss, 5 Sample Datasets, and 17 Target Datasets. Additionally, we collect general infor- mation about task types and domains associated with each dataset to provide a task description. For testing selected models on target tasks, we utilized same prompting strategy outlined in Radford et al. (2021), ensuring consistency in our evaluation methodology.
exp (sim (Im (x), Tm (v))) exp (sim (m(xx), , Tm (1)))
(())) = Σ Wmy"
Σ
N'EYr
(8)
where Wm,y denotes the ensemble weight obtained from the output probability entropy H of each model within Fý- Wm, is defined as:
H ((sim (x), Tm (v)l V YrH) H (Isim(Im (2), Tm (8) V E
Wm,y
Σ
Model Hub. We leverage the open-clip library (Ilharco et al., 2021), which encompasses a diverse set of pre-trained VLMs across multiple architectural frameworks, such as (Tooovvvvts et al., 2021) and ConvNet(Liu et al., 2022). These models have been pre-trained on variety of large- scale datasets, such as WIT (Srinivasan et al., 2021) and LAION-2B (Schuhmann et al., 2022). We select 49 models from this library to form our model hub for the purpose of our experiments. All models used in the experiments are directed downloaded from the library.
ET
(9)
Vishniakov et al. (2024) has shown VLMs predictions fre- quently exhibit overconfidence, which can degrade the per- formance of ensemble predictions, especially when the model makes incorrect predictions. To address this, we introduce Wmy based on the probability entropy H as a measure of uncertainty and assign lower weights to models with high confidence when they are overconfident, which reduces the impact of overconfidence, potentially erroneous predictions on the final ensemble output, thereby enhancing the robustness and accuracy of model's overall predictions.
Datasets. We utilized datasets, ImageNet (Deng et al., 2009), ImageNet- V2 (Recht et al., 2019), SangeNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a). as Sample Datasets for semantic graph construction. Additionally, we used 17 commonly used datasets and their task general in- formation as Target Datasets to evaluate VLM selection and reuse methods in zero-shot visual tasks (as shown in Table 5). These datasets demonstrate diversity terms of
Then, the class with the highest confidence is selected as the prediction ý for æ:
())= = arg max p (x:)
(10)
VEY,
Flow of model selection and reuse of MLL Paradigm are summarized in Algorithm 1.
6

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
Table Comparison the zero-shot performance on downstream tasks by reusing a single model (A: 1). The best performance highlighted in bold.
Methods
CIFARI00
Country211
CLEVR-D
DTD
DMLab
Flowers102
MNIST
OxfordPet
PCam
INB
0.8599
0.3121
0.1262
0.6787
0,1940
0.8761
0.7956
0,9401
0.5332
ModelGPT
0.8599
0.3121
0.1262
0,6787
0.1940
0.8761
0,5648
0.9401
0,4990
Proposal
0.8773
0.3159
0.1361
0.6910
0.2111
0.8914
0.8101
0.9428
0.5003
Methods
FER2013
Food101
GTSRB
RESISC45
Rendered-SSS -ST2
StanfordCars
STL10
UCF101
Avg.
INB
0.2859
0.9553
0,5391
0.6139
0.5199
0.9487
0.9889
0.7702
0.6434
ModelGPT 0.4014
0.9553
0.5391
0.6139
0.5800
0.9487
0.9639
0.7702
0.6367
Proposal
0.4933
0.9566
0.5636
0.6437
0.5206
0.9568
0.9878
0.7961
0.6620
Table Comparison of the shot performance 17 downstream tasks by ensemble models per class (I: = 3). The best performance highlighted in bold.
Methods
CIFAR100
Country211
CLEVR-D
DTD
DMLab
Flowers102
MNIST
OxfordPet
PCam
INB
0.8977
0.3228
0.1048
0.6968
0.1296
0.8873
0.7847
0,9433
0.5002
ModelGPT
0.8949
0.3243
0.0907
0.6957
0,1338
0.8876
0.7888
0.9490
0,5002
Proposal
0.8923
0.3238
0.1171
0.7053
0.1573
0.8720
0.8101
0.9428
0.5003
Methods
FER2013
Food101
GTSRB
RESISC45
Rendered-SSTT
StanfordCars
STL10
UCF101
Avg.
INB
0.2636
0.9606
0.5938
0.6615
0.5332
0.9540
0.9949
0.7966
0.6498
ModelGPT 0.2653
0.9611
0.5980
0.6555
0.5299
0.9533
0.9957
0.7933
0.6480
Proposal
0.4933
0.9566
0.5636
0.6800
0.5233
0.9541
0.9854
0.8092
0.6639
domain, number of classes, and task types. They encompass various domains, including animals, food, text, landscapes, remote sensing, medical, and transportation. Additionally, they cover a range of tasks such as image classification, geo- localization, optical character recognition, facial expression recognition, land cover classification, and object distance estimation. To eliminate interference of predication from ditional modules during evaluation, all tasks can be assessed using the same VLM architecture.
of the semantic graph. We randomly selected up 75 pictures for each node, which are all from the sample datasets and reflected the distribution of the node's concepts, In addition, we used the OpenAl text-embedding- 3-large model to obtain their caption embeddings. By calculating the cosine similarity between these embeddings, we can accurately match the correspondence between the semantic graph nodes and the downstream task classes.
Compared Methods. Initially, we compare our proposal with ImageNet Baseline (INB), which simply select the model which has best performance the ImageNet in the model hub to reuse. Additionally, we compare it with a VLM selection method called ModelGPT (Zohar et al., 2023). ModelGPT employs generated captions and syn- onyms for target task classes as substitutes for images of those classes, then evaluates the performance of VLMs by measuring their ability correctly classify the captions and synonyms into their corresponding classes, which serves as the reuse metric in combination with INB. A linear model is then learned between the reuse metric and ground-truth performance on training downstream tasks. Finally, the zero-shot ability of VLMs on the target task is predicted using this linear model and the reuse metric.
Evaluation Metrics. our benchmark, methods are ex- pected to select models from hub of 49 pre-trained VLMs and reuse them across 17 target datasets downstream tasks achieve better performance. Notably, all models selected for use without additional fine-tuning, all downstream tasks are zero-shot. We use Acc. to evaluate methods' performance on both downstream target tasks and the average performance across all tasks.
5.2. Experiment Setup
Semantic Graph Construction. In order to build rich semantic graph G. we used WordNet synsets to carefully select 9055 nodes, covering many fields such as animals, tools, clothing, vehicles, plants, ensuring the representative-

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
Average acturacy on 17 tasks
Scalability of the model hub
Table 3. Average performance of our proposal 17 downstream tasks compared with different a by reusing single model (k = 1)
0.65
a
0.5
0.6
0.7
0.8
0.9
0.60
Avg.
0.6569
0,6571
0.6620
0.6600
0.6590
0.55
INB
MxtcKiPT Proposal
0.50
Table 4. Average Performance and Interface Time Cost of our pro- posal on 17 downstream tasks by different reusing model count k.
13
19
25
31
37
43
49
Number of VLM in the model hub
Interface Time Cost Compared with k= =
k
Average Accuracy
Figure 3. Average performance on 17 downstream tasks with the scaling of the model hub.
1
0.6620
1.000
2
0.6637
1.927
3
0.6639
2.233
4
0.6571
2.467
Implementation Details. We adopt the official code to im- plement ModeIGPT. For fair comparison, the experiment utilizes the ground truut performance of VLMs on Sample Datasets for ModelGPT to train its linear model, and then evaluate on the benchmark. For both INB and ModelGPT, the experiment selects the model with the highest predictive performance given by the method for reuse in the target task. Specifically, we employ the same prompting strategy outlined in Radford et al. (2021), which uses the prompt "a photo of {class) where "(class)* "i is replaced by the task class. All selected models are utilized without any further tinentuning given that all downstream tasks are conducted zero-shot manner. Additionally, the weight a for model selection in our setting is set to 0.7. All experiments are conducted on NVIDIA A800 GPUs.
5
0.6556
2.700
6
0.6594
2.900
each class tends to yield better results. Additionally, when the models available in the model hub are generally weak, as seen in several datasets, such as CLEVR-D, relying on ensemble methods may introduce more noise than benefit. In these cases, single model per class often provides the ultimate balance between simplicity and effectiveness.
Scalability of Model Hub. We design a scenario where the model hub starts from scratch and gradually expands until it contains all available VLMs. Figure 3 provides a detailed illustration of the average performance of 17 down- stream tasks throughout 30 randomly generated expansion schemes. The results show that the model hub grows, our method can more efficiently reuse the well well-oerronming VI.M models for various tasks, reducing the limitations in model selection and boosting system performance across a range of visual tasks. This shows that our method not only highly effective the present but also holds the potential for continued improvement as the model hub grows.
5.3. Experiment Results
Zero-shot Performance. In our setup, the goal is opti- mize the performance of VLMs downstream zero-shot visual tasks. Therefore, in Table and Table 2, we com- pare the performance of different model selection methods across 17 target datasets. We set two values for the count of reused models, specifically 1 and 3, to test the ef- fects of using a single model versus an ensemble of three models per class. In the ensemble setup, we first selected the top-k models from each method and then applied the same reuse strategies discussed in Section 4.4. The results show that our method achieves high performance on most downstream tasks. ModelGPT largely aligns with INB, indicating a strong correlation in their selection strate- gies. When INB fails to select performmmnm model, ModelGPT also struggles with selection. By comparing different counts i of reused models, MLL demonstrates that reusing the model with the best performance per class is often sufficient to outperform baseline methods in most downstream tasks, highlighting the practicality of the MLL paradigm. We also find that in datasets with a limited num- ber of classes, such as PCam, employing a single model for
Ablation Study. To analyze the influence of key hyperpa- rameters in our proposal, we conducted an ablation study focusing on the effects of the parameter a (as defined in Equation 7) and the reuse model count k. The impact of a is presented in Table 3. The results demonstrate that our proposal consistently outperforms the compared meth- ods across range of a values, highlighting the robustness of the model to this parameter. Additionally, the effect of l is shown in Table 4, where the results show that us- ing a smaller number of models strikes an optimal balance between performance and computational efficiency, with minimal performance loss compared to larger ensembles.
8

---

Vision- Vision-Language Model Selection and Reuse for Downstream Adaptation
6. Conclusion
References
In this paper, we explore how to select and reuse pre-trained VLMs for specific downstream task. To the best of our knowledge, this problem has rarely been studied. To ad- dress this, we propose novel paradigm called Model Label Learning (MLL) that assigns each VLM label to describe its utility on representative visual concepts. The MLL paradigm contains three key modules: model labeling, model selection, and model reuse. The proposal is highly efficient, scalable, and convenient for both model devel- opers and users. Moreover, we introduced a benchmark for evaluating pre-trained VLM selection and reuse meth- ods that contain 49 pre-trained VLMs and 17 target datasets. Experiments demonstrate that the proposal can achieve state- of-the-art VLM selection performance, and the ability to deal with downstream tasks could grow with the scale of the model hub, showing the potential of building large model hubs with advanced model selection mechanisms.
Alex, K. Learning multiple layers of features from tiny images. 2009.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 10od-101- mining discriminative components with random forests. In Proceedings of the 13th European Conference on Com- puter Vision, pp. 446-461, 2014.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 105(10):1865-1885 2017.
Cimpoi, M.. Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Coates, A., Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceed- ings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 215-223, 2011.
In future work, will endeavor to develop novel model hub based on the MLL paradigm presented in this paper, allowing valid VLM developers from all over the world to submit their models. When users work on visual tasks, they will be able to select and reuse models from the hub. The limitation of this paper is that the current implementa- tion focuses solely on VLMs and visual classification tasks. We will further attempt extend our paradigm to more model types that have significant architectural differences compared with VLMs, and more complex tasks.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proceedings of 2009 IEFE Conference on Computer Vision and Pattern Recognition, pp. 248 248-255, 2009.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenbomn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Code Availability Statement
The implementation code of benchmark and our proposal for the MLL paradigm is available ht ttsp:: : / /iibbbb. com/ Bann/LADDS22MML
Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data determines distributional robustness in Contrastive Language Image Pre-training (CLIP). In Proceedings of the 39th International Confer- ence on Machine Learning, pp. 6216662266 62344,444 2022.
Acknowledgements
This research was supported by National Science Founda- tion of China (62306133, 62250069) and Key Program of Jiangsu Science Foundation (BK20243012). We would like to thank the reviewers for their constructive suggestions.
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang. X., Huang, T., Wang, X., and Cao, Y. EVA: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19369 2023.
Impact Statement
This paper aims to explore how to select and reuse pre- trained VLMS for a specific downstream task. We are con- vinced that the majority of the impacts are positive, which is not necessary specifically highlight any particular impacts in this paper. Moreover, we expect that the responsible use of technology will naturally promote discussions around best practices and regulations for implementing methods.
Goodfellow, J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D.,, Lee, D.-H., et al. Challenges in representation learn- ing: A report on three machine learning contests. In Proceedings of the 20th International Conference Neu- ral Information Processing, pp. 117-124, 2013.
Guo, L.-Z., Zhou, Z., Li, Y.-F., and Zhou, Z.-H. Identi- fying useful learnwares for heterogeneous label spaces.

---

Vision-Langune Model Selection and Reuse for Downstream Adaptation
In Proceedings of the 40th International Conference on Machine Learning, pp. 12122-12131, 2023.
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A ConvNet for the 2020s. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 11976-1 1196, 2022.
Guo, L.-Z., Jia, L.-H., Shao, J.-J,, and Li, Y-F. Robust semi- supervised learning in open environments. Frontiers of Computer Science, 1988):193445, 2025.
Meng, F., Shao, W., Peng, Z., Jiang, C., Zhang, K., Qiao, Y., and Luo, P. Foundation model is efficient multimodal multitask model selector. In Advances in Neural Informa- tion Processing Systems, pp. 33065-33094, 2023.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp.
Miller, G. A. WordNet: A lexical database for English. Communications of the ACM, 38(11)339-41, 1995.
2021a.
Nguyen, C., Hassner, T., Seeger, M., and Archambeau, C. LEEP: A new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference on Machine Learning, pp. 7294- 7305, 2020.
Hendryeks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Nilsback, M.-E. and Zisserman, A. Automated flower classi- fication over large number of classes. In Proceedings of 2008 6th Indian Conference on Computer Vision Graph- ics & Image Processing, pp. 722-729, 2008.
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C.,, Car- lini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L.
OpenCLIP, 2021.
OpenAL. GPT-4 technical report. CoRR, abs7233.087777 2023.
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. Sung, Y-H.,, Li, Z., and Duerig, T. Scaling up visual and vision- usioonlanguage representation learning with noisy text supervision. Proceedings of the 38th International Conference on Machine Learning, pp. 4904- 4916, 2021.
Park, N., Rossi, R. A., Ahmed, N., and Faloutsos, C. MetaGL: Evaluation- free selection of graph learning mod- els via meta-learming. In Proceedings of the 11th Interna- tional Conference on Learning Representations, 2023.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, Cats and dogs. In Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498- 3505, 2012.
Johnson, J., Hariharan, B., Van Der Maaten, L.., Fei-Fei, L., Lawrence Zitnick, C.,, and Girshick, R. CLEVR: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910 2017.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Interna- tional Conference on Machine Learning, pp. 2021.
Krause, J.. Stark, M., Deng, J., and Fei-Fei, L. 3D ob- ject representations for fine- fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 554-561, 2013.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do Ima- geNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learn- ing. pp. 1389954000 2019.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE,
2324,, 1998.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.. Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. LAION-5B: An open large-scale dataset for training next generation image-text models. In Advances in Neural Information Processing Systems, pp. 1-278-25294, 2022.
Li, J., Li, D., Xiong, C., and Hoi, S. BLIP: Bootstrapping eigguage-image pre-training for unified laniuael understanding and generation. In Proceedings of the 39th International Conference on Machine Learning, pp- 12888-12900 2022.
Li, Y.F.. Guo, L.-Z., and Zhou, Z.H. Towards safe weakly supervised learning. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 43(1):334-3444 2021.
Soomro, K., Zamir, A. R., and Shah, M. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212. 2012.
10

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., and Najork, M. WTT: Wikipedia-basee image text dataset for multimodal multilingual machine learning. In Proceed- ings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp- 2443-2449, 2021.
You, K., Liu, Y., Wang, J., and Long, M. LogME: Practical assessment of pre-trained models for transfer learning. In Proceedings of the 38th International Conference on Machine Learning, pp. 12133-12143 2021.
Yu, J., Wang, Z., Vasudevan, V., Yeung, L.,, Seyedhosseini, M., and Wu, Y. CoCa: Contrastive captioners image- text foundation models. Transactions on Machine Learn- ing Research, 2022.
Stallkamp. J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural Networks, 32: 323-332, 2012.
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A. S., Neu- mann, M., Dosovitskiy, A., al. A large-scale study of representation learning with the visual task adaptation benchmark. CoRR, abs/1910.048671 2019.
Tan, P., Tan, Z.-H., Jiang, Y., and Zhou, Z.-H. Handling leamwares developed from heterogeneous feature spaces without auxiliary data. In Proceedings of the 32nd Inter- national Joint Conference on Artificial Intelligence, pp. 4245, 2023.
Zhang, Y.-K., Huang, T.-.,, Ding, Y--.,, Zhan, D.-C., and Ye, H.J. Model spider: Learning to rank pre-trained models efficiently. Advances in Neural Information Pro- cessing Systems, pp. 13692-13719, 2024.
Tan, Z.-H., Liu, J.-D., Bi, X.-D., Tan, P., Zheng, Q.C.,, Liu, H.-T., Xie, Y., Zou, X.-C., Yu, Y., and Zhou, Z.-H. Beimingwu: A learnware dock system. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 5773-5782, 2024.
Zhou, Z., Guo, L.-Z., Song, P.-.,, and Li, Y--F You only submit image find the most suitable generative model. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems Workshop on Diffusion Models, 2023.
Tran, A. T., Nguyen, C. V., and Hassner, T. Transferability and hardness of supervised classification tasks. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1395-1405, 2019.
Zhou, Z.-H. and Tan, Z.-H. Learwaare: Small models do big. CoRR, abs/2210.03647, 2022.
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant CNNs for digital pathol- ogy. Proceedings of the 21 International Conference on Medical Image Computing and Computer Assisted Intervention, pp. 210-218, 2018.
Zohar, O., Huang, S.-C., Wang, K.-.,, and Yeung, S. LOVM: Language-only vision model selection. In Ad- vances in Neural Information Processing Systems, pp. 33120-33132, 2023.
Vishniakov, K., Shen, Z., and Liu, Z. ConvNet vs Trans- former, Supervised vs CLIP: Beyond ImageNet accuracy. In Proceedings of the 41st International Conference Machine Learning, 2024.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning ro- bust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, 2019.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistae, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of-the-arr natural language processing. In Proceedings of 2020 Conference Empir- ical Methods in Natural Language Processing, pp. 38-45, 2020.
Wu, X.-Z,, Xu, W., Liu, S., and Zhou, Z.-H. Model reuse with reduced kernel mean embedding specification. JEEE Transactions on Knowledge and Data Engineering. 35 (1):699-710, 2021.

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
A. Details of Benchmark
In this section, we provide detailed insights into our benchmark utilized for evaluating VLM selection and reuse methods. Table outlines the datasets used in the benchmark, highlighting the type of domain and task for each dataset. This breakdown is essential for understanding the context and effectiveness of the models assessed in our study. Table presents general information on the model hub, including model architecture, pre- rre-trained datasets, parameters, FLOPs, and accuracy on ImageNet.
Table 5. Details on the datascts used in the benchmark, which contain the type of domain and task.
Dataset
Domain
Task
ImageNet (Deng et al., 2009)
natural picture
image classification
ImageNet-Yo V2 (Recht al., 2019)
natural picture
image classification
ImageNet-Skette (Wang et al., 2019)
Sketch picture
image classification
ImageNet-A A (Hendrycks et al., 2021b)
natural picture
image classification
ImageNet-R (Hendrycks ct al., 2021a)
domain picture
image classification
(e.g., art, cartoon)
CIFAR 100 (Alex, 2009)
natural picture
image classification
Country211 (Radford et al., 2021)
natural picture
geo-localization
CLEVR-D (Johnson et al., 2017)
natural picture
object distance estimation
DTD (Cimpoi ct al., 2014)
texture picture
image classification
DMLab (Zhai al., 2019)
natural picture
object distance estimation
Flowers 102 (Nilsback & Zisserman, 2008)
flower picture
image classification
FER2013 (Goodfellow et al., 2013)
facial picture
facial expression classification
Food101 (Bossard et al., 2014)
food picture
image classification
GTSRB (Stallkamp et al., 2012)
traffic picture
image classification
MNIST (LeCun et al., 1998)
digit picture pel photograph medical picture
image classification
OxfordIITPPe (Parkhi al., 2012)
image classification
PCam (Veeling et al., 2018)
image classification
Rendered SST2 (Radford et al., 2021)
text picture satellite picture
optical character recognition
RESISC45 (Cheng et al., 2017)
land cover classification
StanfordCars (Krause et al., 2013)
car picture
image classification
STL10 (Coates et al., 2011)
natural picture
image classification
UCF101 (Soomro et al., 2012)
vidco frame
action recognition
12

---

Vision-Language Model Selection and Reuse for Downstream Adaptation
Table Details on model hub used the benchmark, which contain the model architecture, pre-trained datasets, parameters, FLOPs,
Accuracy ImageNet
ID
Model Architecture
Pretrained Dataset
Params (M)
FLOPs (B)
ImageNet Acc.
1
RN50
openai
102.01
18.18
0,5982
2
RN50
cc 2m
102.01
18.18
0,3591
3 RN101
openai
119.69
25.5
0.6228
4
RN101
yfcc15m
119.69
25.5
0.3407
5
RNIOI-quickgelu
openai yfcc 5m
119.69
25.5
0.6228
6
RN101-quickgelu
119.69
25.5
0,3487
7
RN50x4
openai
178.3
51.82
0.6627
8
RN50x64
openai
623.26
552.65
0.7391
9
ViT-B-32
openai
151.28
14.78
0.6332
10
ViT-B-32
laion2b_el6
151.28
14.78
0.6565
11
ViT-B-32
datacommp_xls
151.28
14.78
0.6917
12
ViT-B-32
commonpool.mmciipp -2kkbmbbkk
151.28
14.78
0,2725
13
ViT-B-32-255
Battacomp_s34b__0e _b66k
151.29
17.46
0.7281
14
VIT-B-32
1T-B-32-quickgelu
laion400m.e31
151.28
14.78
0.6294
15
ViT-B-32-quice quqickuuu
metaclip_fullee
151.28
14.78
0,6766
16
ViT-B-16
openai
149.62
41.09
0.6834
17
ViT-B-16
laion2b.s34b.bbe _38kk
149.62
41.09
0.7023
18
ViT-B-16
datacomp_Ls1b..b
149.62
41.09
0.6310
19
ViT-B-16
commonpool I laion slb b8k
149.62
41.09
0,5526
20
ViT-B-16
dfn2b
149.62
41.09
0.7624
21
VIT-B- Ii-B--lo-quickeluu
metaclip_fie fullcc
149.62
41.09
0.7212
22
ViT-B-16 11-B--6
laion400m e31
208.38
64.03
0.6904
23
ViT-L-144
openai laion400m e31
427.62
175.33
0.7554
24
ViT-L-14
427.62
175.33
0.7271
25
VIT-L-14
datacomp _1__xsss111 s1333. b90k
427.62
175.33
0.7921
26
ViT-L-14
commonpool xl clip s13b b90k
427.62
175.33
0.7637
27
I4T-L-144-uicceeluu
metaclip fullce dfn2b
427.62
175.33
0.7917
28
ViT-L--44-quickeeluu
427.62
175.33
0.8141
29
ViT-L-14-336
openai
427.94
395.22
0.7656
30
ViT-H-14
laion2b s32b b79k
986.1 11
381.68
0.7796
31
ViT-H--14-quicgelluu
metaclip_fullce dfnSb
986.1
381.68
0.8051
32
VTT-H-14-38--quikkceuu
986.71
1054.05
0.8437
33
ViT-g-14
laion2b_s12b b42k
1366.68
581.15
0.7663
34
ViT-bigG-14
laion2b_s39b_b 60k
2539.57
1065.36
0.8009
35
roberta- ViT-B-32
laion2b.s12bbb324
212.72
105.87
0.6171
36
xlm-roberta-base--ie ViT-B-32
laion5b_s13b_990
366. 12
105.87
0.6236
37
convnext_base
laion2b.s13b.b8//
179.39
49.38
0.7078
38
convnext.base..u w_320
laion.aesthetiicss13.b.b
179.39
71.94
0.7167
39
Lonveext_large...
laion2b_s26..11024
351.77
107.5
0,7591
40
convnext large d 3.320
laion2b s29b b131k
351.77
157.98
0.7660
41
.xlaae
1200.58
443.03
0.7947
42
coca-ViT-B-32
laion2b-s13b.b900
253.56
33.34
0.6331
43
coca ViT-L-14
laion2b s13b b90k
638.45
214.52
0.7561
44
EVA01-g-14
laion400m b41k
1136.44
547.36
0,7852
45
EVA02-B-16
merged2b.ssee btt...8bb111..11
149.69
41.09
0.7472
46
EVA02-L-14-325
merged2b s6b b61k
428.08
395.16
0.8039
47
FVA02-F--14
laion2b s4b bl 15k
4704.59
2311.42
0.8196
48
nllb-clip-base
vI
501.89
369.6
0.2432
49
nllb-clip- base--iglipp
507.47
472.91
0.3909
13


Reviews:
[{"review_id": "NUGz3GOvRF", "reviewer": "Reviewer_zHkU", "summary": "This paper introduces Model Label Learning (MLL) for selecting and reusing pre-trained VLMs for downstream tasks. This method aims to address the challenge of choosing the best VLM from a growing hub, given their diverse performance across tasks and primarily it is impractical to evaluate them exhaustively. The core idea of MLL is to assign \"model labels\" to each VLM, describing its capabilities based on pre-testing on a semantic graph of visual concepts. MLL then utilizes these labels to efficiently select and reuse VLMs for new tasks by semantically matching task requirements to model labels. The MLL paradigm consists of three modules: model labeling (pre-testing VLMs on a semantic graph and creating labels), model selection (matching task descriptions to model labels for VLM selection), and model reuse (ensembling selected VLMs). In addition, the authors introduce a new benchmark for VLM selection, comprising 49 VLMs and 17 downstream datasets. Experimental results on this benchmark show the effectiveness of MLL compared to baselines like ImageNet Baseline (INB) and ModelGPT, demonstrating improvement in zero-shot downstream tasks.\n\n## update after rebuttal\n\nMy questions have been addressed by the rebuttal and I have thus increased my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "### C1: MLL is effective for selecting and reusing VLMs for downstream tasks, leading to improved performance.\n- E1: Tables 1 and 2 show improved average accuracy and F1-score compared to INB and ModelGPT across 17 downstream tasks for both single model (k=1) and ensemble (k=3) settings. This provides initial evidence. Then, Table 3, 4 demonstrate the robustness to hyper-parameters.\n- P1: I think this claim might be strengthened by including statistical significance test. It would help determine whether the average improvement across 17 datasets (1.86% over INB, 2.53% over ModelGPT) represents a meaningful pattern.\n\n### C2: MLL is computationally efficient in the model selection process.\n- E2: The paper states that model labeling is target task independent and pre-computed, making the selection process efficient. This is a design feature suggesting methodology efficiency.\n- P2: However, empirical evidence of the actual computational time for model selection and comparison to baselines is not provided  (although Table 4 hints at interface time costs). The cost of labeling stage itself is not quantified.\n\n\n### C3: MLL is scalable as the model hub grows.\n- E3: Figure 3 shows that average performance generally increases as the number of VLMs in the hub grows, suggesting scalability.\n- P3: The rate of performance improvement with hub size and the saturation point are not fully explored. The computational cost of maintaining and updating model labels as the hub grows is not discussed. Moreover, the experiment in Figure 3 is based on random expansion, which might not fully reflect real-world hub growth scenarios, such as chronological additions based on when models are published, preferential inclusion of higher-performing models first, or somehow strategic additions based on identified performance gaps. I personally think that the random expansion approach does not capture these realistic growth patterns, this likely matters because the order in which models are added could significantly affect the performance improvement curve. For instance, if high-performing models happen to be added early in many random schemes, this could show rapid initial improvements that might not be achievable in practice.\n\nIn brief, the claims made are initially supported by the presented data, but it is better to be strengthened with more empirical evidence, statistical validation, and more detailed analyses of costs and scalability.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally sensible and appropriate for the problem of VLM selection and reuse.\n\nThe MLL method with its three modules is a logical approach to address the VLM selection problem. Semantic labeling provides a structuredway to understand VLM capabilities without task-specific evaluation. Semantic matching for selection is an efficient alternative to exhaustive search. Ensemble reuse leverages the strengths of multiple selected models.\n\nThe introduced benchmark of 49 VLMs and 17 downstream datasets is a strong evaluation criterion, covering a range of domains and tasks relevant to VLMs. The authors propose to use accuracy and F1-score, which are standard metrics for the evaluated tasks. Comparison against INB and ModelGPT provides relevant baselines.", "theoretical_claims": "There are no explicit theoretical claims made in the submission excerpt that require proof checking. The paper is primarily methodologically and empirically driven. Claims of efficiency and scalability are based on the design of the method and empirical observations, not formal theoretical derivations.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound in principle but require more detailed reporting and validation.\n\n#### Soundness\nUsing INB and ModelGPT as baselines is appropriate and allows for comparison to a simple baseline and a relevant existing VLM selection method. The use of a diverse benchmark is a strength. The ablation studies on $\\alpha$ (table 3) and model count $k$ (table 4) demonstrate an effort to understand the impact of key hyperparameters and design choices.\n\n#### Issues and areas for improvements:\n1. statistical significance: as aforementioned, this submission lacks mention of significance test to validate the observed performance differences.\n2. more detailed result: while average performance is reported, the readers might also wonder: are the improvements consistent across all datasets? Also, are there specific datasets or task types where MLL excels or underperforms? \n3. computational cost: the analysis of the cost of model labeling should also be reported.", "relation_to_broader_scientific_literature": "This paper appears to be relating to three research areas:\n\nVLM: The paper directly relates to a challenge in VLMs - how to efficiently and effectively utilize the increasing number of pre-trained models.\n\nModel Selection: The paper also relates to literature on model selection, particularly for pre-trained models. It acknowledges and contrasts with existing methods like NCE, LEEP, LogME, and ModelGPT. \n\nLearnware/Model Hubs: The paper explicitly connects to the \"learnware\" paradigm and the concept of model hubs. It builds upon the idea of model specification and efficient model selection in heterogeneous model repositories. MLL can be viewed as a specific instantiation of learnware principles tailored for VLMs.", "essential_references_not_discussed": "Primarily I don't think there are significant missing of essential literature; however, to strengthen the literature context and ensure comprehensiveness, the authors might consider discussing and citing works in the following more specific areas:\n\n- semantic similarity measures: as MLL relies on semantic similarity measure for comparing captions, discussing relevant papers on semantic textual similarity and different metrics would be beneficial.\n- ensemble methods for VLMs: If the ensemble reuse module could be extended to more ensembling techniques, making a discussion on relevant works on ensemble methods, would also be important.", "other_strengths_and_weaknesses": "I summarize the strengths and weaknesses I identified below. They may overlap with the previously discussed points:\n\n### strengths\n\n1. The paper addresses a practical problem that will become increasingly important as more pre-trained VLMs become available.\n2. The MLL method is conceptually simple, with clearly separated components for labeling, selection, and reuse.\n3. The approach is computationally efficient for users, as the potentially expensive model labeling process occurs only once when models are added to the hub\n4. The benchmark will be valuable for future research in VLM selection\n\n### Weaknesses:\n1. The approach depends heavily on the the quality of the constructed semantic graph, but the graph construction process isn't explored in depth.\n2. The computational overhead of the model labeling process itself isn't thoroughly discussed, which could be significant for large models\n3. More empirical results (e.g., statistical significance tests) should be reported as well.\n4. Analysis of failure cases where MLL does not perform very well would also be meaningful.", "other_comments_or_suggestions": "I have no more comments except for the existing sections.", "questions_for_authors": "1. how sensitive is the approach to the choice of nodes in the semantic graph, and would a different semantic graph graph construction lead to significantly different results?\n2. What is the computational overhead of the model labeling process, particularly for large models?\n3. How would the framework handle non-classification tasks?\n4. The entropy-based ensemble weighting scheme is relevant to model overconfidence, but have you explored other weighting strategies?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pjTShGoGdU", "reviewer": "Reviewer_Hxkz", "summary": "The paper explores a practical VLM reuse problem and proposes Model Label Learning (MLL), an efficient approach for selecting and reusing pre-trained Vision-Language Models (VLMs) for downstream tasks. The framework comprises three modules: (1) Model Labeling, which assigns labels to VLMs based on their capabilities; (2) Model Selection, which matches these labels to task requirements; and (3) Model Reuse, which employs an ensemble of selected models. Additionally, a large-scale benchmark, including 49 VLMs and 17 datasets, is introduced to evaluate MLL’s effectiveness. Experimental results demonstrate its promising scalability and performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The paper claims that the performance of VLMs can vary significantly depending on the dataset domain. To support this claim, the authors conduct extensive experiments, as shown in Figure 1. Specifically, Figure 1(a) illustrates that VLMs exhibit different strengths across various visual tasks, with no single model consistently outperforming all others across every task. Figure 1(b) demonstrates that even within the same task, different VLMs achieve varying levels of performance across specific classes.", "methods_and_evaluation_criteria": "To achieve effective reuse of VLMs, the paper proposes Model Label Learning (MLL), an efficient approach for selecting and reusing pre-trained Vision-Language Models (VLMs) for downstream tasks. Specifically, the framework consists of three modules: (1) Model Labeling, which assigns labels to VLMs based on their capabilities; (2) Model Selection, which matches these labels to task requirements; and (3) Model Reuse, which employs an ensemble of selected models. The proposed framework is reasonable for addressing this problem.\n\nAdditionally, a large-scale benchmark, including 49 VLMs and 17 datasets, is introduced to evaluate the effectiveness of MLL.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The paper presents comprehensive experiments involving 49 VLMs and 17 target task datasets, along with sufficient analysis. The experimental results demonstrate the effectiveness of the proposed method in selecting and reusing VLMs.\n\nIn addition to the experiments mentioned above, the authors should provide more experimental results to further validate the proposed method. For example, for each target dataset, the highest performance achieved by any model in the model hub should be included as a reference. This would help assess the effectiveness of the proposed method in selecting models.", "relation_to_broader_scientific_literature": "Unlike other VLM studies, this paper explores a practical VLM reuse problem, aiming to effectively reuse existing models to improve performance on target datasets.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The main strength of this work is that the problem it explores is both practical and meaningful. Additionally, it demonstrates good scalability, as the use of a semantic graph enables MLL to expand as new models or tasks are added, making it adaptable to diverse visual tasks. The main weakness is that more experimental results should be provided. Please refer to Experimental Designs Or Analyses for details.", "other_comments_or_suggestions": "In addition to VLMs for image classification tasks, more VLM studies have explored dense-level recognition tasks, such as detection and segmentation. The authors could consider incorporating more types of VLMs into the proposed framework, enabling it to handle a wider range of tasks.", "questions_for_authors": "Please refer to above sections for details.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IfH7HWAobV", "reviewer": "Reviewer_Vdth", "summary": "This paper presents a framework for organizing models that facilitates the storage, labeling, and reuse of vision-language models (VLMs). This system enhances overall performance compared to using a single VLM. A model labeling process is introduced to precisely describe the functionality of each VLM, enabling subsequent identification and reuse. The authors propose a comprehensive benchmark involving 49 VLMs and 17 target tasks to evaluate and demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "The authors claim that improved performance can be achieved with the proposed model organization framework because different Vision-Language Models (VLMs) have their own advantages in handling various models, and the proposed method can fully utilize these advantages. The experiments conducted in this paper have thoroughly demonstrated the authors' claim. They not only show that the proposed method achieves better overall performance on 17 tasks but also indicate that as the number of models increases, the overall performance can further improve.", "methods_and_evaluation_criteria": "The proposed method seems to be work. To evaluate the performance of the method, the authors introduce a semantic graph with labels and corresponding images to pre-test the performance of each model. Subsequently, when reusing each model, the authors propose a model identification strategy to locate each model in an embedding space and ensemble models for subsequent tasks. The overall framework has potential to demonstrate the effectiveness of a collection of models. The benchmark and evaluation in this paper are valid and can reflect the performance of each method.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Both the experimental design and analyses are comprehensive. The experiments are conducted on multiple datasets with up-to-date comparison methods. Moreover, the authors give scalability and ablation studies to further demonstrate the effectiveness of proposed methods.", "relation_to_broader_scientific_literature": "The method presented in this paper is significant, as it can be utilized to manage existing Vision-Language Models (VLMs) to address new downstream tasks with improved performance, rather than constructing new VLMs, which is costly. These techniques can assist platforms like HuggingFace in better organizing models.", "essential_references_not_discussed": "No necessary reference is omitted.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe problem addressed in this paper is significant and intriguing. Managing large Vision-Language Models (VLMs) in the model hub is crucial because training new models for new downstream tasks is costly, whereas existing models can be utilized to handle these tasks.\n2.\tThe proposed method is logical, as the pipeline of pre-testing, model identification, and model reuse appears effective. With a more comprehensive semantic graph and an increased number of models in the model hub, this method can achieve better performance.\n3.\tThe experiments in this paper are thorough. The authors have developed a benchmark with 49 VLMs and 17 tasks, which thoroughly evaluates the performance of each method. The proposed method achieves the best overall performance, demonstrating the effectiveness of the proposed framework.\n\nWeakness: \n\nThe primary limitation of this paper is that the model hub is not sufficiently large to fully enhance the effectiveness of the proposed method. Nevertheless, I believe the current benchmark is adequate to assess the effectiveness of each method and demonstrate the proposal's effectiveness. In this context, I do not perceive any significant weaknesses in the paper.", "other_comments_or_suggestions": "Please see the strengths and weaknesses above.", "questions_for_authors": "- Have you made the computational cost analysis for the proposed method? Is the proposed method more efficient or slower than the previous works?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QGuj6aaMTY", "reviewer": "Reviewer_9kf2", "summary": "The paper deals with Model Label Learning (MLL) to select and reuse pre-trained VLMs for specific downstream tasks. It addresses the challenge of choosing the best VLMs from numerous options, each with varying performance on different tasks and different classes. The MLL approach includes model labeling, selection, and reuse, utilizing a predefined concept pool to bridge the understanding of those concepts from candidate VLMs and target tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "1. The contribution of this work appears to be weak. The so-called \"new benchmark\" is not a separate contribution, as it is inherently tied to the task or paradigm presented. Therefore, they should be considered a single contribution rather than two distinct ones.\n2. The validation and analysis provided in the paper are weak (see Experimental Designs Or Analyses). Additionally, the improvements demonstrated by the proposed model are limited: In the best set of experiments (0.6639, k=3), the author proposed method outperformed the comparison method in only half of the downstream tasks. This raises concerns about the effectiveness of the proposed model selection method.", "methods_and_evaluation_criteria": "The process of constructing the reference dataset, profiling the target dataset, and conducting multi-model inference, replying on a combination of language generation models, embedding models, and similarity computation, is complex and time-consuming. As can be seen in Tab.2, a suboptimal choice of k can lead to a decline in results. \nPlus the details of this experiment are missing, it is important to question how robust the selection of k=3 is across 17 different downstream tasks. If the value of k is uncertain unless given a validation split, it becomes challenging to demonstrate how effective the procedures could be in practical applications.", "theoretical_claims": "The proposed method is straightforward and, for the most part, clearly explained.\nTwo minor suggestions: \n1. The logic from eq6 to eq7 is hard to follow.\n2. \\mathcal{T}_m(D) should be \\mathcal{T}_m(d) in eq2.", "experimental_designs_or_analyses": "The submission has not been well validated, and analysis is weak and not insightful enough. Besides above mentioned weakness, \n1. important baselines are missing. For example, since ensemble typically enhances performance and random sampling can ensure diversity in recognizing different classes, it is crucial to conduct random ensemble with majority voting / prob. distribution average / etc as ablation study, other than the parameter analysis presented in Section 5.3.\n2. how do the randomly selection reference images affect the robustness of the proposed method?\n3. how do specific downstream tasks correlate with different model candidates? For example, why does the method show significant improvement on datasets DMLab, RESISC45 and FER2013, while performing at a comparable level on other datasets?", "relation_to_broader_scientific_literature": "The proposed method, while simple and straightforward, lacks convincing validation and practical applicability. The method's effectiveness is limited even after model ensemble. It also raises important questions about the robustness and practicality of its approach, especially considering the complexity and uncertainty involved in key processes like target benchmark profiling, model selection and dataset correspondence. These factors suggest that while the paper addresses a relevant and meaningful task, its contributions may not significantly advance the field without further validation and refinement.", "essential_references_not_discussed": "Given that the authors present the proposed paradigm as one of their main contribution, it is essential to thoroughly evaluate its effectiveness compared to other training-free methods for VLMs, such as but not limited to test-time adaptation.", "other_strengths_and_weaknesses": "While the importance of the task is acknowledged, the presented method has not been convincingly validated and appears to be impractical.", "other_comments_or_suggestions": "Please see comments above.", "questions_for_authors": "How is X_v determined for each v that is randomly collected?  Is there a one-to-one correspondence for each v, as defined by WordNet, across all 5 datasets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors explore how to select and reuse pre-trained VLMs for a specific downstream task, which is an important problem but has been rarely studied. To address this problem, this paper proposes a novel paradigm called Model Label Learning (MLL) that assigns each VLM a label to describe its utility on representative visual concepts, containing three key modules: model labeling,
model selection, and model reuse. The effectiveness of the proposed method is well validated by extensive experiments.

The importance of this problem is recognized by all reviewers at the beginning. However, some technical concerns still existed. After the rebuttal, all technical concerns are addressed by the authors. One remaining issue is a concern regarding the time complexity of the newly proposed framework. After checking this by the area chair and discussing it with other reviewers, this concern is minor and can be concluded as a potential limitation, instead of a reason to reject this novel paper.
Decision: Accept (poster)