Pre-Trained Vision-Language Model Selection and Reuse for Downstream Tasks

Authors: Hao-Zhe Tan, Zhi Zhou, Lan-Zhe Guo, Yu-Feng Li
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Vision-Langage Model; Model Selection; Model Reuse

Abstract:
Pre-trained Vision-Language Models (VLMs) are becoming increasingly popular across various visual tasks, and several open-sourced VLM variants have been released. However, selecting the best-performing pre-trained VLM for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data limitations. To address this problem, this paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL). The proposal contains three key modules: \emph{model labeling}, which assigns labels to each VLM to describe their specialty and utility; \emph{model selection}, which matches the requirements of the target task with model labels; and \emph{model reuse}, which applies selected VLMs to the target task in an ensemble manner. The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent and the ability could grow with the number of candidate VLMs. We also introduce a new benchmark for evaluating VLM selection methods, including 49 VLMs and 17 target task datasets. Experimental results clearly demonstrate the effectiveness of the proposed method for selecting and reusing VLMs.

Decision: reject
Reviewer scores: 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730271825662

thread_type:
official_review

<thread_object>
creator: reviewer_stka

<content>
summary:
The paper explores a practical VLM reuse problem and proposes Model Label Learning (MLL), which efficiently selects and reuses pre-trained Vision-Language Models (VLMs) for downstream tasks. The framework consists of three modules: model labeling, which assigns labels to VLMs based on their capabilities; model selection, which matches these labels to task requirements; and model reuse, which employs an ensemble of selected models. In addition, a large-scale benchmark, including 49 VLMs and 17 datasets, is introduced to evaluate MLL’s effectiveness, with experimental results showing promising scalability and effectiveness.

strengths:
1. The problem explored in this work is practical and meaningful. The proposed MLL framework provides an efficient way to select and reuse VLMs by leveraging a semantic graph and task-specific labels.

2. The method demonstrates good scalability. The use of a semantic graph allows MLL to expand as new models or tasks are added, making it adaptable to diverse visual tasks.

3. The paper is well-organized and easy to follow.

weaknesses:
1. Regarding the scalability of the constructed semantic graph, if new nodes are added to the graph, is it necessary to add images to the sampled dataset to represent these new nodes? Additionally, have the authors considered using different datasets as the sampled dataset? If so, would different datasets impact the final performance?

2. For each target dataset, the highest performance achieved by any model in the model hub should also be included as a baseline result. This would help evaluate the effectiveness of the proposed method in selecting models.

3. As K is a core hyperparameter, more experiments analyzing its impact should be included, as the paper currently only presents results for K=1 and K=3. A more comprehensive analysis of K, including performance and computational cost at various values, is suggested.

questions:
In addition to the points listed in weakness, the VLMs in the current model hub are primarily designed for image classification tasks. Have the authors considered expanding the proposed pipeline to accommodate more complex tasks, such as segmentation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730464058961

thread_type:
official_review

<thread_object>
creator: reviewer_eqsx

<content>
summary:
The paper introduces Model Label Learning (MLL), a new approach for selecting and repurposing Vision-Language Models (VLMs) for downstream tasks. It comprises three main components: model labeling to categorize VLMs by their expertise, model selection to align VLMs with task requirements, and model reuse to integrate chosen VLMs into an ensemble for task application.

strengths:
**[New perspective]** This work focuses on the selection and reuse of pre-trained VLMs to better suit the need of specific downstream tasks, which is novel and practical.

**[Good presentation]** This paper is well-written, making it easy to follow.

**[Thorough evaluation]** Extensive experiments have been done to evaluate the effectiveness of the proposed strategy.

weaknesses:
**[Need more explanation]** 
- In Figure 1, the details of the evaluated VLMs are missed. Please add this information in the caption for better understanding.
- The paper missed the introduction of ImageNet Baseline (INB). Is the best-performing model on ImageNet, i.e., EVA02-E-14?

**[Could be improved]** 
- In line 245, this work randomly selects images $X_v$ from sample datasets to serve as representations for each node. Is there a more elegant solution for this, e.g., using the mean of several samples from the same class?
- For model reuse, the work selects top-k models with a simple ensemble approach. It would be nice to discuss or compare more advanced ensemble approaches in VLMs, e.g., “Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models, ICML 2024”.

**[Experiments]** 
- In Table 1, both INB and ModelGPT use the best-performing single model alone for evaluation. It would be nice to leverage them to select more models with ensemble for prediction when comparing the proposed method with 3-model ensemble. For example, the authors can select top-3 models on ImageNet for INB and do similar things for ModelGPT. Including this comparison can enhance the understanding of the effectiveness of the proposed method.
- Since the proposed MLL introduces three procedures, each costing extra time, could the authors provide the additional time introduced? This could offer insights on the trade-off between performance and time.

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653634903

thread_type:
official_review

<thread_object>
creator: reviewer_nhl5

<content>
summary:
Selecting the best-performing pre-trained Vision-Language Models (VLMs) for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data limitations. 
To address this problem, this paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL). 
The proposal contains three key modules: model labeling, which assigns labels to each VLM to describe their specialty and utility; model selection, which matches the requirements of the target task with model labels; and model reuse, which applies selected VLMs to the target task in an ensemble manner. 
The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent and the ability could grow with the number of candidate VLMs.

strengths:
- The proposed method is easy to understand.

weaknesses:
- The novelty of the proposed method is weak. The main contribution of this paper is the model selection when ensembling multiple VLMs. However, there is no discussion or experimental analysis of the selected models during this process. What models are selected will give the readers a hint about the proposed method's characteristics or advantages.
- The analysis in this paper is too simple. After the model selection, what models are selected? As the main contribution is the model selection, the authors should show the selected models to understand the proposed method's characteristics and advantages.
- As a design choice analysis, the authors only tried K values to be 1 and 3. Although finding the best hyper-parameter is essential, why didn’t the authors try other values for K? The number of selecting models K is more important than the size of the model hub.
- Other essential design choice analyses are also missing. For example, in Eqn 8, why did the authors give high loss weight to models with high entropy? Is it the best choice of the weight values? Also, in Eqn 7, how is the hyper-parameter alpha decided, and how does it affect the model's performance?
- More importantly, the comparison with recent models is missing. There are several ways to improve VLMs without training, at least with the improved prompt-based approaches [1,2]. The authors should show the advantages of ensembling the models instead of the existing ways of improving VLMs. Also, ensembling models increases the number of total parameters. The authors should analyze the efficiency of the model ensemble compared to the existing approaches.

[1] Visual Classification via Description from Large Language Models. ICLR 2023.

[2] What does a platypus look like? Generating customized prompts for zero-shot image classification. ICCV 2023.

questions:
Please refer to the questions in the weakness.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an MLL framework that efficiently selects and reuses VLMs by leveraging a semantic graph, providing scalability for diverse visual tasks. The motivation is good, and the method demonstrates good scalability, making it adaptable to new models and tasks. The paper is well-organized and easy to follow. However, the novelty of the approach is limited, with insufficient analysis on model selection and hyperparameter choices. There is a lack of comparison to recent models and an inadequate discussion on the performance and efficiency of the proposed method.

Despite the reviewers acknowledging the contribution of the paper, the novelty remains a significant concern. After carefully reviewing the paper and the rebuttal, the AC, considering the emphasis on originality of ICLR, decided to reject the paper.