Title: Foundation Model is Efficient Multimodal Multitask Model Selector
Venue: NeurIPS 2023
Accepted: yes
Keywords: transfer learning

model selection

foundation model
Abstract: This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering.A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metrics to measure models’ transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable in a multi-modal multi-task scenario. To tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models to transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into a unified noisy label embedding. EMMS can estimate a model’s transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability of pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the state- of-the-art method LogME enhanced by our label embeddings, EMMS achieves 9.0%, 26.3%, 20.1%, 54.8%, 12.2% performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5.13×, 6.29×, 3.59×, 6.19×, and 5.66× speedup in wall-clock time, respectively. The code is available at https://github.com/OpenGVLab/Multitask-Model-Selector.

Foundation Model is Efficient Multimodal Multitask Model Selector

Abstract

This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering.A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metrics to measure models' transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable in a multi-modal multi-task scenario. To tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models to transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into a unified noisy label embedding. EMMS can estimate a model's transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability of pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the stateof-the-art method LogME enhanced by our label embeddings, EMMS achieves 9.0%, 26.3%, 20.1%, 54.8%, 12.2% performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5.13 × , 6.29 × , 3.59 × , 6.19 × , and 5.66 × speedup in wall-clock time, respectively. The code is available at https://github.com/OpenGVLab/MultitaskModel-Selector.

…

Prior

Multi-task

man and girl and

horses are near

contained fire

…

beagle

classification

<3,20,408,307>

regression

man and  ...

contained fire

caption

referring

adult

sheep

+

one-hot

real-value

text

bounding box

Model Feature

Pre-trained

Model

multi-task scenarios

…

(a)

…

Image

Classification

0.2

0.4

0.0

0.6

0.8

1.0

LogME

TransRate

NLEEP

PACTran

Image

Captioning

Visual Question

Answering

Text Question

Answering

Referring

Expression

Comprehension

Weighted Kendall's

!

!

(b)

0.64

0.61

0.56

0.67

0.56

0.77

0.48

0.53

0.47

0.51

0.40

1 Introduction

Pre-trained models (such as neural network backbones) are crucial and are capable of being fine-tuned to solve many downstream tasks such as image classification [1], image captioning [2], question answering [3], and referring segmentation [4]. This 'pre-training → fine-tuning' paradigm shows that the models pre-trained on various datasets ( e.g., ImageNet [1]and YFCC100M [5]) by many objectives ( e.g., supervised and self-supervised) can provide generic-purpose representation, which is transferable to different tasks. A large number of pre-trained models have been produced with the rapid development of network architecture research, such as convolutional neural networks (CNNs) [6, 7, 8] and transformers [9, 10, 11]. When given a large collection of pre-trained models to solve multiple multi-modal tasks, an open question arises: how to efficiently predict these models' performance on multiple tasks without fine-tuning them?

Existing works [12, 13, 14, 15, 16, 17, 18, 19] answered the above question using model selection approaches, which are of great benefit to transfer learning. For example, when a neural network is properly initialized with a better pre-trained checkpoint, it will achieve faster convergence and better performance on a target task [20, 21]. However, it is challenging to quickly identify an optimal one from a large collection of pre-trained models when solving each multi-modal task. This is because of two reasons. Firstly, the ground truth of model ranking can only be obtained by brute-force fine-tuning and hyper-parameter grid search, which are computationally expensive [22]. Secondly, the recent methods [12, 13, 14, 23] that can estimate the transferability of pre-trained models are not generic enough for a variety of multi-modal tasks. For instance, an approach [14, 12] that relies on the prior knowledge of a single specific task would be ineffective in others.

To address the above challenges, we need a unified representation to represent diverse label formats in each multi-modal task e.g., categories, texts and bounding boxes. Existing methods cannot be employed in multi-task scenarios because they only receive labels with one-hot or real-valued vectors, as shown in Fig.1 . For example, LEEP [12] and PACTran [24] are carefully designed for classification tasks. GBC [25] and TransRate [23] measure transferability using class separability, which relies on prior knowledge in classification task. Although LogME [13] can be used in both classification and regression tasks, it relies on real-valued labels, making it inapplicable in other label formats such as text descriptions.

In contrast to the above works, we propose an Efficient Multi-task Model Selector with an acronym, EMMS, which can select the most appropriate pre-trained model for solving each multi-modal task.

This is achieved by employing foundation models, such as CLIP [26] and GPT-2 [27], to transform diverse label formats into a unified label embedding space. The estimated label embedding contains more rich information than the conventional one-hot and real-valued label encoding.

In this way, EMMS can measure the compatibility between the models' features and corresponding label embeddings on various tasks, as shown in Fig. 1 and Fig. 2 . This results in a more generic assessment of the models' transferability than previous methods. Specifically, EMMS treats the estimated label embeddings as noisy oracles of the ground-truth labels, and it turns a log-likelihood maximization problem into a simple weighted linear square regression (WLSR). We propose an alternating minimization algorithm to solve WLSR, which can be solved with a theoretical convergence guarantee efficiently. Extensive experiments validate the effectiveness of EMMS on multiple tasks, including image classification [1], image captioning [2], question answering on both image [28] and text [29], referring comprehension [4], and landmark detection [30].

The contributions of this work are summarized as follows. (1) We propose a generic transferability estimation technique, namely Efficient Multi-task Model Selector (EMMS). Equipped with a unified label embedding provided by foundation models and a simple weighted linear square regression (WLSR), EMMS can be fast, effective, and generic enough to assess the transferability of pre-trained models in various tasks. (2) We propose a novel alternating minimization algorithm to solve WLSR efficiently with theoretical analysis. (3) Extensive experiments on 5 downstream tasks with 24 datasets demonstrate the effectiveness of EMMS. Specifically, EMMS achieves 9.0%, 26.3%, 20.1%, 54.8%, 12.2%, performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5.13 × , 6.29 × , 3.59 × , 6.19 × , and 5.66 × speedup in wall-clock time compared with the state-of-the-art method LogME enhanced by our label embeddings, respectively.

2 Related Work

Transferability Estimation. Model selection is an important task in transfer learning. To perform model selection efficiently, methods based on designing transferability metrics have been extensively investigated. LEEP [12] pioneers to evaluate the transferability of source models by empirically estimating the joint distribution of pseudo-source labels and the target labels. But it can only handle classification tasks with supervised pre-trained models because the modeling of LEEP relies on the classifier of source models. Recent works propose several improvements over LEEP to overcome the limitation. For example, NLEEP [31] replaces pseudo-source labels with clustering indexes. Moreover, LogME [13], TransRate [23], and PACTran [24] directly measure the compatibility between model features and task labels. Although fast, these metrics can only be used on limited tasks such as classification and regression. This work deals with model selection in multi-task scenarios. We propose EMMS to evaluate the transferability of pre-trained models on various tasks.

Label Embedding. Label embedding represents a feature vector of task labels, which can be generated in various ways. The classical approach is to use one-hot encoding to represent the labels as sparse vectors, which is widely used in image classification. Another way is to transform labels into vectors by embedding layers. For example, an RNN module is employed to generate label representation in [32], which is encouraged to be compatible with input data vectors in text classification tasks. In addition, it is also common to treat the labels as words and use techniques such as word2vec [33] or GloVe [34] to learn vector representations of the labels. The main obstacle in the multi-task scenario is how to deal with diverse label formats. In this work, we follow the idea of word embedding and treat task labels as texts, which are then transformed into embeddings by publicly available foundation models [26, 27].

Foundation Models. CLIP [26] is the first known foundation model which learns good semantic matching between image and text. The text encoder of CLIP can perform zero-shot label prediction because it encodes rich text concepts of various image objects. By tokenizing multi-modal inputs into homogeneous tokens, recent work on foundation models such as OFA [35] and Uni-Perceiver [36] use a single encoder to learn multi-modal representations. In this work, we utilize the great capacity of foundation models in representing image-text concepts to generate label embedding. It is noteworthy that although foundation models can achieve good performance in various downstream tasks, they may not achieve good zero-shot performance on many tasks[37, 38, 39] and it is still computationally expensive to transfer a large model to the target task [40, 41]. On the contrary, a multi-task model

selector can quickly select an optimal moderate-size pre-trained model that can generalize well in target tasks. In this sense, a multi-task model selector is complementary to foundation models.

3 Preliminary of Model Selection

Problem Setup. A target dataset with N labeled samples denoted as T = { ( x n , y n ) } N n =1 and M pre-trained models { ϕ m = ( θ m , h m ) } M m =1 are given. Each model ϕ m consists of a feature extractor θ m producing a D -dimension feature (i.e. ˆ x = θ m ( x ) ∈ R D ) and a task head h m outputting predicted label given input x [6, 9]. In multi-task scenarios, the ground-truth label comes in various forms, such as category, caption, and bounding box, as shown in 1 . The task of pre-trained model selection is to generate a score for each pre-trained model thereby the best model can be identified to achieve good performance for various downstream tasks.

Ground Truth. The ground truth is obtained by fine-tuning all pre-trained models with hyperparameters sweep on the target training dataset and recording the highest scores of evaluation metrics [31, 13] (e.g. test accuracy and BLEU4 [42]) . We denote ine-tuning scores of different models as { G m } M m =1 . Since fine-tuning all models on all target tasks requires massive computation cost, research approaches design lightweight transferability metrics which offer an accurate estimate of how well a pre-trained model will transfer to the target tasks.

Transferability Metric. For each pre-trained model ϕ m , a transferability metric outputs a scalar score T m based on the log-likelihood, as written by

where ( x n , y n ) denotes the n -th data point in target dataset T . A higher log-likelihood value for T m indicates that the model ϕ m is likely to achieve better performance on the intended task. Numerous transferability metrics have been proposed by modeling prediction probability p ( y n | x n ; θ m , h m ) in various ways. Although being efficient, they can hardly be used in multi-task scenarios.

Challenges in Multi-task Scenarios. Existing transferability metrics fail to generalize to various tasks for two reasons. Firstly, existing methods such as LEEP and LogME can only deal with realvalue label formats. But y n can be a sentence of words in the task of image caption. Secondly, a large number of the previous metrics estimate transferability through the target task's prior information such as maximizing inter-class separability, which is inapplicable in multi-task scenarios except for the classification. To overcome these difficulties, we introduce a simple regression framework with unified label embeddings provided by several foundation models in Sec.4.

4 Our Method

In this section, we introduce our Efficient Multi-task Model Selector (EMMS). To overcome the difficulty of diverse label formats, EMMS employs foundation models to transform various labels into unified label embeddings in Sec.4.1. By treating label embeddings provided by multiple foundation models as noisy oracles of ground truth labels, EMMS can calculate transferability metric under a simple weighted linear square regression (WLSR) framework in Sec.4.2. We design an alternating minimization algorithm to solve WLSR efficiently in Sec. 4.3. The illustration of our EMMS is provided in Fig. 2 .

4.1 Foundation Models Unify Label Embedding

In general, label embeddings or label representations should encode the semantic information such that two labels with low semantic similarity have a low chance to be grouped. A common scheme is to represent label embedding as a one-hot vector. However, one-hot representation can not embed labels with text formats such as captions in the image caption task. Following the design in multi-modality foundation models [42], we treat labels with diverse formats as a text sequence, which can be encoded by pre-trained foundation models, as shown in Fig. 2 .

Label Embedding via Foundation Models (F-Label). Thanks to the great representational capacity, the foundation model can construct label embedding (termed F-label) while preserving its rich

!

"

!

"

"

"

#

#

!

#

#

#

"

$ %

&

'

!

#

'

#

#

'

"

#

"

$

beagle

classification

man and girl and horses

are near contained fire

caption

A woman in a white

blouse holding a glass

of wine

<231><122>

<424><463>

referring

What is the woman

feeding the giraffe?

Carrot

visual question answering

(a) Examples of text labels in various tasks

(b) Graph model of regression with multiple noisy labels

#

$

'

%

#

'

$

#

Foundation

Model

Label

Embedding

Noise

Strength

#

y

people

Boy wearing red

shirt is running

through woodland

Child runs near

some trees

baby

boy

girl

man

woman

1.0

0.8

0.6

0.4

0.2

0.0

1.0

0.8

0.6

0.4

0.2

0.0

label embedding

one-hot

label embedding

one-hot

fruits reptiles

flowers

(a) Example of label embedding in image classification

(b) Example of label embedding in image caption

semantic information of labels. Given a label y in the target task, the label embedding z ∈ R L is obtained by z = F ( y ) / ∥ F ( y ) ∥ 2 where F can be instantiated by various foundation models to process diverse label formats. ℓ 2 normalization is utilized to normalize the representations extracted from different foundation models. Moreover F can be implemented as CLIP [26], BERT [43] and GPT-2 [27] when task label y is text. Note that label embedding extraction can be fast enough with GPU parallel computation. We provide the runtime analysis in Appendix Sec.C.

Benefits of F-Label. F-Label has several advantages over one-hot label representations. Firstly, it embeds richer semantic information than one-hot label, leading to accurate modeling of the semantic relationships between different labels. As shown in Fig.3 , F-Label leads to a higher correlation between fine-grained classes than one-hot encoding. Secondly, compared with one-hot labels, F-label can be obtained in a variety of tasks as long as the task label can be transformed into a text sequence. With the assistance of F-Labels, model selection can be established in multi-task scenarios.

4.2 Regression with Unified Noisy Label Embeddings

To estimate the transferability of pre-trained models, the relationship between model features ˆ x ∈ R D and F-Label z ∈ R L should be modeled in order to calculate the transferability score T m in Eqn. (1). On the other hand, since a semantic label y can be embedded by several foundation models, the label embedding set can be constructed as Z = { z k = F k ( y ) / ∥ F k ( y ) ∥ 2 , k ∈ [ K ] } where { F k } K k =1 denotes K foundation models. Now, we utilize data points { (ˆ x n k , z n 1 , · · · , z n K ) } N n =1 to model the relationship between model features and F-Labels.

Setup. As shown in Fig.2 , we assume that true label embedding z is a linear mapping of the model feature with additive Gaussian noise with a variance of σ 2 0 , as given by z = z 0 + ϵ = w T ˆ x + ϵ and ϵ ∼ N (0 , σ 2 0 I L ) where z 0 = w T ˆ x is the regression prediction, w ∈ R D × L and ϵ are regression weights and regression error, respectively, and I L is a L-by-L identity matrix.

We assume that F-labels { z k } K k =1 obtained from different foundation models are oracles that independently provide noisy estimates of the true label embedding z . Formally, we have P ( z k | z ) = N ( z, σ 2 k I L ) . By the above setup, EMMS would be performed with noisy labels. Hence, EMMS tends to select pre-trained models robust to the label noise.

Reasonableness of the Linear Assumption. Specifically, EMMS assumes that the true label embedding z is a linear mapping of the model feature with Gaussian noise. The linear assumption is reasonable in image and text classification tasks because a linear classifier is usually used when the pre-trained model is transferred to a target task, which is commonly used in recent methods. For example, LogME [13] assumes that: z ← N ( w T ˆ x, β -1 ) , which implies that there is a linear mapping from the model feature space to the label space. PACTran [24] also has a similar setting. The difference is that LogME takes a one-hot label as the true label embedding, which limits its applicability. But our EMMS treat the true label embedding z as an implicit variable. And F-Labels { z k } K k =1 obtained from different foundation models are assumed to be noisy oracles of true label embedding z . Since labels in many tasks can be easily encoded into F-Lablels, our EMMS can be used as a multitask model selector. We verify effectiveness the linear assumption in various multi-model tasks with extensive experiments in Sec.5.

Computation of Log-Likelihood. To model the relationship between model features and F-Labels, we need to estimate regression weights w , strengths of label noises { σ k } K k =0 . For simplicity of notation, we consider the case L = 1 , i.e. F-labels are scalars. Given N data points, the loglikelihood is given by

where A 1 = ∏ K k =0 1 /σ k , A 2 = ∑ K k =0 1 / 2 σ 2 k , A n 3 = ∑ K k =0 z n k /σ 2 k , and A n 4 = ∑ K k =0 ( z n k ) 2 /σ 2 k . The detailed derivation of Eqn.(2) is provided in the Appendix Sec.A.

Maximizing Log-likelihood as Weighted Linear Square Regression (WLSR). The remaining issue is to determine parameters w and { σ k } K k =0 by maximizing the log-likelihood in Eqn. (2). But it can be intractable because w and { σ k } K k =0 are heavily coupled. To mitigate this issue, we turn the log-likelihood maximization into a weighted linear square regression by rearranging Eqn. (2) as -L = 1 2 ∥ Xw -Zt ∥ 2 2 + R ( { σ k } K k =0 , where X ∈ R N × D is the data matrix whose n -th row is model feature (ˆ x n ) T , w ∈ R D × 1 are weight parameters, Z ∈ R N × K is F-Label matrix whose k -th column is the label embedding z k , and t ∈ R K × 1 satisfies that 1 T K t = 1 , t ≥ 0 which is a ( K -1) -D simplex denoted as △ K -1 . R ( · ) is a regularization term parameterized with { σ k } K k =0 . We provide the derivations in Appendix Sec.A.

We note that the computational intractability comes from the data-dependent regularizer R ( · ) . For efficient computation, we drop R ( · ) , turning the log-likelihood maximization into a problem of WLSR, as given by

When considering the case L > 1 , Eqn. (3) becomes min w ∈ R D × L ,t ∈△ K -1 1 2 ∥ Xw -Zt ∥ 2 F where Z ∈ R N × L × K and ∥·∥ F is Frobenius norm. From Eqn. (2) and Eqn. (3), s ( w,t ) is an approximation

Algorithm 1 Alternating Minimization

1: Input: Model feature X ∈ R N × D ; FLabel matrix Z ∈ R N × K ; Learning stepsizes η and β for w and t , respectively;

2: Output: Score of WLSR ;

3: Initialize t = 1 K 1 K and w = 1 D 1 D ;

4: while s not converge do

5: s = 1 2 ∥ Xw -Zt ∥ 2 2 ;

6: w ← w -ηX T ( Xw -Zt ) ;

7: while t not converge do

8: t ← t -βZ T ( Zt -Xw ) ;

9: t = Π △ K -1 ( t ) ; // Projection

10: end while

11: end while

12: Return: s

of negative log-likelihood. Hence, a smaller s ( w,t ) indicate the larger T m in Eqn. (1) and better transferability. We design an efficient algorithm to solve WLSR.

Algorithm 2 Fast Alternating Minimization

1: Input: Model feature X ∈ R N × D , FLabel matrix Z ∈ R N × K ;

2: Output: Score of WLSR ;

3: Initialize t = 1 K 1 K and w = 1 D 1 D ;

4: while s not converge do

5: s = 1 2 ∥ Xw -Zt ∥ 2 2 ;

6: w = ( X T X ) -1 X T Zt ; // LSR for w

7: t = ( Z T Z ) -1 Z T Xw ; // LSR for t

8: t = Sparsemax( t ) ; // Projection

9: end while

10: Return: s

4.3 Fast Computation by Alternating Minimization

Algorithm. The optimization problem in Eqn. (3) can be formulated to a classical second-order conic program[42][44](simply called SOCP ). However, the excessive data in our problem leads to a large dimension of the variable, making it inefficient for standard solvers. Therefore, we are motivated to find the smooth structure of the problem and design an alternating minimization algorithm to achieve fast computation. As shown in Algorithm 1 , we separately fix w and t to optimize the other one until the function value in Eqn. (3) converges. Specifically, when we fix t , the whole problem degenerates to a least square problem with respect to w . When we fix w , we also need to solve a least square problem concerning t under the simplex constraint.

Convergence Analysis. We will prove the convergence property of the function value. Indeed, we prove a stronger condition that the function value decreases after each round of iterations on w and t . From the monotone convergence theorem, the convergence can thus be derived. We first present the decreasing result of inner loop of t by Theorem 1 and the same property holds for the update of s . Then the convergence of the whole algorithm can be derived by Theorem 2. The detailed proofs are placed in the Appendix Sec.A.

Theorem 1. Suppose s ( w,t ) = 1 2 ∥ Xw -Zt ∥ 2 F where X ∈ R N × D , Z ∈ R N × K , w ∈ R D × 1 and t ∈ △ K -1 , the inner loop of t in Algorithm 1 lines 7 -10 decreases after each iteration. Specifically, denote β = 1 / ∥ 2 Z T Z ∥ and t + = Π △ K -1 ( t -β ∇ s ( w,t )) . For any t ∈ △ K -1 , s ( w,t + ) -s ( w,t ) ≤ -1 2 β ∥ t -t + ∥ 2 ≤ 0 .

Theorem 2. Suppose s ( w,t ) = 1 2 ∥ Xw -Zt ∥ 2 2 where X ∈ R N × D , Z ∈ R N × K , w ∈ R D × 1 and t ∈ △ K -1 , the function value in Algorithm 1 will converge. Specifically, denote w ⋆ , t ⋆ as the result after one iteration of w,t respectively, we have 0 ≤ s ( w ⋆ , t ⋆ ) ≤ s ( w ⋆ , t ) ≤ s ( w,t ) .

Computational Speedup. Although this algorithm 1 guarantees convergence, it is a bit timeconsuming due to the two-level loop, we optimized this part and achieved similar results in very little time. Since the least squares solution is extremely fast, we performs least squares on w and t , and then replace projection onto simplex with explicit Sparsemax transformation [45, 46], iteratively. The fast solver is illustrated in Algorithm 2 . we experimentally verify its convergence and find that the approach achieves impressive speedup.

5 Experiment

This section evaluates our method EMMS on different downstream tasks, including image classification, image caption, visual question answering, text question answering and referring expression comprehension. We put more experiments details in Appendix Sec.B. Moreover, we conduct a detailed ablation study to analyze our EMMS in Appendix Sec.C

5.1 Training Details

Benchmark. For image classification , We adopt 11 classification benchmarks , including FGVC Aircraft [47], Caltech-101 [48], Stanford Cars [49], CIFAR-10 [50], CIFAR-100 [50], DTD [51], Oxford 102 Flowers [52], Food-101 [53], Oxford-IIIT Pets [54], SUN397 [55], and VOC2007 [56]. For image caption , We use Flickr8k [57], Flickr30k [58], FlickrStyle10KHumor [59], FlickrStyle10K-Romantic [59] and RSICD [60]. For visual question answer , We apply COCOQA [61], DAQUAR [62] and CLEVR [63]. For text question answer and referring expression comprehension , we separately use SQuAD1.1 [64] ,SQuAD2.0 [65] and RefCOCO [66], RefCOCO+ [66], RefCOCOg [67] .

Ground truth. In order to obtain the ground truth, we finetune all pre-trained models on all target datasets with a grid search of hyper-parameters. Details of target datasets and fine-tuning schemes are described in Appendix Sec.B.

Evaluation protocol. To assess how well a model selector predict the transferability of pre-trained models, we calculate the rank correlation between { T m } M m =1 and { G m } M m =1 . Following the common practice [13, 31], we use weighted Kendall's τ w . The larger τ w indicates a better correlation and better transferability metric. For computation complexity, we record the runtime of executing algorithm

over all models given the feature and label on a target task and analyzed the computational complexity of EMMS as well as LogME. (Details can be found in Appendix Sec.C)

Baseline. For the image classification task, we choose NLEEP [31], TransRate [23], and LogME [13]as the baseline; for other multimodal tasks, we choose LogME with F-Label as the baseline; in addition, for the VQA task, we additionally compare PACTran [24]. Details of baselines and why we choose them are described in Appendix Sec.B.

5.2 Image Classification with ViT Models

Vision transformer [9] (ViT) models have been increasingly used for a variety of tasks and have achieved better results than CNN models. The architecture of ViT models are more complex than CNN models. Hence, how to do the model selection on ViT models is a more challenging and rewarding task. Details of pre-trained models are described in Appendix Sec.B.

Performance and wall-clock time comparison. As shown in Table.1 , our EMMS achieve the best average τ w on 11 target datasets and the best τ w on 9 target datasets with relatively short time. For example, EMMS outperforms LogME by 0.182 and 0.139 rank correlation τ w on Aircraft, and VOC2007, respectively, showing the effectiveness of our EMMS in measuring the transfer-ability of pre-trained ViT models. On the other hand, for the remaining 2 target datasets (i.e. CF-10, DTD), our EMMS still has a marginal gap compared to the best-performing transferability metric. Besides, we find that the effect of model selection of EMMS in ViT models selection has an improvement compared to CNN models selection, we guess F-Label has spatial similarity with the model feature of ViT-base model because the foundation models are mostly transformer-based, which can model the relationship between model feature from Vit-base models and F-Labels more accurately.

5.3 Image Captioning

Here we treat image caption as a vocab-based classification task. That is we use a vocabulary and classify the caption into the index of some words in the vocabulary. Afterward, training is done according to the classification task criteria .Here we calculate the average τ w and time of LogME with K single F-label from K foundation models we use respectively. We wants to select the best combination of image encoder and language encoder. Details of pre-trained models and the model architecture are described in Appendix Sec.B.

Performance and wall-clock time comparison. As shown in Table.2, EMMS is significantly ahead of baseline in both time and effect for each dataset. For example, EMMS outperforms LogME with the relative improvements of 39% and 37% in rank correlation τ w on Flickr8k and Flickr30k, respectively. In addition, the time of EMMS is reduced by 83.7% and 79.8% relative to LogME on these two datasets, which shows the efficiency of our algorithm. The average rank correlation τ w alone the five datasets is 0.64, which denotes EMMS has sufficient confidence.

5.4 Text Question Answering

For natural language understanding, we consider Text Question Answering (TQA) as a reading comprehension task, where the response to each question is a text segment extracted directly from the affiliated reading passage, or the question may indeed be deemed unanswerable. Details of pre-trained models and how to finetune are described in Appendix Sec.B.

Performance and wall-clock time comparison. In Table 3, the performance improvement of EMMS on the TQA is consistent with the enhancements observed in the earlier mentioned computer vision tasks. More specifically, our EMMS attains accuracies of 60.3% and 46.3% on the Stanford Question Answering Dataset (SQuAD) versions 1.1 and 2.0 respectively, using rank correlation τ w as an evaluation metric. This represents a significant relative increment of 11.2% and 13.2% compared to the performance of LogME.

5.5 Referring Expression Comprehension

Referring expression comprehension (REC) is a widely challenging task because it requires precise alignment between linguistic concepts and image features. To address this, the objects in each image are represented as a sequence of discrete tokens, while their bounding box corner coordinates are turned into integer location tokens. This allows for a unified F-Label to be extracted using various language models. More details about the pre-trained models can be found in Appendix Sec.B.

Performance and wall-clock time comparison. As shown in Table 4, our EMMS continues to exhibit its superiority in the enhancement of performance on the REC task, an instance-level crossmodal localization task. Specifically, the proposed EMMS produces accuracies of 45.8%, 54.9%, and 52.1% on the RefCOCO, RefCOCO+, and RefCOCOg datasets respectively. This significantly surpasses its counterpart, LogME, in terms of margins when evaluated with rank correlation τ w .

5.6 Ablation Analysis

Comparison with different number of F-Label Here we denote the number of F-Label is K and choose the image caption task to illustrate the impact of K on our solution. As shown in Table 6. We find that increasing K in a certain range brings a gain in effectiveness to our method, but when K becomes larger, the time also increases and we find that K = 4 is not as effective as K = 3 . We believe that the increase in K brings difficulties in fitting the true Label, resulting in a loss of effectiveness. Therefore, we use K = 3 for the sake of effect and time.

Performance on F-Label using small model On the one hand, using foundation model can extract the joint embedding compared to the small model, which allows EMMS to be extended to tasks with multiple forms of labels. On the other hand, the foundation model can handle many types of tasks,

so we can use the foundation model for different tasks for label embedding. As shown in Table 5, we experimentally demonstrate that the use of the foundation model leads to more accurate F-Label extraction and thus to an improvement in the performance of the method.

The effect of using a single foundation model We investigate how EMMS is influenced when only a single foundation model is provided. We conduct experiments on image classification and image captioning. We consider EMMS with the single foundation model including language foundation model (1) GPT-2 [27], (2) BERT [43] , (3) RoBerta [68], and multimodal foundation model (4) CLIP [26], (5) FLAVA [69], and (6) AltCLIP [70]. For comparison, we include the result of our EMMS with default setting (K=3, i.e. CLIP, BERT, and GPT-2) and the result of previous state-ofthe-art methods obtained from LogME, NLEEP and TransRate. The results are reported in Table 20 and Table 5.

We have several observations. (1) Different downstream tasks prefer F-Labels obtained from different foundation models. No single foundation model is dominant in all target tasks. In particular, CLIP is not the best model for extracting F-Labels. (2) For image captioning, multimodal foundation models are more appropriate for extracting F-Labels than language foundation models. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.

6 Conclusion

How to select a pre-trained model for different tasks quickly and effectively is an important issue in the field of transfer learning. This paper proposes an efficient multi-task model selector(EMMS) that can be applied to many types of tasks. EMMS uses foundation model for Label embedding in order to transform diverse label formats of different tasks into the same form and see them as noisy labels. To estimate a model's transferability, EMMS model this problem as a simple weighted linear regression, which can be solved use an alternating minimization algorithm. Compared with existing methods, EMMS achieves the first model selection in multi-task scenarios, including image caption, referring segmentation, etc., with high speed and great results. For the limitations of the method, if the foundation model generalize very poor on downstream tasks, it may lead to low-quality label embedding, which is a drawback of our method. Moreover, building a holistic benchmark of various label embeddings would be useful in many applications such as multi-modal adaptation [71]. We leave it as a future work.

7 Acknowledgments

This paper is partially supported by the National Key RD Program of China No.2022ZD0161000, National Key RD Program of China(NO.2022ZD0160100) and the General Research Fund of Hong

Kong No.17200622. Besides, thanks Ruimao Zhang from CUHK(SZ) for thoughtful discussion and Prof. Anthony Man-Cho So from CUHK for his valuable discusion about solving the WLSR in the paper. At last, this work was done during the internship at Shanghai Artificial Intelligence Laboratory.

References

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 , 2014.

Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International conference on machine learning , pages 2397-2406. PMLR, 2016.

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18155-18165, 2022.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 59(2):64-73, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision , pages 568-578, 2021.

Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning , pages 7294-7305. PMLR, 2020.

Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning , pages 12133-12143. PMLR, 2021.

Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in a self-challenging fisher space. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV , pages 286-302. Springer, 2022.

Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pretrained models are from neural collapse on the target dataset informs their transferability. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5549-5558, 2023.

Xiaotong Li, Zixuan Hu, Yixiao Ge, Ying Shan, and Ling-Yu Duan. Exploring model transferability through the lens of potential energy. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5429-5438, 2023.

Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etran: Energy-based transferability estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18613-18622, 2023.

Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. arXiv preprint arXiv:2306.03900 , 2023.

Zhigang Hu, Yuhang Huang, Hao Zheng, Meiguang Zheng, and JianJun Liu. Graph-based fine-grained model selection for multi-source domain. Pattern Analysis and Applications , pages 1-12, 2023.

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems , 27, 2014.

Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4918-4927, 2019.

Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3712-3722, 2018.

Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In International Conference on Machine Learning , pages 9201-9225. PMLR, 2022.

Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pacbayesian metrics for estimating the transferability of pretrained models to classification tasks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV , pages 252-268. Springer, 2022.

Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using bhattacharyya class separability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9172-9182, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015.

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. arXiv preprint arXiv:1808.07036 , 2018.

Yue Wu and Qiang Ji. Facial landmark detection: A literature survey. International Journal of Computer Vision , 127:115-142, 2019.

Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2663-2673, 2021.

Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernock` y, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech , volume 2, pages 1045-1048. Makuhari, 2010.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 1532-1543, 2014.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052 , 2022.

Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16804-16815, 2022.

Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016 , 2022.

Wenqi Shao, Yutao Hu, Peng Gao, Meng Lei, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, et al. Tiny lvlm-ehub: Early multimodal experiments with bard. arXiv preprint arXiv:2308.03729 , 2023.

Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265 , 2023.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning , pages 2790-2799. PMLR, 2019.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 , 2021.

Farid Alizadeh and Donald Goldfarb. Second-order cone programming. Mathematical programming , 95(1):3-51, 2003.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Miguel Sousa Lobo, Lieven Vandenberghe, Stephen Boyd, and Hervé Lebret. Applications of second-order cone programming. Linear algebra and its applications , 284(1-3):193-228, 1998.

Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning , pages 16141623. PMLR, 2016.

Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 443-451, 2019.

Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Finegrained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop , pages 178-178. IEEE, 2004.

Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting a large-scale dataset of fine-grained cars. 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 ieee conference on computer vision and pattern recognition (cvpr). Las Vegas, NV, USA , 1:770-78, 2016.

Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-338, 2010.

Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk , pages 139-147, 2010.

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision , pages 2641-2649, 2015.

Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3137-3146, 2017.

Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing , 56(4):2183-2195, 2017.

Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems , 28, 2015.

Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. Advances in neural information processing systems , 27, 2014.

Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2901-2910, 2017.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822 , 2018.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14 , pages 69-85. Springer, 2016.

Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 11-20, 2016.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15638-15650, 2022.

Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. Altclip: Altering the language encoder in clip for extended language capabilities. arXiv preprint arXiv:2211.06679 , 2022.

Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramana. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. arXiv preprint arXiv:2301.06267 , 2023.

Charles L Byrne. Alternating minimization and alternating projection algorithms: A tutorial. Sciences New York , pages 1-41, 2011.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 , 2019.

Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555 , 2020.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2820-2828, 2019.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

X Chen, S Xie, and K He. An empirical study of training self-supervised visual transformers. arxiv e-prints. arXiv preprint arXiv:2104.02057 , 2021.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12009-12019, 2022.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6904-6913, 2017.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems , 32, 2019.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020.

Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543 , 2021.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping languageimage pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 12888-12900. PMLR, 2022.

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems , 34:9694-9705, 2021.

Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning , pages 23318-23340. PMLR, 2022.

Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint arXiv:2002.11770 , 2020.

A Method

Here we derive in detail the regression with Unified Noisy Label Embeddings that appear in the method section of the text in Sec.A.1 and give complete proof of the convergence of the method in Sec.A.2.

A.1 Regression with Unified Noisy Label Embeddings

Setup. we assume that label embedding z is a linear mapping of the model feature with additive Gaussian noise with a variance of σ 2 0 , as given by z = z 0 + ϵ = w T ˆ x + ϵ and ϵ ∼ N (0 , σ 2 0 I L ) where z 0 = w T ˆ x is the regression prediction, w ∈ R D × L and ϵ are regression weights and regression error, respectively, and I L is a L-by-L identity matrix.

We assume that F-labels { z k } K k =1 obtained from different foundation models are oracles that independently provide noisy estimates of the label embedding z . Formally, we have P ( z k | z ) = N ( z, σ 2 k I L ) . Without loss of generality, we assume that L = 1

Then the joint probability over noisy labels for a fixed n , That is, for given x n , we have:

Due to the independence between z k and x , using the real label z , we can rewrite it as:

And using the independencies among z k , we have:

Due to P ( z k | z ) = N ( z, σ 2 k I L ) , we can rewrite it as :

which can be calculated as :

where A 1 = ∏ K k =0 1 /σ k , A 2 = ∑ K k =0 1 / 2 σ 2 k , A n 3 = ∑ K k =0 z n k /σ 2 k , and A n 4 = ∑ K k =0 ( z n k ) 2 / 2 σ 2 k

Consider the joint probability over all N instances, we have:

where X ∈ R N × D denotes the feature matrix, N is the number of data points and D is the number of features.

Then given N data points, the negative log-likelihood is given by

where L 1 and L 2 are given by

Since L 1 is independent of input data, we focus on L 2 . To simplify the notation, we re-denote γ k = 1 /σ 2 k and Γ = ∑ K k =1 γ k . Using this notation, L 2 can be rearranged as:

Hence, the negative likelihood in Eqn.(10 can be written as

where R ( γ k ) = L 1 + ∑ K k =1 γ k ( z n k ) 2 -(1+ γ 0 Γ(Γ+ γ 0 ) ( ∑ K k =1 γ k z n k ) 2 . The computational intractability of Eqn.(16) comes from the regularization term R ( γ k ) . Note that the coefficient Γ γ 0 Γ+ γ 0 > 0 and ∑ K k =1 γ k Γ = 1 . By removing regularizer R ( γ k ) and positive scale parameter Γ γ 0 Γ+ γ 0 , the minimization of negative log-likelihood can be approximately treated as a weighted linear square regression, as given by

In Eqn.(17), X ∈ R N × D is the data matrix whose n -th row is model feature (ˆ x n ) T , w ∈ R D × 1 are weight parameters, Z ∈ R N × K is F-Label matrix whose k -th column is the label embedding z k , and t ∈ R K × 1 satisfies that 1 T K t = 1 , t ≥ 0 which is a ( K -1) -D simplex denoted as △ K -1 .

A.2 Convergence Analysis and Proof Outline

We will prove the convergence property of the function value. Indeed, we demonstrate a stronger condition that the function value decreases after each round of iterations on w and t . From the monotone convergence theorem, the convergence can thus be derived. For other convergence properties of alternating minimization, readers can refer to the literature [72], which can be of independent interest.

In the proof, we exploit the smoothness of the function and design a projection gradient descent method with sufficient decrease for the constraint optimization problem. The sufficient decrease in the unconstrained problem is a direct corollary.

Definition 1. A function f ( x ) : R d → R is said to be β -smooth with constant β if

Lemma 1. Suppose X is the simplex constraint, and y ∈ R d , Π denotes the projection operator. Then the inequality holds

Proof. For the projection Π X ( y ) , it is a convex optimization problem and can be formulated as

where x T 1 = 1 and x > 0 . We denote x ⋆ as the optimal solution to the problem. For the convex optimization problem, it holds for all x ∈ R d that

Therefore we can derive

The lemma is proved.

Lemma 2. Let f be a β -smooth function. For any x, y ∈ dom( f )

Proof.

The last inequality holds because f is a β -smooth function.

Lemma 3. Suppose the function f is the β -smooth function, and X is the simplex constraint. For any x, y ∈ X , let x + = Π X ( x -1 β ∇ f ( x )) and g X ( x ) = β ( x -x + ) . Then the inequality holds

Proof. Using Lemma. 1, we have

By using Lemma. 2 and the fact f ( x + ) -f ( y ) = f ( x + ) -f ( x ) + f ( x ) -f ( y ) , we have

Theorem 3. Suppose s ( w,t ) = 1 2 ∥ Xw -Zt ∥ 2 F where X ∈ R N × D , Z ∈ R N × K , w ∈ R D × 1 and t ∈ △ K -1 , the inner loop of t in Algorithm lines 7 -10 decreases after each iteration. Specifically, denote β = 1 / ∥ 2 Z T Z ∥ and t + = Π △ K -1 ( t -β ∇ s ( w,t )) . For any t ∈ △ K -1 , s ( w,t + ) -s ( w,t ) ≤ -1 2 β ∥ t -t + ∥ 2 ≤ 0 .

which is equivalent to

Proof. Since we fix parameter w and consider the optimization problem of variable t , we denote s ( t ) = s ( w,t ) , which leads to ∇ s ( t ) = -2 Z T ( Xw ⋆ -Zt ) . For any t 1 , t 2 ∈ dom( s )

According to the definition 1, it shows that the f ( t ) is β -smooth, where β = ∥ 2 Z T Z ∥ . We denote t ∈ △ K -1 to be the initial point and t + to be the result of one iteration of t , where t + = Π △ K -1 ( t -1 β ∇ f ( t )) . From Lemma 3, we can replace x + , y and x with t + , t , and t , repsectively. In this way, the inequality holds

Therefore, according to Monotone convergence theorem , the function value of the iterative algorithm will converge.

Theorem 4. Suppose s ( w,t ) = 1 2 ∥ Xw -Zt ∥ 2 2 where X ∈ R N × D , Z ∈ R N × K , w ∈ R D × 1 and t ∈ △ K -1 , the function value in Algorithm will be convergent. Specifically, denote w ⋆ , t ⋆ as the result after one iteration of w,t respectively, we have 0 ≤ s ( w ⋆ , t ⋆ ) ≤ s ( w ⋆ , t ) ≤ s ( w,t ) .

Proof. In the first step, we denote t ∈ △ K -1 is the initial point, then use gradient descent algorithm to calculate w ⋆ . Since the optimization problem for w is a convex optimization problem and use lemma 2, the decreasing property for the gradient part can be derived. That is, for each w ∈ R D × 1 , we have s ( w ⋆ , t ) ≤ s ( w,t ) . In the second step, we fix w as w ⋆ , from Theorem 3, we have s ( w ⋆ , t ⋆ ) ≤ s ( w ⋆ , t ) . Therefore, the value of s ( w,t ) satisfies: 0 ≤ s ( w ⋆ , t ⋆ ) ≤ s ( w ⋆ , t ) ≤ s ( w,t ) , from Monotone convergence theorem , s ( w,t ) converges to the limiting point. As shown above, the overall convergence of our algorithm is guaranteed.

B Experiment

In this section, we present detailed descriptions of datasets in Sec. B.2, pre-trained models and baselines in Sec. B.3, and ground-truth scores in Sec. B.4 in various target tasks. More ablation studies can be found in Sec. C.

Foundation Models. On image classification, image captioning, referring expression comprehension, and visual question answering, we use foundation models CLIP [26], BERT [43] and GPT-2 [27]. On text question answering, we use foundation models GPT-2 [27], BART [73], and ELECTRA [74]. CLIP was trained on a large dataset of images and their corresponding captions, which can understand the relationship between images and text. BERT is a pre-trained language model that can understand and generate natural language. GPT-2 was trained on a large corpus of text and can be fine-tuned for specific tasks such as text completion and text summarization. Bart is a sequence-to-sequence model, which is both auto-regressive and bidirectional. Electra is a different type of language model that key idea is to pre-train a generator model to produce fake data and shows promising results in various NLP tasks.

Interpretation of weighted Kendall's tau. The Kendall's τ represents the ratio of concordant pairs minus discordant pairs when enumerating all pairs of { T m } M m =1 and { G m } M m =1 as given by

where sgn( x ) returns -1 if x < 0 and 1 otherwise. In this work, a weighted version of Kendall's τ , denoted as τ w , is employed to assess transferability metrics considering that a top-performing model is always preferred for target tasks in transfer learning. In principle, a larger τ w implies the transferability metric can rank pre-trained models better. And if a metric can rank top-performing models better, τ w would be also larger. We also use other measurements to assess the performance of transferability metrics in Table 17 of Sec. C.

B.1 More experimental results

B.1.1 Performance on Image Classification with CNN Models

Performance and wall-clock time comparison. We compare EMMS with previous LEEP, NLEEP, LogME, and TransRate. As shown in Table.7, our EMMS achieve the best average τ w on 11 target datasets and the best τ w on 6 target datasets. Compared to NLEEP, which is the most effective other than EMMS, we have almost 1/40 of the time of NLEEP.

B.1.2 Performance on Visual Question Answering

To further demonstrate the generality of EMMS in multi-model tasks, we show how EMMS can work for VQA. We follow previous practice ( [24]) which treats VQA as a classification task (vocab-based VQA). That is, we construct a vocabulary based on the top answers in the training sets and classify them into some of those labels. The models to be selected and the architecture is the same as in the image captioning .

Performance and wall-clock time comparison. As shown in Table.8, EMMS is clearly ahead of PACTran in terms of results and time, proving that EMMS has the ability to handle multi-modal tasks very well. We can find that EMMS outperforms PACTran on all datasets. In particular, EMMS achieves 93.8% and 93.7% gain over PACTran on the COCO-QA and CLEVR datasets with rank correlation τ w while reducing time consumption by 75.1% and 34.3% respectively compared to Pactran. This indicates that EMMS performs well on both ordinary VQA datasets(DAQUAR, COCO-QA) as well as VQA datasets(CLEVR) that focus on inference capabilities.

B.1.3 Regression

In addition to image classification and a variety of multi-modal tasks, here we show that EMMS can also be used for regression tasks. The daatasets for regression task we use is CUB200 [75] and IIIT

Pets [54]. The input is an image containing various birds and pets, respectively. We need to predict the coordinates of the bird's or pet's bounding box in the image and mean square error (MSE) on the test data is the ground-truth. The pre-trained models used are the same as the image classification task with CNN models and the only baseline is LogME. We extract F-Labels using Bert and RoBerta.

As shown in Table 9, EMMS significantly outperforms LogME, with 29.5% and 13.9% performance improvement on CUB and Pets respectively.

B.2 Descriptions of Datasets

B.2.1 Image Classification

For image classification, we adopt 11 classification benchmarks , including FGVC Aircraft [47], Caltech-101 [48], Stanford Cars [49], CIFAR-10 [50], CIFAR-100 [50], DTD [51], Oxford 102 Flowers [52], Food-101 [53], Oxford-IIIT Pets [54], SUN397 [55], and VOC2007 [56]. These datasets cover a broad range of classification tasks, which include scene, texture, and coarse/finegrained image classification, which are widely used in transfer learning. In particular, CF10 and VOC2007 are typical coarse-grained classification datasets, Aircraft, and Cars are typical fine-grained classification datasets, and CF100 contains both coarse- and fine-grained classifications.

B.2.2 Image Captioning

For image captioning, We use Flickr8k [57], Flickr30k [58], FlickrStyle10K-Humor [59], FlickrStyle10K-Romantic [59] and RSICD [60]. Among them, Flickr8k and Flickr30k have commonly used image captioning datasets for natural images and have no emotional color; RSICD is a commonly used image captioning dataset in remote sensing; Flickr10k-H and Flickr10k-R are also image captioning datasets for natural images, but their images are depicted with humorous and romantic emotional colors, respectively.

B.2.3 Visual Question Answering

For visual question answering, we apply COCOQA [61], DAQUAR [62] and CLEVR [63].Among them, DAQUAR is an early VQA dataset on real images; CLEVR is a synthetic dataset, which is a visual scene composed of some simple geometric shapes, focusing on evaluating the inference ability of VQA models; the questions and answers of COCO-QA are generated by NLP algorithms, and the images are from the COCO dataset, which is also a commonly used VQA dataset.

B.2.4 Text Question Answering

For text question answering, we separately use SQuAD1.1 [64] ,SQuAD2.0 [65], which are collections of question-answer pairs derived from Wikipedia articles and are widely used in text question answer.

B.2.5 Referring Expression Comprehension

For referring expression comprehension, we separately use RefCOCO [66], RefCOCO+ [66] and RefCOCOg [67].Specifically, RefCOCO includes instances where there is only one object of its kind in the image, while RefCOCO+ includes instances where multiple objects of the same kind exist in the image.

B.3 Pre-trained Models and Baselines

B.3.1 Image Classification

Pre-trained Models. For CNN-based models, We select 11 widely-used CNN models including ResNet-34 [6], ResNet-50 [6], ResNet-101 [6], ResNet-152 [6], DenseNet-121 [7], DenseNet-169 [7], DenseNet-201 [7], MNet-A1 [76], MobileNetV2 [77], GoogleNet [8], and InceptionV3 [78]. All these models are trained on ImageNet dataset [1], which are widely used within the field of migration learning. For ViT-based models, we collect 10 ViT models including ViT-T [9], ViT-S [9], ViT-B [9], DINO-S [79], MoCov3-S [80] , PVTv2-B2 [11], PVT-T [11], PVT-S [11], PVT-M [11], and Swin-T [10], which are widely used in various vision tasks. Besides, we append EMMS with one-hot label, which degenerates to a linear regression whose label is the one-hot vector. We fine-tune these models on the 11 target datasets to obtain the ground truth.

Comparison Baselines. Here we use some of the latest methods as baselines, including LEEP [12], NLEEP [31], LogME [13], and TransRate [23], which have been experimented with model selection on image classification tasks.

B.3.2 Image Captioning

Pre-trained Models. We use a classic and effective image captioning model architecture, which contains an image encoder and a language encoder to extract the features of the image and the corresponding caption, then fuses the image feature and the text feature and input it to the classifier. We aim to choose the best combination of image encoder and language encoder. Besides, We finetune each model in COCO Caption [81] and use these as the pre-trained models.

Specifically, We separately use ViT-B [9],Swin-B [10], Swinv2-B [82] as image encoder and Bert [43], Roberta [68], Bart [73] as language encoder, and use VisionEncoderDecoderModel from HuggingFace as the model architecture. Following the setting in PACTran [24], We finetune the model in COCO Caption [81] and use these as the pre-trained models. Following common practice( [83]) , we treat image captioning as a vocab-based classification task. That is we use a vocabulary and classify the caption into the index of some words in the vocabulary. Afterward, training is done according to the classification task criteria.

Comparison Baselines. In this common setup, each caption is converted to a matrix Y ∈ R L × N , where L denotes the length of the caption after padding or truncation and N denotes the size of the vocabulary, and each row in the matrix is a one-hot vector. Since N is generally very large, Existing model selection metrics do not scale to this case due to the huge amount of time spent. The only baseline we use is to model the fused feature with F-label using LogME since only LogME can handle the regression task. Here we calculate the average τ w and time of it with K single F-label from K foundation models we use respectively.

B.3.3 Visual Question Answering

Pre-trained Models. The model architecture and the model selection settings are the same as in the image captioning, Following the setting in PACTran [24], here we use the model after finetune on VQA-v2 [83] as the pre-trained model waiting for selection and treat VQA as a vocab-based classification task.

Comparison Baselines. Here we calculate the average τ w and time of it with K single F-label from K foundation models we use respectively. And in addition to that, the three methods proposed in PACTran [24] are added here, which are the only methods currently applied to VQA tasks.

B.3.4 Text Question Answering

Pre-trained Models. The selected models include BERT-Large [43], RoBERTa-Large [68], XLNetLarge [84], DeBERTa [85] (XLarge), DeBERTa-V2 [85] (XLarge and XXLarge), DeBERTa-V3 [86] (Base, Small, XSmall). More specifically, we simultaneously input the question and passage into the aforementioned models, utilizing the distinctive symbol [SEP] to demarcate them. By stacking the predicted head onto each model, we could further fine-tune the model such that it can predict the start and end positions of the answer within the passage. This is achieved by using two binary classifiers, where one is dedicated to identifying the start position and the other to pinpointing the end.

Comparison Baselines. Here we calculate the average τ w and time of it with F-labels from K foundation models respectively.

B.3.5 Referring Expression Comprehension

Pre-trained Models. The candidate multi-modal architectures considered for REC task incorporate Blip [87], ALBEF [88], CLIP [26] (ViT-B-32, ViT-B-16, ViT-L-14, ViT-L-14-336, RN50), OFA [89] (Base, Large, Huge). In practice, we respectively extract the visual and textual representations from each of these models and feed them into a multi-modal interaction module followed by a stacked detection head, and further fine-tune the model to generate the ground truth of model selection.

Comparison Baselines. Here we calculate the average τ w and time of LogME with K single F-label from K foundation models we use respectively.

B.4 Fine-tuning Score on Various Target Tasks

B.4.1 Image Classification

Fine-tuning Details. The ground truth of the problem of pre-trained model ranking is to fine-tune all pre-trained models with a hyper-parameters sweep on target datasets. Given the model and the target dataset, two of the most important parameters would be learning rate and weight decay in optimizing the model [90]. Therefore, we carefully fine-tune pre-trained models with a grid search of learning rate in { 1 e -1 , 1 e -2 , 1 e -3 , 1 e -4 } and weight decay in { 1 e -3 , 1 e -4 , 1 e -5 , 1 e -6 , 0 } . And using SGD optimizer. After determining the best hyper-parameters candidate, we fine-tune the pre-trained model on the target dataset with the candidate and then obtain the test accuracy as the ground truth. We use a Tesla V100 with a batch size of 128 to perform finetuning. All input images are resized to 224 × 224 . To avoid random error, we repeat the above fine-tuning procedure three times and take an average to obtain the final fine-tuning accuracy. For reference, we list the fine-tuning accuracy of supervised CNN models in Table.10, and vision transformer models in Table 11, respectively.

B.4.2 Image Captioning and Visual Question Answering

Fine-tuning Details. The setting of finetune here is approximately the same as in image classification. We carefully fine-tune pre-trained models with a grid search of learning rate in { 1 e -4 , 1 e -5 , 1 e -6 } and weight decay in { 1 e -4 , 1 e -5 , 1 e -6 } . And using AdamW optimizer. After determining the best hyper-parameters candidate, we fine-tune the pre-trained model on the target dataset with the candidate and then obtain the test BLEU-4 and accuracy as the ground truth. However, since Flickr10k-H and Flickr10k-R do not provide a test set, we use a 6:1 ratio to divide the original training set of 7000 images into a training set and a test set. For visual question answering, Due to the lack of a test set for CLEVR dataset, we also assign its training set as training set and test set in the ratio of 6:1. We use an Nvidia A100 with a batch size of 64 to perform finetuning. All input images are resized to 224 × 224 . To avoid random error, we repeat the above fine-tuning procedure three times and take an average to obtain the final fine-tuning accuracy. For inference, We use BLEU-4 as the score for the model with image captioning and accurarcy as the score for the model with VQA. we list result of image captioning models in Table.12, and visual question answering models in Table 13, respectively.

B.4.3 Text Question Answering

Fine-tuning Details. The accuracy of most models in TQA is provided by DeBERTa [85, 86], except for DeBERTa-V3 [86](Base, Small, XSmall). Following the setting of Bert [43], we finetune these models with a batch size of 24 for 2 epochs. We use AdamW optimizer with an initial learning rate of 3 e -5 , polynomial decay. The Dev F1 score is used for pre-trained model ranking. All experiments are implemented on an NVIDIA Tesla A100 GPU. The finetune accuracy is shown in Table 14.

B.4.4 Referring Expression Comprehension

Fine-tuning Details. For referring expression comprehension, the standard metric Acc@0.5 on the validation set is used as the ground truth. For finetuning, we use a batch size of 128 with a resolution of 512 × 512 for each image. We finetune the models on each dataset for 12 epochs with a learning rate of { 3 e -5 , 5 e -5 } and weight decay in { 1 e -3 , 1 e -5 } using Adam optimizer. The best performance on the validation set for each task is reported among these hyper-parameters. Table 15 shows the performance of referring expression comprehension models.

Table 16: The fine-tuning MSE on test set of models used in regression on 2 target tasks.

B.4.5 Regression

Fine-tuning Details. For regression, mean square error (MSE) on the test data is the ground truth. For finetuning, we use a batch size of 64 with resolution of 224 × 224 for each image. we carefully fine-tune pre-trained models with a grid search of learning rate in { 1 e -1 , 1 e -2 , 1 e -3 , 1 e -4 } and weight decay in { 1 e -3 , 1 e -4 , 1 e -5 , 1 e -6 , 0 } with SGD optimizer. The fine-tuning MSE on test set of models used in regression is in Table 16

C More Ablation Analysis

The Efftiveness of EMMS under Various Measurements. In addition to weighted Kendall's tau, we employ various other measures to evaluate our EMMS. These include Kendall's tau ( τ ), Pearson's correlation ( r ), weighted Pearson's correlation ( r w ), and topk relative accuracy, denoted as Rel@ k , which represents the ratio between the best fine-tuning accuracy achieved on the downstream task using the top-k ranked models and the best fine-tuning precision achieved with all models. We test the robustness of our transferability metrics to different measurements on the Flickr8k and RSICD datasets for image captioning tasks, as shown in Table 17. Our EMMS consistently outperforms the previous transferability metric, including LogME and TransRate. Under the aforementioned measurements, demonstrating the superiority of our EMMS.

The Effect of Label Embedding In some multimodal tasks or text tasks, including image captioning or text question answering. Label emebdding directly affects the applicability of existing model selection metric to these tasks. In addition, even in classification tasks, the use of F-Label can also bring improvements in results. Here we focus on the comparison between label embedding and direct one-hot vectors for image classification tasks in CNN-based models. As shown in Table 18, the use of F-Label can bring performance improvement compared to One-Hot vector, the average τ w increase from 0.618 to 0.636; furthermore, the use of multiple F-Label also brings some improvement compared to the average of single F-Label with τ w increasing from 0.636 to 0.664.

The Effect of Computational Speedup. Here we experimentally demonstrate the effect of our accelerated algorithm. As shown in Table 21, the algorithm is similar to the in-accelerated version in terms of results, but much shorter in terms of the wall-clock time.

Comparison with different number of iterations The number of iterations affects the EMMS time, here we conduct experiments on the VQA task for the effect of the number of iterations on the results. As shown in Table .19, we find that the number of iterations does not have a large impact on the performance of our method, and even a small number of iterations can guarantee the final result(e.g. the number of iterations is 1). We believe that firstly our method converges very fast. And secondly, for the ranking problem of model ranking, even if the convergence is not sufficient, the original order can still be maintained to a large extent in EMMS, thus ensuring the effect.

The effect of using a single foundation model We investigate how EMMS is influenced when only a single foundation model is provided. We conduct experiments on image classification and image captioning. We consider EMMS with the single foundation model including language foundation model (1) GPT-2 [27], (2) BERT [43] , (3) RoBerta [68], and multimodal foundation model (4) CLIP [26], (5) FLAVA [69], and (6) AltCLIP [70]. For comparison, we include the result of our EMMS with default setting (K=3, i.e. CLIP, BERT, and GPT-2) and the result of previous state-ofthe-art methods obtained from LogME, NLEEP and TransRate. The results are reported in Table 20 and Table 5.

We have several observations. (1) Different downstream tasks prefer F-Labels obtained from different foundation models. No single foundation model is dominant in all target tasks. In particular, CLIP is not the best model for extracting F-Labels. (2) For image classification, both language and multimodal foundation models are competent for acquiring F-Labels. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.

The Wall-clock Time of Label Embedding. For classification tasks, since the maximum number of categories is often only a few hundred, Label Embedding is very fast. Here we focus on documenting the time required for multimodal tasks, e.g. image captioning, text question answering, and referring expression comprehension, where label embedding is more time-consuming. For each task, we use 8 Nvidia A100 GPUs for label embedding, with a batch size of 512 for each GPU. The running time of label embedding for image captioning, text question answering, and referring expression comprehension is shown in Table 22. We measure the time for each dataset on the same CPU device (AMD EPYC 7H12 with 64-Core Processor) for three times and take the average as the final result.

The computational complexity of EMMS. We compare the computational complexity between LogME and EMMS in Table 23. We see that EMMS has lower computation complexity than LogME(F) because LogME(F) needs several iterations (T=3 on average) to converge. Moreover, EMMS allows for full vector computation and can be efficiently solved by existing scientific computation packages such as np.linalg.lstsq. Nevertheless, LogME(F) cannot be written in fully vectorized form because the model parameters in LogME(F) are highly coupled. Hence, LogME(F) can only be excuted in a while loop.

In addition, in the classification task, we compare EMMS and LogME. EMMS usually has higher computation complexity because D 2 ≫ C . In some cases, when the number of categories C and the iteration number T are large, EMMS could be faster than LogME with vector computation. For example, we find that C = 397 and T = 4 . 46 on average over all models when LogME is convergent on the Sun397 dataset. It results in higher time complexity than LogME, as indicated in Table 23. We further verify this by implementing LogME with T = 1 . As shown in Table 24, EMMS spends more time in calculating the transferability than LogME (T=1) on all datasets. However, LogME performs much worse than EMMS because it does not converge when T = 1 .

Comparison with variants of existing methods. To further validate the efficacy of EMMS, we compare it with TransRate using F-Labels on image classification. To this end, we estimate the mutual information of the model feature and F-label following TransRate. Specifically, denote model feature X ∈ R N × D 1 , and the F-Label Z k ∈ R N × D 2 , we estimate the mutual information of X and Z k after the discretization operation for each dimension of D 2 separately and then take average to obtain the final score.

Moreover, we implement two baselines based on TransRate. When K = 1 , we instantiate the F-Label as the CLIP embedding. When K = 3 , we instantiate the F-Labels as the embedding collection extracted from the CLIP, BERT, and GPT-2. In this case, the final score is averaged over three F-Labels. The results are shown in Table 25, where we can see that our EMMS consistently outperforms TransRate with F-Labels (both K=1 and K=3).

Peer reviews:
comments: SUMMARY:
This paper proposed an efficient multi-task model selector (EMMS) to address the inapplicability in a multi-modal multi-task scenario. Specifically, the proposed method achieves a new state-of-the-art in performance and speedup through the incorporation of design elements such as the F-label, Weighted Linear Square Regression, Fast Computation by Alternating Minimization.

STRENGTHS:
1.	The motivation for addressing “a unified representation to represent diverse label formats” is clearly presented and validated. And use foundation model is an impressive way.
2.	The detailed derivation and experiment in this paper are comprehensive, providing strong evidence of the validity of the proposed model.

WEAKNESSES:
1.	It is recommended to include a complete demo in the code address of the paper, rather than a short python file.
2.	The phrase "5.13×, 6.29×, 3.59×, 6.19×, and 5.66× speedup in wall-clock time" appears to describe the speed efficiency of the EMMS (One) method. Could you clarify why EMMS's performance varies, being faster than LogME in some instances and slower in others, as indicated in Table 1?
3.	In Table 1, there seems to be some ambiguity about the effect of F-label on transferability assessment, since the perforamnce of the EMMS(one) and EMMS is identical in some rows.
4.	It is recommended that the paper include a broader array of foundational models for F-label. Is CLIP currently the model with the highest effects?

QUESTIONS:
na

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on an under-explored problem of estimating neural network transferring capability without actually fine-tuning the multi-modal multi-task model on individual downstream tasks. The problem is well-motivated and is of great practical importance. The solution proposed in this paper is straightforward, which essentially is to encode the target label (in the text form) with a foundation model. The embedding of the label text is then treated as the target and a model's transferability can be estimated through a simple weighted linear regression, which is then solved by an alternating minimization algorithm. The paper experiments with 5 downstream tasks and 24 datasets, and show that the proposed estimation method (EMMS) is fast and effective on most tasks.\n\nSTRENGTHS:\n1. The task is practically important and under-explored in previous work. As the authors mentioned, the previous estimation is usually limited by the classification tasks and the proposed work is more flexible as it directly encodes the label as a text sequence, therefore it can be used for multiple different types of tasks.\n2. The proposed method is intuitive and straightforward, and the algorithms for solving the problem are clear. The theoretical proof looks reasonable although I didn't carefully check all the details in the equations.\n3. The approximation with algorithm significantly speed up the computation when estimating the performance.\n4. This paper performs extensive experimentations by fine-tuning the expensive models on many downstream tasks (to obtain ground truth) and the proposed approach shows superior performance over other methods with regard to correlation and the wall clock speed.\n\nWEAKNESSES:\n1. The Figure 1 bottom is not very informative -- the radar chart is most effective for showing multivariate data with quantitative variables, but in this setting the variables are all binary (applicable/inapplicable), therefore it's actually carrying limited information. Probably replace the figure with a table of checkbox to illustrate the advantage of the proposed approach.\n2. As I mentioned in the limitations, some important assumptions are not validated in the paper. Specifically the key assumption that directly leads to the weighted linear square regression problem and alternating minimization algorithm is the linear mapping from model feature space to the F-label space. More justifications about the assumptions are needed.\n3. I'm also concerned about the choice of evaluation metric of weighted Kendall's $\\tau_w$. If I understand correctly, as a ranking metric it focus more on the relative orders of the elements in the rank, but ignores the actual number. In other words for A>B>C>D, it doesn't care whether A is just marginally higher than B or is much higher than B. It seems wasteful since we already obtain the ground truth downstream task model performance through expensive fine-tuning, but with the $\\tau_w$ metric only the relative orders of the ground truths are useful. Therefore I'm wondering if it's possible to directly estimate the actual performance. I might be wrong or missed some important considerations so any clarifications or rationales are appreciated.\n4. Some minor issues: in the tables with actual results, some numbers are bolded as the best performance, but in many cases there is a tie between two numbers in the table. Probably it's better to highlight both.\n\nQUESTIONS:\nI'm curious if there are other metrics that can be used to compare the proposed methods with other approaches, e.g. other than $\\tau_w$ and wall-clock time. For the former as I stated in the weaknesses section, it seems that it only cares about relative orders of candidate pre-trained models; for the latter the wall-clock time is easily affected by multiple factors (e.g. multi-threading, concurrent execution of other tasks in multi-core environments, etc.) I would like to learn more about the rationale for choosing these metrics.\nAlso it seems in this paper only the linear mapping is considered and it is used as an assumption without further justification. This may need more clarifications as one can easily come up with alternative approaches for estimate the mapping from features to label embeddings.\n\nLIMITATIONS:\nThis paper has a dedicated section to discuss the limitations and I agree with the authors that the quality of the foundation model (which is mainly responsible for the encoding of the label text) plays critical roles in the effectiveness. However I think there exists some other limitations which are also worth mentioning. Specifically, some important assumptions in this paper is in L164 and L168, where the label embedding is assumed as a linear mapping of the model feature. This assumption seems harsh and I think a more thorough discussion about the assumption will be useful. There is no obvious potential negative societal impact with this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes to utilize large-scale foundation models for efficient multi-task model selector (EMMS). Concretely, the authors utilize foundation model to transform different label format (category label, text, bounding boxes) into unified noisy label embeddings. EMMS then could measure the compatibility between the models’ features and corresponding
59 label embeddings via weighted linear regression. Experiments on 5 downstream tasks with 24 multi-modal tasks shows the proposed method's effectiveness and efficiency.

STRENGTHS:
- the motivation of this paper is important: investigating how to evaluate foundation model on a set of multi-modal tasks without finetuning all the target tasks.

- the writing is clear and easy to follow.

- the authors also provide code for reproducibility.

- The experiments show improvement on the 5 downstream tasks with 24 datasets.

WEAKNESSES:
As the authors mentioned in the limitation section, the proposed method is bottlenecked by the capabilities of the chosen foundation model. One could further ask, if the foundation model is good enough, why don't we just use the foundation model for the intended downstream tasks? For example, use CLIP for image classification?

QUESTIONS:
please refer to the weakness section

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EMMS, an efficient multi-task model selector for predicting the performance of pre-trained neural networks on multi-modal tasks without fine-tuning. EMMS employs large-scale foundation models to transform diverse label formats into a unified noisy label embedding. Through weighted linear regression and an alternating minimization algorithm, EMMS accurately estimates transferability. Experimental results demonstrate superior performance and significant speedup compared to existing methods.\n\nSTRENGTHS:\n1.\tGeneric Transferability Estimation Technique: The proposed method, Efficient Multi-task Model Selector (EMMS), offers a generic approach to estimate transferability. By utilizing a unified label embedding derived from foundation models and employing a simple weighted linear square regression (WLSR), EMMS becomes a fast and effective method for assessing the transferability of pre-trained models across different tasks.\n\n2.\tNovel Alternating Minimization Algorithm: The paper introduces a novel alternating minimization algorithm specifically designed to solve the WLSR problem. This algorithm ensures efficient and accurate estimation of transferability within the EMMS framework.\n\n3.\tThe authors did extensive experiments across different tasks and the results demonstrate the efficacy and effectiveness of the proposed method.\n\nWEAKNESSES:\n1.\tThe proposed method is complicated and difficult to follow. I can barely understand how to use WLSR to maximize the log-likelihood, which may bring extra complexity for re-producing the method.  \n2.\tOne of the simple baselines could be estimating the mutual information between the $\\hat{x}$ and $Z$. I doubt the effectiveness of this baseline, but it could be nice to have.  \n3.\tMore experiment details need to be provided like, what are the foundation models used in different tasks. Another ablation could be studying the effectiveness of different foundation models. I guess CLIP could be the strongest model to use in those tasks if we just use one single foundation model.  \n\nMinor:  \nLine 111, `denote finetuning score`.  \nPlease provide more description of Figure 3 (b) as it is not straightforward to understand the confusion matrix of image captions.\n\nQUESTIONS:\nPlease address the comments in weakness section.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

comments: SUMMARY:
This paper introduces the Efficient Multi-task Model Selector, which utilizes foundation models to convert diverse labels into unified label embeddings. These embeddings are then used to calculate a transferability metric within a weighted linear square regression (WLSR) framework. The proposed method achieves impressive results in multi-task scenarios.

STRENGTHS:
1. The paper presents a novel and unified strategy for model selection. The use of label embeddings enables the capturing of fine-grained semantics, leading to improved estimation in downstream tasks.

2. The effectiveness of the proposed method is supported by a comprehensive set of experiments covering various downstream tasks.

3. The derivation of equations is provided, establishing a solid foundation for the main contribution of the paper.

WEAKNESSES:
No questions.

QUESTIONS:
No questions.

LIMITATIONS:
No questions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-18