Foundation Model is Efficient Multimodal Multitask Model Selector
Authors: Fanqing Meng
Wenqi Shao
zhanglin peng
Chonghe Jiang
Kaipeng Zhang
Yu Qiao
Ping Luo
Venue: NeurIPS 2023 poster (2023)
Keywords: transfer learning
model selection
foundation model

Abstract: This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering.A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metrics to measure models’ transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable in a multi-modal multi-task scenario. To tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models to transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into a unified noisy label embedding. EMMS can estimate a model’s transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability of pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the state- of-the-art method LogME enhanced by our label embeddings, EMMS achieves 9.0%, 26.3%, 20.1%, 54.8%, 12.2% performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5.13×, 6.29×, 3.59×, 6.19×, and 5.66× speedup in wall-clock time, respectively. The code is available at https://github.com/OpenGVLab/Multitask-Model-Selector.

Foundation Model is Efficient Multimodal Multitask Model Selector
Fanqing Meng
Wenqi Shao OpenGVLab, Shanghai AI Laboratory shaowenqi@pjlab. org cn
OpenGVLab, Shanghai AI Laboratory
Shanghai Jiao Tong University mengf mannfaaggin33@@maill
Zhanglin Peng The University of Hong Kong
Chonghe Jiang
The Chinese University of Hong Kong
Kaipeng Zhang
Yu Qiao
OpenGVLab, Shanghai Al Laboratory
OpenGVLaa, Shanghai Al Laboratory
Ping Luo
The University of Hong Kong OpenGVLab, Shanghai AI Laboratory pluo@cs hku edu
Abstract
This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering.A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metries measure models' transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into unified noisy label embedding. EMMS can estimate model's transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the state- of-the-an art method LogME enhanced by our label embeddings, EMMS achieves 9.0%, 26.3%, 20.1% 1.. 54.8%. 12.2% performance gain image recognition, referring, captioning. visual question answering, and text question answering, while bringing 5.13x, 6.29x, 3.59 6.19x, and 5.66x speedup in wall-clock time, respectively. The code available at /ttttp/s//ggtuuu..oo////nn..t/ad Maaauuititsl/////aiiiass. Selet-Sel
multi-modal multi-task scenario. To
Corresponding Authors: shaowenqi @ww.apppraaaa........mmmmm pluo@ saluoccs..knn hk.k..dduu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

O
dasshcation
bnagle
Prir Model Selectors
O
regression
<2,0, 488, 2075
caption
nat and
copared I's text
Model Selector
refariing
sayp
trained Model
bounding Label Format
Model Feature
multi multi-task task scenarios
EMMS
NLEEP
1.0
Loghi
PACTran
Weighted Kemdall's F
0.8
0,77
0.67
TransRate
(Onc)
0.61
(b)
0.56
0.47
0,51
0.4
8.40
Imsage Chassi ChasiBortin
Im-gga Capdioning
Vissal Ouestion Amwering
Text Qsettio Anverring
Reteriigg
Exp Compre Compretirntoo ririoo
Figure I: Comparison between prior me-trained model selectors and our multi-task model selector. (a) denotes that a model selector mcasures transferability by modeling the compatibility between the model feature and task label. Previous model selectors only receive labels with one-hor or real-valued vectors. Our multi-task model selector be employed in various tasks with diverse label formats. (b) denotes that our proposed EMMS applicable and effective various downstream tasks while previous transferability metrics can be only used classification or regression tasks.
1 Introduction
Pre-trained models (such as neural network backbones) are crucial and are capable of being fine-tuned to solve many downstream tasks such as image classification [1. image captioning (2). question answering [3], and referring segmentation [4]. This "pre-training > fine-tuning" paradigm shows various datasets (e.g., ImageNet Maad YFCC1001 (5)) by many objectives (e.g., supervised and self-supervised) can provide generic-purpose representation, which transferable to different tasks. A large number of pre-trained models have been produced with the rapid development of network architecture research, such as convolutional neural networks (CNNs) 66, 7 X and transformers ( 10 [1]. When given a large collection of pre- pre-trained models to solve multiple multi-modia tasks, an open question arises: how to efficiently predict these models' performance on multiple lasks without fine-tuning them?
that the models pre-trained
answered the above question using model selection approaches, which are of great benefit to transfer learning. For example, when neural network is properly initialized with a better pre-trained checkpoint, will achieve faster convergence and better performance target task [20 21]. However, it is challenging to quickly identify optimal one from a large collection of pre-trained models when solving each multi-modal task. This is because of two reasons. Firstly, the ground truth of model ranking only be obtained by brute-force fine-tuning and hyper-paramen parameter grid search, which are computationally expensive [22]] Secondly, the recent methods [12 B [4] 23] that can estimate the transferability of pre-trained models not
Existing works [12 13 15
16
17
18 [9]
generic enough for a variety of multi-modal tasks. For instance, an approach [14] 12 that relies on the prior knowledge of single specific task would be ineffective in others.
To address the above challenges, we need unified representation to represent diverse label formats in each multi-modal task e.g., categories, texts and bounding boxes. Existing methods cannot be employed in multi-task scenarios because they only receive labels with one-hot& or real-valued vectors, as shown in Fig M. For For example, LEEP [2] and PACTran 24 are carefully designed for classification tasks. GBC [25 and TransRate 23 measure transferability using class separability, which relies on prior knowledge in classification task. Although LogME [33 can be used in both classification and regression tasks, relies on real-valued labels, making it inapplicable in other label formats such as text descriptions.
In contrast to the above works, we propose an Efficient Multi-task Model Selector with an acronym, EMMS, which can select the most appropriate pre-trained model for solving each multi-modal task.

---

This is achieved by employing foundation models, such as CLIP (26 and GPT-2 227. to transform diverse label formats into unified label embedding space. The estimated label embedding contains more rich information than the conventional one-hot and real-valued label encoding.
In this way, EMMS measure the compatibility between the models' features and corresponding label embeddings on various tasks, as shown in Fig. aad Fig. 2 This results in more generic assessment of the models' transferability than previous methods. Specifically, EMMS treats the estimated label embeddings as noisy oracles of the ground-truth labels, and it turns a log-likelihood maximization problem into simple weighted linear square regression (WLSR). We propose an alter- nating minimization algorithm to solve WLSR, which can be solved with theoretical convergence guarantee efficiently. Extensive experiments validate the effectiveness of EMMS on multiple tasks, including image classification []] image captioning 2) question answering on both image (28] and
text [29] referring comprehension
and landmark detection 30
The contributions of this work are summarized as follows. (1) We propose a generic transferabilityy estimation technique, namely Efficient Multi-task Model Selector (EMMS). Equipped with a unitied label embedding provided by foundation models and simple weighted linear square regression (WLSR), EMMS can be fast, effective, and generic enough assess the transferability of pre-trained models in various tasks. (2) We propose novel alternating minimization algorithm to solve WLSR efficiently with theoretical analysis. (3) Extensive experiments on 5 downstream tasks with 24 datasets demonstrate the effectiveness of EMMS. Specifically, EMMS achieves 9.0%, 26.3%, 20.1 20.1%, 54.8%, 12.2%. performance gain on image recognition, referring, captioning, visual question answering. and text question answering, while bringing 5. 1,13x, 6.29x, 3.59x, 6.19x, and 5.66x speedup in wall-clock time compared with the state-of-the-arr method LogME, enhanced by our label embeddings, respectively.
2 Related Work
Transferability Estimation. Model selection is an important task in transfer learning. To perform model selection efficiently. methods based on designing transferability metrics have been extensively investigated. LEEP [12] pioneers to evaluate the transfcrability of source models by empirically estimating the joint distribution of pseudo-source labels and the target labels. But it can only handle classification tasks with supervised pre-trained models because the modeling of LEEP relies on the classifier of source models. Recent works propose several improvements over LEEP to overcome the limitation. For example, NLEEP 31 replaces pseudo- source labels with clustering indexes. Moreover, LogME [13], TransRate 23 and PACTran 24 directly measure the compatibility between model features and task labels. Although fast, these metrics can only be used on limited tasks such as classification and regression. This work deals with model selection in multi-task scenarios. We propose EMMS to evaluate the transferability of pre-trained models on various tasks.
Label Embedding. Label embedding represents a feature vector of task labels, which can be generated in various ways. The classical approach is to use one-hot encoding to represent the labels as sparse vectors, which is widely used in image classification. Another way is to transform labels into vectors by embedding layers. For example, an RNN module is employed to generate label representation in [32, which is encouraged to be compatible with input data vectors in text classification tasks. In addition, it is also common to treat the labels as words and use techniques such as word2vec [33 or Glo [44] to learn vector representations of the labels. The main obstacle in the multi-task scenario is how to deal with diverse label formats. In this work, we follow the idea of word embedding and treat task labels as texts, which are then transformed into embeddings by publicly available foundation models [26 (7].
Foundation Models. CLIP (26] is the first known foundation model which learns good semantic matching between image and text. The text encoder of CLIP can perform zero-shot label prediction because it encodes rich text concepts of various image objects. By tokenizing multi-modal inputs into homogeneous tokens, recent work on foundation models such as OFA [35] and Uni-Uercee Uni-Perceiver (36) use a single encoder to learn multi-modaa representations. In this work, we utilize the great capacity of foundation models in representing image-text concepts to generate label embedding. is noteworthy that although foundation models can achieve good performance in various downstream tasks, they may not achieve good zero-shot performance on many tasks 37
38 33) and it still computationally On the contrary, multi-takk model
expensive to transfer a large model to the target task 40
3

---

selector can quickly select an optimal moderate- size pre-trained model that can generalize well in target tasks. In this sense, a multi-task model selector is complementary to foundation models.
3 Preliminary of Model Selection
Problem Setup. A target dataset with N labeled samples denoted as 1 ((("","") y"" and M pre-trained models (thm = (0.: Nm ) m= are given. Each model thn consists of a feature extractor Om producing D-dimension feature (i.e. ã 0m (a: E R
RRP) and task head hm outputting predicted label given imput x [ 99. In multi-task scenarios, the ground-truth label comes in various forms, such category, caption, and bounding box, shown in [ The task of pre-trained model selection to generate a score for each pre-trained model thereby the best model can be identified to achieve good performance for various downstream tasks.
Ground Truth. The ground truth obtained by fine-tuning all pre-trained models with hyper- parameters sweep on the target training dataset and recording the highest scores of evaluation metrics denote ine-tuning scores of different models as {Gm m=1- Since fine-tuning all models on all target tasks requires massive computation cost, research approaches design lightweight transferability metrics which offer an accurate estimate of how well pre-trained model will transfer to the target tasks.
31 (e.g. test accuracy and BLEU4 (42))
M
Transferability Metric. For each pre-trained model Oms transferability metric outputs a scalar score Tm based on the log-likelihood, as written by
N
Tm
log p(ym (zin Om, 1mm)
where (In, yn) denotes the n-th data point in target dataset T. A higher log-likelihodd value for Tm indicates that the model 6m is likely achieve better performance the intended task. Numerous transferability metries have been proposed by modeling prediction probability p(y, z 6n. hm) in various ways. Although being efficient, they can hardly be used in multi-task scenarios.
Challenges in Multi-task Scenarios. Existing transferability metries fail to generalize to various tasks for two reasons. Firstly, existing methods such as LEEP and LogME can only deal with real- value label formats. But y/ can be a sentence of words the task of image caption. Secondly, a large number of the previous metrics estimate transferability through the target task's prior information such as maximizing inter-class separability, which is inapplicable in multi-task scenarios except for the classification. To overcome these difficulties, we introduce simple regression framework with unified label embeddings provided by several foundation models in See 4
Our Method
In this section, we introduce Efficient Multi-task Model Selector (EMMS). To overcome the difficulty of diverse label formats, EMMS employs foundation models to transform various labels into unified label embeddings in 4. By treating label embeddings provided by multiple foundation models as noisy oracles of ground truth labels, EMMS can calculate transferability metric under a simple weighted linear square regression (WLSR) framework in See 4.2 We design an alternating
minimization algorithm to solve WLSR efficiently in Sec. 4.3 The illustration of our EMMS is provided in Fig. 2
4.1 Foundation Models Unify Label Embedding
In general, label embeddings or label representations should encode the semantic information such that two labels with low semantic similarity have a low chance be grouped. A common scheme is to represent label embedding as a one-hot vector. However, one-hot representation can not embed labels with text formats such as captions in the image caption task. Following the design in multi multi-modality foundation models [42], we treat labels with diverse formats as a text sequence, which can be encoded by pre-trained foundation models, shown in Fig. 2]
Label Embedding via Foundation Models (F-Label). Thanks to the great representational capacity, the foundation model can construct label embedding (termed F-label) while preserving its rich

---

clasifiation
capiies
F1
21
Fx
Foundatio
Midel
contiee
referring
Viaua goedios anonaring
F2
22
Label Emiheddinng
Neive
Fi
2K
OE
Strougth
vid
Ladkig godda
(g) Examples of test bibels varieus casks
(b) Graph model of regresion with multiple noley labels
Figure 2: Overview of our EMMS. (a) shows that labels in various tasks can be expressed by texts. (b) presents the graph model of regression with multiple noisy labels. We use several foundation models to encode text labels as label embeddings which are deemed as noisy oracles of true label embedding Moreover,
linear mapping of model feature & with Gaussian noise [N, o
papk Tryln rquia Bynn
sair russku (hroad) weedlasd
Child
uee trees
label embeddligg
one-hot
Iabel canbedding
one-hot
Example label embedding image classification
(b) Example embeddding imaage caption
Figure 3: Label embedding has richer semantic information than one-hot labels. (a) indicates that in the classification task, F-Label can capture the correlation of labels with different granularity than one-hot encoding. (b) shows that in the image caption task, F-label can model the semantic relevance of two captions corresponding the same image better than the one-hot label.
semantic information of labels. Given label y the target task, the label embedding R- is obtained by = ((vy)/14<(y)) where F' can be instantiated by various foundation models to process diverse label formats. & normalization utilized to normalize the representations extracted from different foundation models. Moreover F can be implemented as CLIP [26), BERT [3] and GPT-2 [7 when task label y text. Note that label embedding extraction can fast enough with GPU parallel computation. We provide the runtime analysis in Appendix Sec.C.
one-hot label representations. Firstly, embeds richer semantic information than one-hot label, leading to accurate modeling of the semantic relationships between different labels. As shown in Fig F-Label leads to a higher correlation between fine-grained classes than one-hot encoding. Secondly, compared with one-hot labels, F-label can be obtained variety of tasks as long as the task label can transformed into sequence. With the assistance of F-Labels, model selection can be established in multi-task scenarios.
Benefits of F-Label. F-Label has several advantages
4.2 Regression with Unified Noisy Label Embeddings
To estimate the transfcrability of pre-trained models, the relationship between model features & R and F-Label E R² should be modeled in order to calculate the transferability score T.m in Eqn. (T) On the other hand, since semantic label y can be embedded by several foundation models, the label embedding set can be constructed as E = {* (1) Fa (2, [K]] where {
denotes K foundation models. Now, we utilize data points { { 21 ... " W to model the relationship between model features and F-Labels.
Setup. As shown in Fig 2 we assume that true label embedding z is linear mapping the model feature with additive Gaussian noise with variance of oZ. given by and 6 ~ NO, Iz) where 2n = wTTh I the regression prediction, w E IR Dx and c are regression weights and regression error, respectively, and In is L-by-L identity matrix.
+ w +
assume that F-labels pendently provide noisy estimates of the true label embedding Formally, we have P(za = N22, TT.) ). By the above setup, EMMS would performed with noisy labels. Hence, EMMS tends to select tre-trained models robust the label noise.
obtained from different foundation models are oracles that inde-
5

---

We provide Reasonableness of the Linear Assumption. Specifically, EMMS assumes that the true label embedding is a linear mapping the model feature with Gaussian noise. The linear assumption is reasonable in image and text classification tasks because a linear classifier is usually used when the pre-trained model is transferred to a target task, which is commonly used in recent methods. For example, LogME [3] assumes that: 2 N(w 2, 8 _-1), which implies that there is linear mapping from the model feature space to the label space. PACTran [24) also has a similar setting. The difference is that LogME takes one-hot label as the true label embedding, which limits its applicability. But our EMMS treat the true label embedding 2 as an implicit variable. And F-Labels {=* obtained from different foundation models assumed to be noisy oracles of true label embedding 2. Since labels many tasks can be easily encoded into F-Lablels, our EMMS can be used as multitask model selector. We verify effectiveness the linear assumption in various multi-model tasks with extensive experiments Sec
Computation of Log-I iikelihood. To model the relationship between model features and F-Labels, we need to estimate regression weights w, strengths of label noises {ax ] For simplicity of notation, we consider the case L 1, ie. F-labels are scalars. Given N data points, the log- likelihood given by
N L= Nlog A log A2
(Ag)
Σ
AT) const
2
4A2
where A1 I 1/0x, A =E 1/20 Ek / and Ex (x2)2/ot. The detailed
derivation Eqn.(2) is provided the Appendix Sec.A.
Maximizing Log-likelihood as Weighted Linear Square Regression (WLSR). The remaining issue is to determine parameters w and {ax 1K by maximizing the log-likelihood in Eqn. ( But can be intractable because we and J are heavily coupled. To mitigate this issue, turn the log-likelihood maximization into weighted linear square regression by rearranging Eqn. (2) as -L =|| Ztllă + [a ) where E RN is the data matrix whose n-th row is model feature ( w Dx11 are weight parameters. Z E NxK is F-Label matrix whose k-th column k satisfies that kttt = 1, 2 0 which is (K - 1)-D
is the label embedding 2g. and simplex denoted as A AK11, R(-) is regularization term parameterized with {ax {o the derivations in Appendix Sec.A.
We note that the computational intractability comes from the data-dependent regularizer R(-). For efficient computation, drop R(.), turning the likeliiid maximization into problem of WLSR, as given by
1 | s(w,t) x(w,))=1lxx
min WERDXI,
Ztll
(3)
When considering the case L > 1, Eqn.
3 becomes EE 11l Zt where Z E N=LK and ||-|| is Frobenius norm. From Eqn. and Eqn. s(w, 8) is an approximation
Algorithm 1 Alternating Minimization
Algorithm Fast Alternating Minimization
I: Input: Model feature X E RND,; F- Label matrix Z E N Learning step- sizes 1/ and B for and I, respectively;
1: Input: Model feature X E RNDD F.
Label matrix RNK,,
2: Output: Score of WLSR;
2: Output: Score of WLSR:
3: Initialize 1 1 к and w = = Ip:
3: Initialize f = K 1 and 1 1p
4: while not converge do
4: while s not converge do
5: 8 |1 w Zt [3:
5: 1 X Ztll::
6: w = (XT '" Zt;
LSR for w |/ LSR for t Projection
6: < 7 7 Zt);
7: t (ZTZ) - Xw;
7: while / not converge do
8: f Sparsemax(()
8:
i. 1 37 (Zt Xw);
9: end while
9:
1
1= IT K- (t);
Projection
10: Return:
end while
11: end while
Return:
of negative log-likelihood. Hence, smaller s(w, 1) indicate the larger Tm in Eqn. ter transferability. We design an efficient algorithm to solve WLSR.
1 and bet-

---

4.3 Fast Computation by Alternating Minimization
Algorithm. The optimization problem in Eqn. can be formulated to classical second-order conic program [2] [4 (simply called SOCP). However, the excessive data in problem leads to large dimension of the variable, making it inefficient for standard solvers. Therefore, we are motivated to find the smooth structure of the problem and design alternating minimization algorithm to achieve fast computation. As shown in Algorithm[[] [ we separately fix w and f to optimize the other one until the function value in Eqn. (3) converges. Specifically, when we fix t, the whole problem degenerates to least square problem with respect to w. When we fix w, we also need to solve a least square problem concerning under the simplex constraint.
Convergence Analysis. We will prove the convergence property of the function value. Indeed, we prove stronger condition that the function value decreases after each round of iterations and t. From the monotone convergence theorem, the convergence can thus be derived. We first present the decreasing result of inner loop of by Theorem and the same property holds for the update of The detailed proofs are
Then the convergence of the whole algorithm can be derived by Theorem 2 placed in the Appendix Sec.A.
Theorem 1. Suppose s(w,t) 111 Ztll? where X RNXD, Z RNNK E RDx1 and E A-1 the inner loop of Algorithm 7 lines decreases after each iteration. Specifically, denote s(w, tt) -(w,t)
2/ ||| 1/ Z|| and 1t AT (Vsw,,)). For any L E AK-1
+1 < O.
Theorem 2. Suppose s(w, - |X Ztl2 where X ENNDD Z E RNNK w RD11 and K-1, the function value in Algorithm 7 will converge. Specifically, denote w², w",t* as the result after one iteration of w, respectively, we have 0 S s(w",
s(w", .() s(w,l).
Computational Speedup. Although this algorithm D guarantees convergence, it is bit time- consuming due the two-level loop, we optimized this part and achieved similar results in very little time. Since the least squares solution is extremely fast, we performs least squares on w and t. and then replace projection onto simplex with explicit Sparsemax transformation 45 46), iteratively. The fast solver is illustrated in Algorithm{] p. we experimentally verify its convergence and find that the approach achieves impressive speedup.
5
Experiment
This section evaluates our method EMMS different downstream tasks, including image classifi- cation, image caption, visual question answering, text question answering and referring expression comprehension. We put more experiments details in Appendix Sec.B. Morcover, conduct detailed ablation study to analyze our EMMS in Appendix Sec.C
5.1 Training Details
Benchmark. For image classification, We adopt 11 classification benchmarks including
FGVC Aircraft [47]. Caltech-101 48 Stanford Cars 44), CIFAR-10 [50), CIFAR-100
[50].
DTD (5N. Oxford 102 Flowers
52 Food-101 [53]. Oxford-IIIT Pets [54 SUN397 [55],
and VOC2007 [56]. For image caption, We use Flickr8k (57] Flickr30k (58) FlickrStyle
Humor [59]. FlickrStyle 10kyyleeeKKRRoaniie [59] and RSICD (60] For visual question answer, We
apply COCOQA 61, DAQUAR (22 and CLEVR 63 For text question answer and referring
expression comprehension, we separately use SQUADI.1 64 .SQuAD2.0 665 and RefCOCO (66),
RefCOCO+ (66]. RefCOCOg [67]
Ground truth. In order to obtain the ground truth, finetune all pre-trained models all target datasets with a grid search of paper-parameters. Details of target datasets and fine-tuning schemes described in Appendix Sec.B.
Evaluation protocol. To assess how well a model selector predict the transferability of pre-trained models, we calculate the rank correlation between (T.. J.M and (Gm J-M Following the common practice [3 3T) we use weighted Kendall's Tw. The larger Tu indicates better correlation and better transfcrability metric. For computation complexity, record the runtime of executing algorithm
7

---

Table I:
Comparison of different transferability metrics on ViT models regarding Tw and the wall- clock time where EMMS(One) denotes EMMS with the one-hot label. Our proposed EMMS achieves the best transfer-ability assessment over 11 target tasks and exhibits higher efficiency than NLEEP.
Aireraft Caltech
Cars
CF-10
CF-100
DID
Flowers
Food
Pets
SUN
VOC
Avg.
Weighted Kendall's tau Ti
LogME
0.299
0.382
0.633
0.741
0.727
0.569
0.512
0,580
0.528
0.619
0.591
0.561
NLEEP
-0.282
0.027
0.693
0.674
0.538
0.123
-0.262
0.105
0.40
0.268
0.109
0,218
TransRate
0.244
0.412
0.487
0.260
0.702
0.533
0.655
0,542
0.707
0.612
0.651
0.527
EMMS(One)
0.412
0.444
0.565
0.740
0.736
0.621
0.562
0,579
0.740
0,592
0.730
0.611
EMMS
0.481
0.444
0.706
0.718
0.745
0.621
0.562
0.673
0.740
0.619
0.730
0.639
Wall-Clock Time (6)
LogME
8.93
10.89
30.28
53.07
62.13
4.78
9.27
104.92
6.28
425.43
7.42
65.76
NLEEP
553.7
716.8
1.1e3
8.0e3
1.2e4
183.7
819.2
3.4e4
256.4
2.7e4
288.3
7719.8
TransRate
19.43
19.21
36,9
61.73
63.82
8.73
18.26
110.79
15.51
89.92
5.11
40.85
EMMS(One)
4.12
4.45
8.07
19.45
26.18
2.65
4.03
39.72
3,50
24.84
4.07
12.82
EMMS
21.31
17.23
28.06
154.61
182.11
13.87
15.95
265.99
17.93
63.86
16.63
72.55
over all models given the feature and label on a target task and analyzed the computational complexity of EMMS well LogME. (Details can be found in Appendix Sec.C)
Baseline. For the image classification task, we choose NLEEP [31], TransRate [23] and LogME as the baseline; for other multimodal tasks, we choose LogME with F-Label as the baseline; in addition, for the VQA task, we additionally compare PACTran 24
Details of baselines and why we
choose them are described in Appendix Sec.B.
5.2 Image Classification with VIT Models
Vision transformer ( (ViT) models have been increasingly used for a variety of tasks and have achieved better results than CNN models. The architecture of ViT models are more complex than CNN models. Hence, how to do the model selection on ViT models is a more challenging and rewarding task. Details of pre-trained models are described in Appendix Sec.B.
Performance and wall-clock time comparison. As shown in Table our EMMS achieve the best average Tu on 11 target datasets and the best Ta on 9 target datasets with relatively short time. For example, EMMS outperforms LogME by 0.182 and 0.139 rank correlation Tu on Aircraft, and VOC2007, respectively, showing the effectiveness of our EMMS in measuring the transfer-ability of pre-trained VIT models. On the other hand, for the remaining 2 target datasets (i.e. CF-10, DTD), our EMMS still has marginal gap compared to the best-performing transferability metric. Besides, we find that the effect of model selection of EMMS ViT models selection has an improvement compared to CNN models selection, we guess F-Label has spatial similarity with the model feature of ViT-base model because the foundation models are mostly transformer- based, which can model the relationship between model feature from Vit-base models and F-Labels more accurately.
5.3 Image Captioning
Here we treat image caption as a vocab-based classification task. That we use a vocabulary and classify the caption into the index of some words in the vocabulary. Afterward, training is done according to the classification task criteria .Here we calculate the average Tu and time of LogME with K single F-label from K foundation models we use respectively. We wants to select the best combination of image encoder and language encoder. Details of pre-trained models and the model architecture are described in Appendix Sec.B.
Performance and wall-clock time comparison. As shown in Table 2 EMMS is significantly ahead of baseline in both time and effect for each dataset. For example, EMMS outperforms LogME with the relative improvements of 39% and 37% in rank correlation Tu on Flickr8k and Flickr30k, respectively. In addition, the time of EMMS is reduced by 83.7% and 79.8% relative to LogME on these two datasets, which shows the efficiency of our algorithm. The average rank correlation T. alone the five datasets is 0.64, which denotes EMMS has sufficient confidence.
8

---

Table Comparison of different transferability metrics on image caption models in rank correlation Th with the ground truth and the wall-clock time. The LogME denotes using LogME with F-Label. Our proposed EMMS achieves the best transfer-ability assessment on each target task with much less time compared to LogME.
F8k
F30k
RSD FIOk-H F10k-R
F8k
F30k
RSD
F10k-H F1Ok-R
Weighted Kendall's tau
ClalkClock Time (s)
LogME
0.483
0,368
0.501
0.780
0.654
425.67
1594.16
973.22
60.35
63.79
EMMS 0.660
0.504
0.704
0.802
0.678
69.01
321.32
88.77
16,56
14.59
5.4 Text Question Answering
For natural language understanding, we consider Text Question Answering (TQA) as a reading comprchension task, where the response to each question is text segment extracted directly from the affiliated reading passage, or the question may indeed be deemed unanswerable. Details pre-trained models and how to finetune are described in Appendix Sec.B.
Performance and wall-clock time comparison. In Table +] the performance improvement of EMMS on the TQA is consistent with the enhancements observed in the earlier mentioned computer vision tasks. More specifically. our EMMS attains accuracies of 60.3% and 46.3% on the Stanford Question Answering Dataset (SQuAD) versions 1.1 and 2.0 respectively, using rank correlation Tw as an evaluation metric. This represents a significant relative increment of 11.2% and 13.2% compared to the performance of LogME.
Table 3: Comparison of different transferability Table 4: Comparison of different transferability
metrics on TQA models in rank correlation Tu metries on referring expression models in rank
with the ground truth and the wall-clock time. correlation Tw with ground truth and the time.
The LogME denotes using LogME with F-Label. The LogME denotes using LogME with F-Label.
SQaI.1
SQu2.0
SQul.1
SQu2.0
Ref
Ref+
Refg
Ref
Ref+
Refg
Weighted Kendall's tau 7u
Wall-Clock Time (8)
Weighted Kendall's s tau tu
Wall-Clock Time (6)
LogME EMMS
0.409
3587.22
3596.23
LogME 0.425
0.389
0.398
2457.87
2478.90
2298.76
0.603
0.463
571.23
589.78
EMMS 0.458
0.549
0.521
454.26
467.92
356.94
5.5 Referring Expression Comprehension
Referring expression comprehension (REC) is widely challenging task because it requires precise alignment between linguistic concepts and image features. To address this, the objects in each image represented sequence of discrete tokens, while their bounding box corner coordinates turned into integer location tokens. This allows for unified F-Label to be extracted using various language models. More details about the pre-trained models can be found in Appendix Sec.B.
Performance and wall-clock time comparison. As shown in Table 4 our EMMS continues to exhibit its superiority in the enhancement of performance the REC task, an instance-leve cross- modal localization task. Specifically, the proposed EMMS produces accuracies of 45.8%, 54.9%, and 52.1% the RefCOCO, RefCOCO+, and RefCOCOg datasets respectively. This significantly surpasses its counterpart, LogME, in terms of margins when evaluated with rank correlation Twr-
5.6 Ablation Analysis
Comparison with different number of F-Label Here denote the number of F-Label is K and choose the image caption task to illustrate the impact of K on our solution. As shown in Table We find that increasing K in certain range brings gain in effectiveness to our method, but when K becomes larger, the time also increases and we find that K = is not as effective as K = 3. We believe that the increase in K brings difficulties in fitting the true Label, resulting in a loss of effectiveness. Therefore, we use K 3 for the sake of effect and time.
Performance on F-Label using small model On the one hand, using foundation model can extract the joint embedding compared to the small model, which allows EMMS to be extended tasks with multiple forms of labels. On the other hand, the foundation model can handle many types of tasks,

---

Table EMMS under different number of F-Label transferability assessment on image caption task. The improvement of K in certain range brought an increase in rank correlation Tw.-
F8k
F30k
RSD
F10k-H FIOK-R
F8k
F30k
RSD
F10k-H Flok-R
Weighted Kendall's tau Tw
Weighted Kendall's tau Tw
K=1
0.490
0.386
0.527
0.772
0.668
K=2
0.574
0.454
0.553
0.762
0.646
3=3
0.660
0.504
0.704
0.802
0.678
K=4
0.660
0.504
0.704
0.802
0.644
so we can use the foundation model for different tasks for label embedding. As shown in Table experimentally demonstrate that the use of the foundation model leads more accurate F-Label extraction and thus to an improvement in the performance of the method.
The effect of using single foundation model We investigate how EMMS is influenced when only a single foundation model is provided. We conduct experiments on image classification and image captioning. We consider EMMS with the single foundation model including language foundation
(3) RoBerta [68], and multimodal foundation model (4) CLIP 26 (5) FLAVA [69]. and (6) AltCLIP 70]. For comparison,
model (1) GPT-2 (27]. (2) BERT 43
include the result of our EMMS with default setting (K=3, i.e. CLIP, BERT, and GPT-2) and the result of previous state-of-
the-art methods obtained from LogME, NLEEP and TransRate. The results are reported in Table 20
and T
We have several observations. (1) Different downstream tasks prefer F-Labels obtained from different foundation models. No single foundation model dominant in all target tasks. In particular, CL.PP not the best model for extracting F-Labels. (2) For image captioning, multimodal foundation models are more appropriate for extracting F-Labels than language foundation models. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.
Table 5: The effect of the single foundation model EMMS. The results are obtained on image captioning regarding Tw.
F8k
F30k
RSD
FIUKH
FIOKR
Avg
SOTA/AII
Weighted Kendall's 's tau Tu
LogME(Clip)
0.530
0.393
0.618
0.764
0.634
0.588
0/5
(I) Gpt2
0.566
0.393
0.431
0.715
0.618
0.545
0/5
(2) Bert
0.395
0.319
0.448
0,802
0.711
0.535
2/5
(3) RoBerta
0.346
0.111
0.587
0.571
0.566
0.436
0/5
(4) CLIPs
0.453
0.393
0.704
0.802
0.634
0.533
2/5
(5) CLIP, p
0.510
0.448
0.704
0.802
0.678
0.628
2/5
(6) FLAVA
0.463
0.382
0.693
0.704
0.678
0.584
0/5
(7) ARCLIP
0.453
0.448
0.623
0.802
0.678
0.601
1/5
EMMS
0.660
0.504
0.704
0.802
0.678
0.670
4/5
6 Conclusion
How to select a pre-trained model for different tasks quickly and effectively is an important issue in the field of transfer learning. This paper proposes an efficient multi-task model Selector(EMMSS) that can be applied to many types of tasks. EMMS uses foundation model for Label embedding in order transform diverse label formats of different tasks into the same form and see them as noisy labels. To estimate a model's 's transfcrability, EMMS model this problem as a simple weighted linear regression, which can be solved use an alternating minimization algorithm. Compared with existing methods, EMMS achieves the first model selection in mulli-task scenarios, including image caption, referring segmentation, etc., with high speed and great results. For the limitations of the method, if the foundation model generalize very poor on downstream tasks, it may lead to low-quality label embedding, which is drawback our method. Moreover, building a holistic benchmark of various label embeddings would be useful in many applications such as multi-modal adaptation [71.. We leave it as a future work.
7 Acknowledgments
This paper is partially supported by the National Key RD Program of China No. No.20222DD061000 and the General Research Fund of Hong
National Key RD Program of
10

---

Kong 1720622. Besides, thanks Ruimao Zhang from CUHK(SZ) for thoughtful discussion and Prof. Anthony Man-Cho So from CUHK for his valuable discusion about solving the WLSR in the paper. At last, this work was done during the internship at Shanghai Artificial Intelligence Laboratory.

---

References
[1] Alex Krizhevsky, nya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[2] Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632 2014.
[3] Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International conference on machine learning, pages 2397-2400 PMLR, 2016.
[4] Zhao Yang. Jiaqi Wang. Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15155-1815,, 2022.
[5] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-7,, 2016.
[6] Kaiming Hc, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[7] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[8] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint Ntt::2010.119299 2020.
[10] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[11] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference computer vision, pages 568-578, 2021.
[12] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure evaluate transfcrability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR, 2020.
[13] Kaichao You, Yong Liu, Jianmin Wang. and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning. pages 12133-12143, PMLR, 2021.
[14] Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability a self-challenging fisher space. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 286-302. Springer, 2022.
[15] Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre- trained models are from neural collapse on the target dataset informs their transferability.. In Proceedings of the IEFF/CVE International Conference on Computer Vision, pages 5549-5558, 2023.
12

---

[16] Xiaotong Li, Zixuan Hu, Yixiao Ge, Ying Shan, and Ling- Yu Duan. Exploring model trans- ferability through the lens of potential energy. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5429-5438, 2023.
[17] Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etran: Energy-based transferability estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 18613-18622, 2023.
[18] Yi-Kai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. arXiv preprint 0rXiv:2306.03900 2023.
[19] Zhigang Hu, Yuhang Huang, Hao Zheng, Meiguang Zheng, and JianJun Liu. Graph-based fine-grained model selection for multi-source domain. Pattern Analysis and Applications, pages 1-12, 2023.
[20] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
[21] Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4918-4927, 2019.
[22] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3712-3722, 2018
[23] Long-Kai Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In International Conference on Machine Learning, pages 9201-9225. PMLR, 2022.
[24] Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac- bayesian metrics for estimating the transferability of pretrained models to classification tasks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 252-268. Springer, 2022.
[25] Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Trans- ferability estimation using bhattacharyya class separability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17229988,, 2022.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748-87688 PMLR, 2021.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[28] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitmick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-244222222 2233,, 2015.
[29] Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. Quac: Question answering in context. arXiv preprint 4rX.v:1808.07036, 2018.
[30] Yue Wu and Qiang Ji. Facial landmark detection: A literature survey. International Journal of Computer Vision, 127:115-142, 2019.
[31] Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2663 2673, 2021.
[32] Tomas Mikolov, Martin Karafíát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Re- current neural network based language model. Interspeech, volume 2, pages 1045-1048. Makuhari, 2010.
13

---

[33] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint 1xXIv:3301.38// 2013.
[34] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 1532-1543, 2014.
[35] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.030552 2022.
[36] Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16804-16815, 2022.
[37] Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding zero-shot adversarial robustness for large-scale models. arXiv preprint arXiv:2212.07016, 2022.
[38] Wenqi Shao, Yutao Hu, Peng Gao, Meng Lei, Kaipeng Zhang, Fanqing Meng, Peng Xu, Siyuan Huang, Hongsheng Li, Yu Qiao, et al. Tiny Ivlm-ehub: Early multimodal experiments with 2023.
bard. arXiv preprint arXiv:2308.0372
[39] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang. Yu Qiao, and Ping Luo. Lvlm-chub: A comprehensive evaluation benchmark for large 2023.
vision-language models. arXiv preprint
[40] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference Machine Learning, pages 2790-2799, PMLR, 2019.
[41] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.045. 10.04544, 2021.
[42] Farid Alizadeh and Donald Goldfarb. Second-order cone programming. Mathematical pro- gramming, (5()):3-1,, 2003.
[43] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1188.0800000 2018.
[44] Miguel Sousa Lobo, Lieven Vandenberghe, Stephen Boyd, and Hervé Lebret. Applications of second-order cone programming, Linear algebra and its applications, 284(1-3):193- 1998.
[45] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pages 1614- 1623. PMLR, 2016.
[46] Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 443-451, 2019.
[47] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aireraft. arXiv preprint arXiv: 1x.Xiv::306.55 1306.1556, 2013.
[48] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pages 178 178. IEEE, 2004.
[49] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013.
[50] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
14

---

[51] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings the IEEE conference on computer vision and paltern recognition, pages 3606-3613, 2014.
[52] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[53] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-min 10i-mmni discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt VI 13, pages 446-461. Springer, 2014.
[54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016 ieee conference on computer vision and pattern recognition (cvpr). Las Vegas, NV, USA, 1:770-78, 2016.
[55] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pailern recognition, pages 3485-3492. IEEE, 2010.
[56] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-338, 2010.
[57] Cyrus Rashtchian, Peter Young, Micah Hodosh, and Julia Hockenmaier. Collecting image annotations using amazon's mechanical turk. In Proceedings of the NAACL MALT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk, pages 139-147, 2010.
[58] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer songe-to-sentenee models. In Proceedings of the IEEE international conference on computer vision, pages 2641-26499 2015.
[59] Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao, and Li Deng. Stylenet: Generating attractive visual captions with styles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3137-3146, 2017.
[60] Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing, 56(4): 2195, 2017.
[61] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015.
[62] Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about world scenes based on uncertain input. Advances neural information processing systems, 27, 2014.
[63] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017
[64] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1X06v/600.05523 2016.
[65] Pranay Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint 1806.0800.00388 2018.
[66] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
15

---

[67] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 11 11-20, 2016.
[68] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: arXiv:/9077 /907. 1692, 2019.
[69] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: foundational language and vision alignment model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recogni- tion, pages 16638-15600 2022.
[70] Zhongzhi Chen, Guang Liu, Bo-Wen Zhang. Fulong Ye, Qinghong Yang, and Ledell Wu. Altelip: Altering the language encoder in clip for extended language capabilities. arXiv preprint MwXiv:2211.0661/ 2022.
[71] Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramana. Multimodality helps unimodality: Cross-moda few-shot learning with multimodal models. arXiv preprint arXiv:2301.08 06261, 2023.
[72] Charles L Byme. Alternating minimization and alternating projection algorithms: A tutorial. Sciences New York, pages 1 1-41, 2011.
[73] Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising eqqeece-to-sequenee pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910. 13461, 2019.
[74] Kevin Clark, Minh-Thang Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint S.Ii:2000.105555 2020.
[75] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 2irds-200-2011 dataset. 2011.
[76] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-ave aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2820-2828, 2019.
[77] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear boutlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018
[78] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 2818-288 18-2822, 2016
[79] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[80] X Chen, S Xie, and K He. An empirical study of training self-supervised visual transformers. arxiv e-prints. arXiv preprint arXiv:2104 100.2104..2077, 2021.
[81] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0000
2015.
[82] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 12009-12019, 2022.
16

---

[83] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 1904-6913 2017.
[84] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[85] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0366 2020.
[86] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with diseeengddd embedding sharing. arXiv preprint 1.0053, 2021.
[87] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[88] Jumnan Li, Ramprasaath Selvaraju. Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
[89] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through learning framework. In International Conference on Machine
a simple Learning, pages 23318-23340. PMLR, 2022.
[90] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint arXiv:2002.1 1770, 2020.
17

---

A
Method
Here we derive in detail the regression with Unified Noisy Label Embeddings that appear in the method section of the text in A.. and give complete proof of the convergence of the method in A.c
A.1
Regression with Unified Noisy Label Embeddings
Setup. we assume that label embedding is a linear mapping of the model feature with additive Gaussian noise with variance of o. given by 20 wT & + and N Tz) where 20 = ==== & is the regression prediction, w E R RDD and are regression weights and regression error, respectively, and I is L-by-L identity matrix.
We assume that F-labels {= K
obtained from different foundation models are oracles that indepen- dently provide noisy estimates the label embedding Formally, we have P(zg|2) N(2, rf
Without loss of generality, we assume that L=
Then the joint probability over noisy labels for fixed n, That is, for given x", we have:
P < w) = P(=f. ... ,kl2,," w) P<el=" wddd
(4)
Due to the independence between 2µ and v, using the real label 2, can rewrite it as:
P T.. x I=" w) P(( = P(222x"" w)dz
(5)
And using the independencies among Zk. we have:
K
(2)
P .. ,,,,,,,,,, = II
Pi=klz,oj ... ,00)
(2t) & I1 Tk
exp
20
(6)
1=1
Due to P(ze2z) N(z Ir.., can rewrite it as:
P(= ......... [ _w) /
EK
R+1 (2T))
2
I
exp
20
dy
which can be calculated
P(# 2 1r" [,(=",w) Al f - A7
Aiy
π
(A3)2
44g
-A
(8)
A2
where A₁ = IT 42
12 Ik= 1/20 =3= 43 -K ot
and A = R
120
Consider the joint probability over all N instances, we have:
P( x
N II
π
e
(AR)2
A7
A₂
1=1
where X E NxD denotes the feature matrix, N is the number of data points and D is the number of features.
Then given N data points, the negative log-likelihood is given by
N
(A
N log A + log A
(A
+const
(10)
2
4A
n=1
L2
where Li and L2 are given by
L1
N
log
Σ
K
N
log oke
L2
(2
EK ///2
(11)
20
K
1/0
k=0
k=U
k=
18

---

Since L1 is independent of input data, we focus on L2. To simplify the notation, we re-denote 7he / and I' = > k= Tkk Using this notation, Le can be rearranged as:
N
K
(E
TK + 70200
L2 =
+
(12)
++
==1
k=1
K
K 7h (=()))
K
(170
T 70
70
)z0
r
20
70
70 T
2/
20
Tk=k)
(13)
==1
k=
k=1
k=
Th0
(30
K
7k r
2
+
(x))) (1
70
7x22)
(14)
70
['[ + 76)
k=1
N
K
K
Σ
70
(w
7
+
7k (=x)) (1 +
mx
)²}
(15)
T(T + 70)
1=1
k=1
Hence, the negative likelihood in Equ. 10
can be written as
1 1'10
Th +)) R(m))
(16)
70
s(10,t
km 2x where R(yx) = +
Ek kmm )x=))² The computational intractability R1x)). Note that the coefficient % 1. By removing regularizer R(*x) and positive scale parameter
(1
%
T(T +oo
of Eqn. 16 comes from the regularization term R
P'o > 0 and
1+770
>x
K
I'o the minimization T-700
A=l
of negative log-likelihood can be approximately treated as weighted linear square regression, as given by
1 VERXXT,EEA K-1 s(w, t) = -11 - Ztll
min
(17)
E Nx the data matrix whose n-th row is model feature (i")T, E RD11 are weight parameters, Z E R N is F-Label matrix whose k-th column is the label embedding =t. and t IR. K satisfies that 1 1, 0 which is a (K 1)-D simplex denoted as A K-11
In Eqn. 17
A.2 Convergence Analysis and Proof Outline
We will prove the convergence property of the function value. Indeed, we demonstrate a stronger condition that the function value decreases after each round of iterations and From the monotone convergence theorem, the convergence can thus be derived. For other convergence properties of alternating minimization, readers refer the literature [72]. which can be of independent interest.
In the proof, we exploit the smoothness of the function and design projection gradient descent method with sufficient decrease for the constraint optimization problem. The sufficient decrease in the unconstrained problem is direct corollary.
Definition 1. A function (xx) _d + IR is said to be 3-smooth with constant if
( -
- [v)) 3 2 yll. Va, Rd.
Lemma Suppose is the simplex constraint. and y Ed, II denotes the projection operator. Then the inequality holds
(Tx - x) y) S.
Proof. For the projection IIx (v), it is convex optimization problem and can be formulated
min f(x) = | - --1
where 1²1 1 and O. We denote x² as the optimal solution the problem. For the convex optimization problem, it holds for all x E Rd that
Vflzt) x²) < 0.
19

---

Therefore we can derive
2(x² - -)) (x² - x)) S O.
The lemma is proved.
Lemma 2. Let f be a 8-smooth function.. For any I, y E dom(f)
[(()) f(v) - - w) s -11--10 11-l y/12.
Proof.
[((x) -(x)-f(0) f(y) (=())) x-) 1 Vf(y t(x -(---))11 -(x=-))) g)dt Vi((y)) v)
$ /. [V + 11x - n)) - V - slld
< I. Bt|| y11²
3 2 y/1
The last inequality holds because / is 3-smooth function.
Lemma 3. Suppose the function
f is the B-smooth function, and X is the simplex constraint. For
any z, y E X. let x 1Ix(x= 3 fxx)) and gx (x) = =3(x - x x²), Then the inequality holds
1
-(x))-((s) s (x)(x) (x- v) 23 11x((1)1
Proof. Using Lemma.
D we have
(xt (xt-(x- (x- 1 ( -v) 50.
which is equivalent to By using Lemma.
V[(x)) (xt y) S gx(x)) y)
2 and the fact f(x*) - f(y) = f(xx)=(++--) f(x) + f(x) - -f(y). we have
f(xt) - f(t) - vf(x) [f(x)((x*>))))))))))))() (x= x) + _ | - x1 Vf(x) - 3)
2
=V)()² y)+ +
23 1 [x)]
<xx(x)² (xt -)) + 23 1gx (z)]
(x+-
v) + 1
=xx(x) (x+
2 11x1x (z)]1
= gx (x+ - x) + -x(x)) (x y) + 1
23 (x)x
= =x) (x y) 8 |lex(=)]1 + 233 [x)x2
-xx(x)² (x- v)
23 11xxxx (x)1.
Theorem 3. Suppose sww, 1)= 1x Ztll2- where X E ERDD, Z RNK RD11 and t t A K-1 the inner loop of Algorithm lines 7- decreases after each iteration. Specifically, t)). For any t E K-1, s(w, ( ssw,,t) f) S
denote 3 = = Z|| and t
It + 2 < 0.
20

---

Proof. Since we fix parameter w and consider the optimization problem of variable t, we denote s(t) s(w, t), which leads to Vs(t) = -22 22 (Xw* Zt). For any 1 t2 dom(s)
V>(()) V=((2)] = || Z11 -27 Z12 < |27" 1 1211-
shows that the f(1) is 8-smooth, where B = 1| Z||. We denote 1 E A K-1 to be the initial point and 1 to be the result of one iteration of where /+ AA- 2 V)()). From Lemma 3 we can replace at, and x with 1+, ,, and I, repsectively. In this way, the inequality holds
According to the definition
0 S s(t)) s(1)
23 11 - ++ s stt)
Therefore, according Monotone convergence theorem, the function value of the iterative algorithm will converge.
Theorem 4. Suppose s(w, t) = 2 1 Ztll2 where X E RNND, Z RNKK, w E Px1 and t E A-1 the function value in Algorithm will be convergent. Specifically, denote w ,t' as the result after one iteration of w, respectively, we have s(w", S s(w", ,t) s(w,t) t).
Proof. In the first step, we denote / E A is the initial point, then use gradient descent algorithm to calculate w*. Since the optimization problem for w is a convex optimization problem and use lemma 2 the decreasing property for the gradient part can be derived. That is, for each w E IR Dx1 have sww", .t) < t(w,t). In the second step, fix w w from Theorem 3 we have s(w", t*)= S s(w",t), Therefore, the value of sww, t) satisfies: 0 0 s(w", t²) S s(w", , t) S s(w, t). from Monotone convergence theorem, s(w, converges the limiting point. As shown above, the overall convergence of our algorithm is guaranteed.
B Experiment
In this section, present detailed descriptions of datasets in Sec. B.2 pre-trained models and baselines in Sec. B33 and ground-truth scores in Sec. B.4 in various target tasks. More ablation studies can be found in Sec.
Foundation Models. On image classification, image captioning, referring expression comprehension, and visual question answering,
foundation models CLIP (26). BERT 43) and GPT-2 (27) On text question answering, we use foundation models GPT-2 (2). BART 73 and ELECTRA 74 CLIP trained large dataset of images and their corresponding captions, which understand the relationship between images and text. BERT is a pre- pre-trained language model that can understand and generate natural language. GPT-2 was trained specific tasks such as text completion and text summarization. Bart a sequence seueence---seeeecce model, which is both auto-regressive and bidirectional. Flectra is different type of language model that key idea is to pre-train a generator model to produce fake data and shows promising results in various NLP tasks.
large corpus of text and can be fine-tuned for
Interpretation of weighted Kendall's tau. The Kendall's 7 represents the ratio of concordant pairs minus discordant pairs when enumerating all pairs of (Tm
JM (T.M and {Gm }M m given by
2
Σ ISKJJM
sgn(G; Gg)-GGG)[nn. TT})
(18)
M(M 1)
where sg)(z) returns 1 x < 0 and otherwise. In this work, a weighted version of Kendall's т, denoted as Tw. is employed to assess transferability metrics considering that nop-performing model always preferred for target tasks transfer learning. principle, larger Tu implies the transferability metric can rank pre-trained models better. And if metric can rank top-performing models better, Tu would also larger. We also use other measurements to assess the performance of transferability metrics in Table 17 of Sec. C
21

---

Table Comparison of different transferability metrics on CNN models regarding Tu and the wall- clock time where EMMS(One) denotes EMMS with the one-hot label. Our proposed EMMS achieves the best transfer-ability assessment over 11 target tasks and exhibits higher efficiency than NLEEP.
Aireraft
Caltech
Cars
CF-10
CF-100
DTD
Flowers
Food
Pets
SUN
vOC
Avg.
Weighted Kendall's ' tau
LEEP
-0.234
0.605
0.367
0.824
0.677
0.486
-0.243
0.389
0.722
0.371
0.409
LogME
0.506
0.435
0,576
0,852
0.677
0.647
0.111
0.385
0.411
0.487
0.669
0.509
NLEEP
-0.41
0.614
0.265
0.818
0.805
0.796
0.122
0.214
0.753
0.925
0.687
0.611
TrunsRate
0.172
0.269
0.172
0.513
0.197
0.336
-0.176
-0.071
0.173
0.612
0.651
0.236
EMMS(One)
0.481
0,546
0.304
0.963
0.804
0.701
0.498
0.588
0.574
0.638
0.707
0.618
EMMS
0.556
0,562
0.565
0,963
0.840
0.720
0.498
0.608
0.604
0.667
0.735
0.664
Clol-Clock Time (8)
LEEP
5.1
4,9
8.3
22.3
23.8
3.5
3.8
37.1
3.9
21.1
4.8
10.4
LogME
30.36
31.24
56.26
90.34
188.3
15.16
22.27
334.53
17.55
180.01
20.05
289.64
NLEEP
253.8
488.7
973.8
1.1e4
1.7c4
146.0
294.0
2.0e4
580.8
8.6e.
678.8
5455.9
TransRate
147.90
163.41
300.29
193.64
75.48
166.24
195.92
60.53
430.33
18.72
165.24
EMM)(One)
17.43
20.53
35.22
70.01
78.24
12.75
18.04
116.23
15.04
70.98
18.42
42.99
EMMS
65.85
63.49
79.79
245.49
295.37
46.38
63.52
417.80
59.64
173.59
64.60)
143.2
Table 8: Comparison of different transferability metries VQA models in rank correlation To with the ground truth and the wall-clock time. The LogME denotes using LogME with F-Label. Our proposed EMMS performs better than PACTran head over 3 target tasks with much less time.
DAQUAR COCO-QA
CLEVR DAQUAR COCO-QA CLEVR
Weighted Kendall's 's Tu
Wall-Clock Time (s)
LogME
0.586
0.591
0.281
116.72
716.35
4665.06
PACTran(Diry
0.671
0.296
0.347
633.16
1169.91
428.03
PACTran(Gam)
0.595
0.419
0.319
614.23
1061.72
428.49
PACTran(Gau)
0.478
0.378
0.415
637.39
1075.88
418.34
EMMS
0.712
0.812
0.804
50.54
263.72
274.56
B.1
More experimental results
B.1.1 Performance on Image Classification with CNN Models
Performance and wall-clock time comparison. We compare EMMS with previous LEEP, NLEEP, LogME, and TransRate. As shown in Table our EMMS achieve the best average Tu 11 target datasets and the best Tu on target datasets. Compared to NLEEP, which is the most effective other than EMMS, have almost 1/40 of the time of NLEEP.
B.1.2 Performance on Visual Question Answering
To further demonstrate the generality of EMMS in multi-model tasks, show how EMMS can work for VQA. follow previous practice 24 which treats VQA as a classification task (vocab-based VQA). That is, construct vocabulary based on the top answers in the training sets and classify them into some those labels. The models to selected and the architecture the same as the image captioning.
Performance and wall-clock time comparison. As shown Table EMMS is clearly ahead of PACTran in terms of results and time, proving that EMMS has the ability to handle multi-modal tasks very well. We can find that EMMS outperforms PACTran on all datasets. In particular, EMMS achieves 93.8% and 93.7% gain over PACTran the COCO-QA and CLEVR datasets with rank correlation Tu while reducing time consumption by 75.1 % and 34.3% respectively compared to Pactran. This indicates that EMMS performs well on both ordinary VQA datasets(DAQUAR, COCO-QA) as well as VQA datasets(CLEVRR that focus on inference capabilities.
B.1.3 Regression
In addition image classification and a variety of multi-modal tasks, here we show that EMMS can also be used for regression tasks. The daatasets for regression task use is CUB200 75 and IIIT
22

---

Pets [54). The input is an image containing various birds and pets, respectively. We need to predict the coordinates of the bird's or pet's bounding box the image and mean square error (MSE) on the test data is the ground-truth. The pre-trained models used are the same as the image classification task with CNN models and the only baseline is LogME. We extract F-Labels using Bert and RoBerta.
As shown in Table EMMS significantly outperforms LogME, with 29.5% and 13.9% performance improvement on CUB and Pets respectively.
Table 9: Comparison of different transferability metrics on regression models in rank correlation with ground truth.
CUB
Pets
Weighted Kendall's tau Tur
LogME
0.464
0.437
EMMS
0.601
0.498
B.2
Descriptions of Datasets
B.2.1
Image Classification
For image classification, we adopt 11 classification benchmarks including FGVC Aircraft 47 Caltech-10 101 [8). Stanford Cars [49]. CIFAR-10 [50), CIFAR-100 [50]. DTD [51]. Oxford 102
Flowers [52] Food-101 [53] Oxford-IIIT Pets (54,, SUN397 (55], and VOC2007 [66] These
datasets cover broad range of classification tasks, which include scene, texture, and coarse/fine- grained image classification, which are widely used in transfer learning. In particular, CF10 and
VOC2007 are typical coarse- coarse-grained classification datasets, Aircraft, and Cars are typical fine-grained classification datasets, and CF100 contains both coarse- and fine-grained classifications.
B.2.2 Image Captioning
For image captioning, We use Flickr8k [57]
Flickr30k (58), BiickSStyle0O-Humme [59].
FlickrStyle10K 10- (59] and RSICD 60 Among them, Flickr8k and Flickr30k have com- monly used image captioning datasets for natural images and have no emotional color; RSICD is commonly used image captioning dataset in remote sensing; Plickr10k-1 and Flickr10k-R are also image captioning datasets for natural images, but their images are depicted with humorous and romantic emotional colors, respectively.
B.2.3
Visual Question Answering
For visual question answering, we apply COCOQA (61. DAQUAR (62 and CLEVR (63 ..mong them, DAQUAR is an early VQA dataset on real images; CLEVR is synthetic dataset, which is visual scene composed of some simple geometric shapes, focusing on evaluating the inference ability of VQA models; the questions and answers of COCO-QA are generated by NLP algorithms, and images are from the COCO dataset, which is also commonly used VQA dataset.
B.2.4 Text Question Answering
For text question answering, we separately use SQuADI.1 1 64 .SQuAD2.0 65 which are collections of question-answer pairs derived from Wikipedia articles and are widely used text question answer.
B.2.5 Referring Expression Comprehension
For referring expression comprehension, we separately use RefCOCO [66] RefCOCO+ [66 and
RefCOCOg [67] Speciffiiy,, RefCOCO includes instances where there is only one object of its kind in the image, while RefCOCO+ includes instances where multiple objects of the same kind exist in the image.
23

---

B.3 Pre-trained Models and Baselines
B.3.1 Image Classification
Pre-trained Models. For CNN-based models, We select 11 widely-used CNN models including
ResNet-34 [6] ResNet-50 [6) ResNet- [6) ResNet-152 (6. DenseNet-122 121 (7]. DenseNet-169
77. DenseNet-201 71 MNet-AI 76.. MobileNetV2 (77), GoogleNet [8], and Inception [78]. All
these models trained ImageNet dataset [1. which are widely used within the field of migration
learning. For ViT-based models, we collect 10 ViT models including ViT-T 211 ViT-S 19, ViT-B
21. DINO-S (9).. MoCov3-S [80] PVTv2-B2 [T. PVT-T MT PVT-S I. PVT-M [T], and
Swin-T 10), which are widely used in various vision tasks. Besides, we append EMMS with one-hot
label, which degenerates to a linear regression whose label is the one-hot vector. We fine-tune these
models on the 11 target datasets to obtain the ground truth.
Comparison Baselines, Here NLEEP 31, LogME [13], and TransRate [23] which have been experimented with model selection on image classification tasks.
some of the latest methods baselines, including LEEP 12]
B.3.2 Image Captioning
Pre-trained Models. We use classic and effective image captioning model architecture, which contains an image encoder and a language encoder to extract the features of the image and the corresponding caption, then fuses the image feature and the text feature and input it to the classifier. We aim to choose the best combination of image encoder and language encoder. Besides, We finetune each model in COCO Caption
[81 and use these the pre-trained models.
Specifically, We separately use ViT-B 2 BB [0]. Swinv2-B 82 as image encoder and Bert 43 Roberta [68] Bart (73 as language encoder, and use VisionEnncodrDDecooee as the model architecture. Following the setting in PACTran (24), We finetune the model in COCO
from HuggingFace
Caption [8]] and use these the pre-trained models. Following common practice( (83). we treat
image captioning as a vocab-based classification task. That is we use a vocabulary and classify the caption into the index of some words in the vocabulary. Afterward, training is done according the classification task criteria.
Comparison Baselines. In this common setup, each caption is converted to matrix Y E R where L denotes the length of the caption after padding or truncation and N denotes the size of the vocabulary, and each row in the matrix is one-hot vector. Since N is generally very large, Existing model selection metrics do not scale to this case due to the huge amount of time spent. The only baseline we use is to model the fused feature with F-label using LogME since only LogME can handle the regression task. Here we calculate the average To and time of it with K single F-label from K foundation models we use respectively.
B.3.3 Visual Question Answering
The model architecture and the model selection settings are the same in the image captioning, Following the setting in PACTran [24], here we use the model after finetune on VQA-v2 [83] as the pre-trained model waiting for selection and treat VQA as vocab-based classification task.
Pre-trained Models.
Comparison Baselines. Here we calculate the average Tu and time of it with K single F-label from K foundation models we use respectively. And in addition to that, the three methods proposed in PACTran 24 are added here, which are the only methods currently applied to VQA tasks.
B.3.4 Text Question Answering
43 RoBERTa-Large 68 XLNet- Large [84). DeBERTa (85) (XLarge), DeBERTa- V2 83 (XLarge and XXLarge). DeBERTa-Ve V3 (86) (Base, Small, XSmall). More specifically, we simultaneously input the question and passage into the aforementioned models, utilizing the distinctive symbol [SEP] to demarcate them. By stacking the predicted head onto each model, we could further fine-tune the model such that it can predict the start
Pre-trained Models. The selected models include BERT-Large
and end positions of the answer within the passage. This is achieved by using two binary classifiers, where one is dedicated to identifying the start position and the other to pinpointing the end.
24

---

Table 10: The fine-tuning accuracy of supervised CNN models on 11 target tasks.
Aireraft
Caltech
Cars
CF-10
CF-100
DTD
Flowers
Food
Pets
SUN
VOC
ResNet-34
84.06
91.15
88.63
96.12
81.94
72.96
95.2
81.99
93.5
61.02
84.6
RexNet-50
84.64
91.98
89.09
96.28
82.8
74.72
96.26
84.45
93.88
63.54
85.8
ResNet-101
85.53
92.38
89.47
97.39
84.88
74.8
96.53
85.58
93.92
63.76
85.68
ResNet-152
86.29
93.1
89.88
97.53
85.66
76.44
96.86
86,28
94,42
64,82
86,32
DenseNet- 121
84.66
91.5
89.34
96.45
82.75
74.18
97.02
84,99
93.07
63.26
85.28
DenseNet-169
84.19
92.51
89.02
96.77
84.26
74.72
97.32
85.84
93.62
64.1
85.77
DenseNet-201
85.38
93.14
89.44
97.02
84.88
76.04
97.1
86.71
94.03
64.57
85.67
MNet-A1
66.48
89.34
72.58
92.59
72.04
70.12
95.39
71.35
91.08
56.56
81.06
MobileNetV2
79.68
88.64
86.44
94.74
78.11
71.72
96.2
81.12
91.28
60.29
82.8
Googlenet
80.32
90.85
87.76
95.54
79.84
72.53
95.76
79.3
91.38
59.89
InceptionV3
80.15
92.75
87.74
96.18
81.49
72.85
95.73
81.76
59.98
83.84
Table 11: The fine-tuning accuracy of vision transformer models on 11 target tasks,
Aircraft
Caltech
Cars
CF-10
DTD
Flowers
Food
Pets
SUN
VOC
VIT
71.26
89.39
82.09
96.52
81.58
71.86
95.5
81.96
91.44
58.4
83.1
VIT-S
73.12
92.7
72
97.69
86.62
75.08
96.79
86.26
94.02
64.76
86.62
VIT-B
78.39
93.47
89.26
98.56
89.96
77.66
97.98
88.96
94.61
68.62
87.88
PVTv2-B2
84.14
93.13
90.6
97.96
88.24
77.16
97.89
88.67
93.86
66.44
86.44
PVT-T
69.76
90.04
84.1
94.87
75.26
72.92
95.8
83.78
91.48
61.86
84.6
PVTS
75.2
93.02
87.61
97.34
86.2
75.77
97.32
86.98
94.13
65.78
86.62
PVTM
76.7
93.75
87.66
97.93
87.36
77.1
97.36
85.56
94.48
67.22
87.36
Swin-T
81.9
88.93
97.34
85.97
77.04
97.4
86.67
94.5
65.51
87.54
MoCov3-S
76.04
82. 18
97.92
85.84
71.88
93.89
90.44
DINO-S
72.18
86.76
79.81
85.66
75.96
95.96
85.69
92.59
64.14
84.8
Comparison Baselines. Here we calculate the average Tu and time of with F-labels from K foundation models respectively.
B.3.5 Referring Expression Comprehension
Pre-trained Models. The candidate multi-modal architectures considered for REC task incorporate Blip (87), ALBEF [88), CLIP (26) (ViT-B-32, VIT-B-16, ViT-L-14, STT-L-14-336 RN50), OFA [89] (Base, Large, Huge). practice, we respectively extract the visual and textual representations from each of these models and feed them into multi-modal interaction module followed by a stacked detection head, and further fine-tune the model to generate the ground truth model selection.
Comparison Baselines. Here we calculate the average Tw and time of LogME with K single F-label from K foundation models we use respectively.
B.4
Fine-tuning Score on Various Target Tasks
B.4.1 Image Classification
Fine-tuning Details. The ground truth of the problem of pre-trained model ranking is fine-tune all pre-trained models with a hyper-parameters sweep on target datasets. Given the model and the target dataset, two of the most important parameters would be learning rate and weight decay optimizing the model [90]. Therefore, we carefully fine-tune pre-trained models with grid search of learning
rate {le 1, 2, le 3, le 4} and weight decay (le 3. 4, le 5, 1e 6, 0). And using SGD optimizer. After determining the best Syper-parameten candidate, we fine-tune the pre-trained model on the target dataset with the candidate and then obtain the test accuracy as the ground truth. We use Tesla V100 with batch size of 128 to perform finetuning. All input images are resized to 224 x 224. avoid random error, we repeat the above fine-tuning procedure three times and take an average to obtain the final fine-tuning accuracy. For reference, we list the fine-tuning accuracy of supervised CNN models in Table T0 and vision transformer models Table |I respectively.
25

---

B.4.2 Image Captioning and Visual Question Answering
Fine-tuning Details. The setting of finetune here is approximately the same as in image classification. We carefully fine-tune pre-trained models with grid search of learning rate in {1e-4, 1e- le 6 and weight decay in (le 4, le 5, le - 6}. And using AdamW optimizer. After determining the best typer-parameten candidate, we fine-tune the pre-trained model on the target dataset with the candidate and then obtain the test BLEU-4 and accuracy as the ground truth. However, since Flickr10k-B and Flickr10k do not provide test set, we use 6:1 ratio to divide the original training set of 7000 images into a training set and a test set. For visual question answering, Due to the lack of test set for CLEVR dataset,
also assign its training set training set and test in the ratio of 6:1. We use an Nvidia A100 with a batch size of 64 to perform finetuning. All input images are resized to 224 224. To avoid random error, repeat the above fine-tuning procedure three times and take an average to obtain the final fine-tuning accuracy. For inference, We use BLEU-4 as the score for the model with image captioning and accurarcy the score for the model with VQA. list result of image captioning models in Table 12 and visual question answering models in Table T3 respectively.
Table 12: The fine-tuning BLEU-4 of image captioning models on 5 target tasks.
Table 13: The fine-tuning accuracy of visual question answering models on 3 target tasks.
F8k
F30k
RSD
F1Ok-H
F1Ok-R
DAQUAR COCO-QA
CLEVR
Vit-Bert
18.51
26.65
31.39
5.31
5.18
Vit-Bert
25.01
55.11
59.29
Vit-Roberta
20.53
23.70
29.92
5.88
5.48
Vit-Roberta
26.38
57.30
62.80
Vit-Bart
21.90
25.13
31.35
5.75
5.53
Vit-Bart
26.30
59.60
64.98
Swinvit-Bert
22.91
33.54
5.67
Swinvit-
28.05
61.72
68.25
Swinvi-RRobert
23.99
28.84
33.07
7.11
5.49
Swinvit-Roberrs
27.75
62.81
66.09
Swinvit-Bart
24.68
28.03
32,99
6.10
5.95
Swinvit-BB Bart
27.06
60.62
67.17
Swin2vit-Bert
25.69
31.33
35.45
5.86
5.49
Swin2vit-Bert
26.45
63.1
67.4
Swin2vit-Rohers
23.40
28.81
36.22
6.80
7.13
Swin2vit-Roherta
26.33
66.54
65.91
Swin2vit-Bart
26.24
30.35
34.72
7.90
5.96
Swin2vit-Bart 26.25
64.4
70.34
B.4.3 Text Question Answering
Fine-tuning Details. The accuracy of most models in TQA provided by DeBERTa for DeBERTa-V3 [B6]e (Base, Small, XSmall). Following the setting of Bert [43] models with a batch size 24 for 2 epochs. We use Adam W optimizer with an initial learning rate of 3e 5, polynomial decay. The Dev FI score is used for pre-trained model ranking. All experiments are implemented on an NVIDIA Tesla A GPU. The finetune accuracy is shown in Table
85
86 except
finetune these
Table 14: The standard metric the Dev F1
The standard metric Acc@0.5 of referring expression comprehension models on 3 target tasks.
Table 15:
score of text question answering models on 2 target tasks.
SQul.1
SQu2.0
RefCOCO
RetCUCO+
RefCOCOg
BERT-Large
90.9
81.8
Blip
88.67
84.68
85,08
RoBERTa-Large
94.6
89.4
ALBEF
87.98
82.20
82.89
CLIP-VIT-B-32
83.20
74.56
76.98
XI.Net-Large
95.1
90.6
CLIP-VIT-B-16
87.35
80.12
81.69
DeBERTa-Large
95.5
90.7
CLIP-VIT---14
90.17
86.09
87.13
DeBERTa-V2-XII.ne
95.8
91.4
LL-P-VVTTL-14-3366
91.67
87.60
87.89
DeBERTa--V--XX..am
96.1
92.2
CLIP-RN50
84.69
76.72
79.39
DeBERTa-V3-Base
93.9
88.4
DFA-Base
88.48
81.39
82,29
DeBBRTT-V3-Small
89.8
82.9
OFA-I Large
90.05
85.80
85.89
DeBERTa-Ve -3--raall
91.5
84.8
OFA- Huge
92.04
87.86
88.07
B.4.4 Referring Expression Comprehension
Fine-tuning Details. For referring expression comprehension, the standard metric Acc@0.5 on the validation set is used as the ground truth. For finetuning, we use a batch size of 128 with resolution of 512 x 512 for each image. We finetune the models on each dataset for 12 epochs with learning rate of (3e 5, 5e 5) and weight decay in {le 3, le 5) using Adam optimizer. The best performance on the validation set for each task reported among these hyper-parameters. Table shows the performance of referring expression comprehension models.
26

---

Table 16: The fine-tuning MSE on test set of models used in regression on target tasks.
CUB
Pets
ResNet-33
4.114e
4.245e
ResNet-50
3.521c 4 4.489c
5
ResNet- 101
2.746c 4 3.224c
5
ResNet- 152
2.539c - 4 2.775c
5
DenseNet-122
5.354e - 1.096e
DenseNet- 169
4.787e - 9.469e
DenseNet-201
4.651e - 1.058e
MNet-A1
1.1475c 1.878c
MobileNetV2
6.253e
9.510c 5
Googlenet
7.192c
1.197e 4
Inception
6.174e 4
9.633e
5
Table 17: EMMS under different measurements of transferability assessment. The results are obtained on Flickr8k and RSICD datasets with image captioning task and Aireraft and DTD datasets with image classification task with ViT-based models. EMMS outperforms Log ME and other baselines under various measures.
Data
Method
Rel@1 1
Rel@3
Fwr
Data
Method
Ret@1
Rel@3
Fw
2
LogME
0.928
1.0
0.799
0.537
0.483
2RRD
LogME
1.0
0.727
0.708
0.518
0.501
EMMS
1.0
1.0
0,741
0.823
0.667
0.660
EMMS
1.0
1.0
0.783
0.765
0.611
0.705
3*Aircrfft
LogME
0.852
0.993
0.407
0.060
0.378
0.299
3DDTD
LogME
0.992
1.0
0,641
0.694
0.556
0.569
TransRate
0.926
0.967
0.457
0.499
0.289
0.244
TransRate
0.992
1.0
0.607
0.676
0.422
0,533
EMMS
0.926
0.967
0.622
0.608
0.511
0.481
EMMS
0.992
1.0
0.704
0.785
0.644
0.621
B.4.5 Regression
Fine-tuning Details. For regression, mean square error (MSE) on the test data is the ground truth. For finetuning, we use a batch size of 64 with resolution of 224 x 224 for each image. fine-tune pre-trained models with grid search of learning rate in {le - 1, le - 2, le - 3, le 4} and weight decay in (le 3. 4, 1e 5, le 6,03 with SGD optimizer. The fine-tuning MSE on test set of models used in regression is in Table 16
carefully
Table 18: The effect of Label Embedding in EMMS. Three variants of EMMS are considered: (1) EMMS with one-hot label; (2) EMMS with single F-Label; (3) EMMS with multiple F-Labels which is the original. We see that label embedding brings some performance improvement to EMMS.
Aircraft Caltech
Cars
CF-10
CF-100
DTD
Flowers Food
Pets
SUN
vOC Avg.
Weighted Kendall's tau Tw
(1)
0.481
0.546
0.304
0.963
0.804
0.701
0.498
0.588
0.574
0.638
0.707
0.618
(2)
0.531
0.562
0.426
0.952
0.804
0.720
0.481
0.602
0.535
0.667
0.726
0.636
(3)
0,556
0,562
0.565
0.963
0.840
0.720
0.498
0.608
0.604
0.667
0.735 0.664
C More Ablation Analysis
The Efftiveness of EMMS under Various Measurements, In addition to weighted Kendall's tau, we employ various other measures to evaluate our EMMS. These include Kendall's tau (7), Pearson's 's correlation (r), weighted Pearson's correlation (rw), and top-k relative accuracy, denoted as Rel@ k which represents the ratio between the best fine-tuning accuracy achieved on the downstream task using the top-k ranked models and the best fine-tuning precision achieved with all models. We test the robustness of our transferability metrics to different measurements on the Flickr8k and RSICD datasets for image captioning tasks, as shown in Table 17 Our EMMS consistently outperforms the previous transferability metric, including LogME and TransRate. Under the aforementioned measurements, demonstrating the superiority of our EMMS.
27

---

The Effect of Label Embedding In some multimodal tasks or text tasks, including image captioning or text question answering. Label emebdding directly affects the applicability of existing model selection metric to these tasks. In addition, even in classification tasks, the use of F-Label can also bring improvements in results. Here we focus on the comparison between label embedding and direct one-hot vectors for image classification tasks in CNN- -NN-based models. As shown in Table [8] the use of F-Label can bring performance improvement compared to One-Hot vector, the average Tu increase from 0.618 to 0.636; furthermore, the use of multiple F-Label also brings some improvement compared to the average of single F-Label with Tw increasing from 0.636 to 0.664.
The Effect of Computational Speedup. Here we experimentally demonstrate the effect of our accelerated algorithm. As shown in Table[[]] the algorithm is similar the in-accelerated version in terms of results, but much shorter in terms of the wall-clock time.
Comparison with different number of iterations The numher of iterations affects the EMMS time, here we conduct experiments on the VQA task for the effect of the number of iterations on the results. As shown in Table [9] we find that the number of iterations does not have a large impact on the performance of our method, and even a small number of iterations can guarantee the final result(e.g g. the number of iterations is 1). We believe that firstly our method converges very fast. And secondly, for the ranking problem of model ranking, even if the convergence is not sufficient, the original order can still be maintained large extent in EMMS, thus ensuring the effect.
Table 19: The effect of the number of iterations r on VQA models in rank correlation Twp We find that even small number of iterations allows the method to maintain its effect.
DAQUAR COCO CLEVR DAQUAR COCO CLEVR
Weighted Kendall's 's tau Tw
Wall-Clock Time (s)
7 3
0.743
0.812
0.804
111.05
735.21
745.11
r
== 2
0.712
0.812
0.804
78.01
536.45
573.22
1
0.712
0,812
0.804
50.54
263.72
274.56
The effect of using single foundation model We investigate how EMMS is influenced when only a single foundation model is provided. We conduct experiments on image classification and image captioning. We consider EMMS with the single foundation model including language foundation model (1) GPT-2 (27). (2) BERT 43)
(3) RoBerta (68), and multimodal foundation model (4) CLIP 126]. (5) FLAVA 1691. and (6) AItCLIP 770. For comparison, we include the result of our EMMS with default setting (K=3, i.c. CLIP, BERT, and GPT-2) and the result of previous state-of-
the-art methods obtained from LogME, NLEEP and TransRate. The results are reported in Table 20
and 5
We have several observations. (1) Different downstream tasks prefer F-Labels obtained from different foundation models. No single foundation model dominant in all target tasks. In particular, CLIP is not the best model for extracting F-Labels. (2) For image classification, both language and multimodal foundation models are competent for acquiring F-Labels. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.
Table 20: The effect of the single foundation model on EMMS. The results are obtained on image classification regarding Tun-
Aireraft Cakech
Cars
CF- 10
CF-100
DTD
Flowers Food
Pets
SUN
VOC
Avg.
SOTA/AIl
Weighted Kendall's 's tau
Previous SOTA
0.299
0.412
0.693
0.741
0.736
0.621
0.655
0.580
0.707
0.619
0.651
0.610
4/11
(1) Gpt2
0.481
0.463
0.448
0.652
0.745
0.621
0.562
0.652
0.740
0.616
0.730
0.610
6/11
(2) Bert
0,481
0.444
0.458
0.718
0.745
0.621
0.562
0.592
0.740
0.616
0.730
0,609
5/11
(3) RoBerta
0.448
0.444
0.507
0.701
0.745
0.608
0.562
0.580
0.740
0.574
0.730
0.604
3/11
(4) CLIP
0.481
0.444
0.496
0.608
0.720
0.621
0,562
0.558
0.740
0.616
0.706
0.595
3/11
(5) FLAVA
0
0.444
0.508
0.741
0.745
0.621
0,562
0.652
0.740
0.574
0.706
0.615
5/11
(6) AIILIPP
0.481
0.444
0.437
0.741
0.745
0.621
0.562
0.580
0.740
0.595
0.730
0.607
6/11
EMMS
0.481
0.444
0.706
0.718
0.745
0.621
0.562
0.673
0.740
0.619
0,730
0.639
8/1
28

---

The Wall-clock Time of Label Embedding, For classification tasks, since the maximum number of categories is often only a few hundred, Label Embedding very fast. Here we focus on documenting the time required for multimodal tasks, e.g. image captioning, text question answering. and referring expression comprehension, where label embedding more time-consuming. For each task, we use 8 Nvidia A100 GPUs for label embedding, with batch size of 512 for each GPU. The running time of label embedding for image captioning, text question answering, and referring expression comprehension is shown in Table 22 We measure the time for each dataset on the same CPU device (AMD EPYC 7H12 with 64-Core Processor) for three times and take the average as the final result.
Table 21:
The effect of computational speedup in image classification with ViT models. We can see that the accelerated version of the algorithm achieves a significant reduction in time while guaranteeing results. Two variants of EMMS are considered: (1) EMMS with normal algorithm: (2) EMMS with fast algorithm.
Aireraft Caltech
Cars
CF-10
CF-100
DTD
Flowers
Food
Pets
SUN
VOC
Avg.
Weighted Kendall's tau Tu
0.564
0.463
0.706
0.718
0.745
0.589
0.592
0.531
0.755
0.532
0.730
0.629
(2)
0.481
0.444
0.706
0.718
0.745
0.621
0.562
0.673
0,740
0.619
0.730
0.639
Wall-Clock Time (S)
102.06
114.72
177.25
718.34
724.5
50.24
87.28
944.57
83.37
336.92
104.9
313.10
(2)
21.31
17.23
28,06
154.61
182.11
13.87
15.95
265.99
17.93
63.86
16.63
72.55
Table 22: The wall-clock time (s) of label embedding in image captioning on 5 target tasks, text ques- tion answering on 2 target tasks, and referring expression comprehension on target respectily...
Task
Image Captioning
Text QA
Referring EC
Dataset
F8k
F30k
RSD
FIOk-H
F10k-R
SQUADI.1
SQuAD2.0
RefCOCO RefCOCO+ RefCOCOg
Time
14.56
89.31
18.92
3.37
3.13
35.67
53.87
49.19
48.88
31.63
The computational complexity of EMMS. We compare the computational complexity between LogME and EMMS in Table 23 We see that EMMS has lower computation complexity than LogME(F) because LogME(F) needs several iterations (T=3 on average) to converge. Moreover, EMMS allows for full vector computation and can be efficiently solved by existing scientific compu- tation packages such as up.linalg.Istso Nevertheless, LogME(F) cannot be written in fully vectorized form because the model parameters LogME(F) are highly coupled. Hence, LogME(F) can only be excuted in while loop.
In addition, in the classification task, we compare EMMS and LogME. EMMS usually has higher computation complexity because D2 >> C. In some cases, when the number of categories C' and the iteration number T are large, EMMS could be faster than LogME with vector computation. For example, we find that C' = 397 and T 4.46 on average over all models when LogME is convergent on the Sun397 dataset. It results in higher time complexity than LogME, as indicated in Table 23) We further verify this by implementing LogME with T = 1. As shown in Table 24] EMMS spends more time in calculating the transferability than LogME (T=1) on all datasets. However, LogME performs much worse than EMMS because it does not converge when T = 1.
Table 23: The comparison of computational complexity between LogME, EMMS(one), and EMMS in image classification. We denote model feature X E RND, and F-labels Z E R Da x with N = 104, D2 x 103, D2 1024, K = 3, and C' s 102. Moreover, T 3 denotes the iteration number of LogME. Moreover, LogME(F) denotes LogME with F-Label.
Complexity
Simplified Complexity
Vistor Compuration
LogME LggME(F)
STCD + (2T + IINCD, + +DD 37 STD21 (27 + TiNCD, + 03 NO0
STCD + ND + -DD +1]
3rDaD + ND. ND,
OMMS(One)
CDI NCD, : Dद NDD
ND
EMMS
ND. + 2ND.D. DJ + D.D, + (K3 K)(N)))+ RS K2 Klog
NDJ DN
29

---

Table 24: The comparison between LogME and EMMS. The results are obtained on image classifica-
tion regarding LogME (T = 1) indicates that the inner loop of LogME only performs once.
Aireraft Caltech Cars
CF-10
CF-100
DTD
Flowers Food
Pets
SUN
voC
Weighted Kendall's 's tau
LogME (T' 1)
0.378
0.341
-0.408
0.645
0.727
0.112
-0.074
0.561
0.528
0.259
-0.04
LogME
0.299
0.382
0.633
0.741
0.727
0.569
0.512
0.580
0.528
0.619
0.591
FMMS(One)
0.412
0.444
0.565
0.740
0.736
0.621
0.562
0.579
0.740
0.592
0.730
EMMS
0.481
0.444
0.706
0.718
0.745
0.621
0.562
0.673
0.740
0.619
0.730
Wall-Clock Time
LogME (7 1)
4.45
4.72
8.18
34.81
40.15
3.65
5.13
53.7
4.59
31,66
6.03
LogME
8.93
10.89
30,28
53.07
62.13
4.78
9.27
104.92
6,28
425,43
7.42
EMMS(One)
4.12
4.45
8.07
19.45
26.18
2.65
4.03
39.72
3,50
24.84
4.07
EMMS
21.31
17.23
28.06
154.61
182.11
13.87
15.95
265.99
19.73
63.86
16.63
Comparison with variants of existing methods. To further validate the efficacy of EMMS, compare it with TransRate using F-Labels on image classification. To this end, we estimate the mutual information of the model feature and F-label following TransRate. Specifically, denote model feature E Dx and the F-Label Z₁ E N2 we estimate the mutual information of and Zg after the discretization operation for each dimension of D2 separately and then take average obtain the final score.
Moreover, we implement two baselines based TransRate. When K 1, instantiate the F-Label as the CLIP embedding. When K = 3, we instantiate the F-Labels as the embedding collection extracted from the CLIP, BERT, and GPT-2. In this case, the final score is averaged over three F-Labels. The results are shown in Table 25] where we can see that our EMMS consistently outperforms TransRate with F-Labels (both K=I and K=3).
Table 25: The comparison between TransRate and EMMS. The results are obtained on image
classification regarding Tw. TransRate (K) indicates that the number of foundation models used.
Aircratt Caltech
Cars CF-10
CF-100
DTD Flowers Food Pets SUN VOC Avg.
Weighted Kendall's Tm
TransRate( K==))
0.297
0.440
0.682
0,655
0.501
0.533
0.548
0.537
0.736
0.533
0.666
0.557
TransRate(K=3)
0.295
0.441
0.682
0.523
0.501
0.542
0.548
0.539
0.730
0.533
0.679 0.546
EMMS
0.481
0.444
0.706
0.718
0.745
0.621
0.562
0.673
0.740
0.619
0.730 0.639
30


Reviews:
[{"review_id": "h4xXiZrrwc", "reviewer": "Reviewer_kTYc", "summary": "This paper introduces the Efficient Multi-task Model Selector, which utilizes foundation models to convert diverse labels into unified label embeddings. These embeddings are then used to calculate a transferability metric within a weighted linear square regression (WLSR) framework. The proposed method achieves impressive results in multi-task scenarios.", "questions": "No questions.", "limitations": "No questions.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel and unified strategy for model selection. The use of label embeddings enables the capturing of fine-grained semantics, leading to improved estimation in downstream tasks.\n\n2. The effectiveness of the proposed method is supported by a comprehensive set of experiments covering various downstream tasks.\n\n3. The derivation of equations is provided, establishing a solid foundation for the main contribution of the paper.", "weaknesses": "No questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the recognition of our work. We are happy to run more experiments if the reviewer has any pieces of interest."}, {"review_id": "SGBH3VKwqZ", "reviewer": "Reviewer_vJKA", "summary": "This paper introduces EMMS, an efficient multi-task model selector for predicting the performance of pre-trained neural networks on multi-modal tasks without fine-tuning. EMMS employs large-scale foundation models to transform diverse label formats into a unified noisy label embedding. Through weighted linear regression and an alternating minimization algorithm, EMMS accurately estimates transferability. Experimental results demonstrate superior performance and significant speedup compared to existing methods.", "questions": "Please address the comments in weakness section.", "limitations": "n/a", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tGeneric Transferability Estimation Technique: The proposed method, Efficient Multi-task Model Selector (EMMS), offers a generic approach to estimate transferability. By utilizing a unified label embedding derived from foundation models and employing a simple weighted linear square regression (WLSR), EMMS becomes a fast and effective method for assessing the transferability of pre-trained models across different tasks.\n\n2.\tNovel Alternating Minimization Algorithm: The paper introduces a novel alternating minimization algorithm specifically designed to solve the WLSR problem. This algorithm ensures efficient and accurate estimation of transferability within the EMMS framework.\n\n3.\tThe authors did extensive experiments across different tasks and the results demonstrate the efficacy and effectiveness of the proposed method.\n\n", "weaknesses": "1.\tThe proposed method is complicated and difficult to follow. I can barely understand how to use WLSR to maximize the log-likelihood, which may bring extra complexity for re-producing the method.  \n2.\tOne of the simple baselines could be estimating the mutual information between the $\\hat{x}$ and $Z$. I doubt the effectiveness of this baseline, but it could be nice to have.  \n3.\tMore experiment details need to be provided like, what are the foundation models used in different tasks. Another ablation could be studying the effectiveness of different foundation models. I guess CLIP could be the strongest model to use in those tasks if we just use one single foundation model.  \n\nMinor:  \nLine 111, `denote finetuning score`.  \nPlease provide more description of Figure 3 (b) as it is not straightforward to understand the confusion matrix of image captions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions. We list responses to the reviewer's concerns below.\n\n**Q1:** \"The proposed method is difficult to follow...\"\n\n**A1:** We are sorry for the vague presentation of our method. In principle, our EMMS is well established by maximizing the log-likelihood of regression with multiple noisy oracles. We describe the detailed setup as follows.\n\n**First**, EMMS assumes that the true label embedding $z$ is a linear mapping of the model feature with Gaussian noise. Moreover, F-Labels $z_k$ obtained from different foundation models are noisy oracles of $z$.  **Second**, by this setup, we write down the log-likelihood $\\mathcal{L} = \\sum_{(\\hat{x},z_k)}\\log P(z_1,\\cdots, z_K|\\hat{x})$ as given by Eqn. (2) of the main text.  **Third**, with further simplification of Eqn. (2), we find that maximizing the log-likelihood can be approximated as a WLSR problem, i.e. Eqn. (3) of the main text. We sincerely suggest the reviewer see the detailed derivations in Sec. B.1 of the Appendix.\n\nIn addition, we claim that our EMMS is easy to implement and can be used in various multimodal tasks. Despite the complexity of derivations, EMMS turns out to tackle a simple WLSR problem, which can be solved with several lines of code by our algorithm, i.e. Algorithm 2 of the main text. For reference, we make our full code public at https://github.com/anonymous123654/AnonymousEMMS.\n\n**Q2:** \"One of the simple baselines could be estimating the mutual information ....\"\n\n**A2:** Thanks for the suggestion. We agree that it is important to compare our EMMS with the approach based on mutual information modelling. In fact, we have compared our EMMS with TransRate [1] in Table 1 of the main text, which models the relationship between the model feature and the one-hot label by mutual information. From Table 1, we can see that EMMS outperforms TransRate on almost all classification datasets. \n\nTo further validate the efficacy of EMMS, we compare it with TransRate using F-Labels on image classification. We estimate the mutual information of the model feature and F-label following TransRate. Specifically, denote model feature $X \\in R^{N \\times D_1}$, and the F-Label $Z_k \\in R^{N \\times D_2}$, we estimate the mutual information of $X$ and $Z_k$ after the discretization operation for each dimension of $D_2$ and then take average to obtain the final score. \n\nMoreover, we implement two baselines based on TransRate. When $K=1$, we instantiate the F-Label as the CLIP embedding. When $K=3$, we instantiate the F-Labels as the embedding collection extracted from the CLIP, BERT, and GPT-2. In this case, the final score is averaged over three F-Labels. The results are shown in Table A, where we can see that our EMMS consistently outperforms TransRate with F-Labels (both K=1 and K=3). \n\n**Table A.** Comparison of different transferability metrics on image classification regarding $\\tau_w$\n| Method       | Aircraft      | Caltech    | Cars      | CF10      | CF100     | DTD       | Flowers   | Food      | Pets      | SUN    | VOC   | Average   | Sota-All   |\n| ------------ | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |--------- |--------- |\n| TransRate(K=1) | 0.297     | 0.440     | 0.682     | 0.655     | 0.501     | 0.533     | 0.548     | 0.537     | 0.736     | 0.533     | 0.666     | 0.557     | 0-11     |\n| TransRate(K=3) | 0.295     | 0.441     | 0.682     | 0.523     | 0.501     | 0.542     | 0.548     | 0.539     | 0.730     | 0.533     | 0.679     | 0.546     | 0-11     |\n| **EMMS**     | **0.481** | **0.444** | **0.706** | **0.718** | **0.745** | **0.620** | **0.562** | **0.673** | **0.740** | **0.619** | **0.730** | **0.639**     | **11-11**     |\n\n[1] Long-Kai Huan, et al. TransRate. In ICML 2022.\n\n**Q3:** \"More experiment details need to be provided like, what are the foundation models used in different tasks.\"\n\n**A3:** Thanks for the suggestion. We have provided the details about the foundation models used in different tasks in Sec. C of the Appendix. For reference, we present it in Table B. Note that CLIP is ineffective in extracting meaningful embedding for pure text tasks without a dedicated design [1]. We use ELECTRA [2] for text question answering instead of CLIP. Electra is a different type of language model which pre-trains a generator model and shows promising results in various NLP tasks.\n\n[1] Libo Qin et al. CLIPTEXT. In ACL 2023.\n[2] Kevin Clark et al. Electra. ICLR 2020.\n\n**Table B.** Foundation models used in different tasks.\n| Task   | Foundation Models  | \n| -------- | --------- | \n|ImgCls, ImgCap, Referring Expression Comprehension, VQA | CLIP, BERT, GPT-2| \n| Text QA | GPT-2, BART, ELECTRA |\n\n\n**Q4:** \"Another ablation ... I guess CLIP could be the strongest model ... single foundation model.\"\n\n**A4:** We have investigated the effect of different foundation models, including three language foundation models and three multimodal foundation models. Please see more details in [General Response Q1/A1](https://openreview.net/forum?id=2ep5PXEZiw&noteId=kK4DhbM069).\n\n    \n**Q5:** \"More description of Figure 3 (b).\"\n\n**A5:** We are sorry for the insufficient explanations. Fig3(b) shows the correlation between captions from different images. Each image has two captions denoted by the brace. Two captions from the same image often exhibit similarity. Note that the one-hot label for a caption is translated from a vocabulary following [3]. The corresponding entry is 1 if the word corresponds to one index of vocabulary. We see that it is easier for EMMS to encode the semantic relevance of two captions corresponding to the same image than one-hot encoding. We will incorporate it into the final version.\n \n[3] Nan Ding et al. PACtran. In ECCV 2022.\n\nWe hope these experiments help address the concerns of the reviewer. We are happy to run more experiments if the reviewer has any pieces of interest."}, {"review_id": "9lNwLlfTrO", "reviewer": "Reviewer_z1Vu", "summary": "\n\nThis paper proposes to utilize large-scale foundation models for efficient multi-task model selector (EMMS). Concretely, the authors utilize foundation model to transform different label format (category label, text, bounding boxes) into unified noisy label embeddings. EMMS then could measure the compatibility between the models’ features and corresponding\n59 label embeddings via weighted linear regression. Experiments on 5 downstream tasks with 24 multi-modal tasks shows the proposed method's effectiveness and efficiency. \n", "questions": "please refer to the weakness section", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the motivation of this paper is important: investigating how to evaluate foundation model on a set of multi-modal tasks without finetuning all the target tasks.\n\n- the writing is clear and easy to follow.\n\n- the authors also provide code for reproducibility.\n\n- The experiments show improvement on the 5 downstream tasks with 24 datasets.", "weaknesses": "As the authors mentioned in the limitation section, the proposed method is bottlenecked by the capabilities of the chosen foundation model. One could further ask, if the foundation model is good enough, why don't we just use the foundation model for the intended downstream tasks? For example, use CLIP for image classification?\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and helpful suggestions. We have provided a detailed general response to the concerns of all the reviewers. We address the reviewer's concern as follows.\n\n**Q1:** \"As the authors mentioned in the limitation section, the proposed method is bottlenecked by the capabilities of the chosen foundation model. One could further ask, if the foundation model is good enough, why don't we just use the foundation model for the intended downstream tasks? For example, use CLIP for image classification?\"\n\n**A1:** It is a significant question. It is known that foundation models have achieved great success in various tasks. Why do we still need to perform model selection on some conventional models? We have provided some explanations in related work in Sec. 2. Here, we answer this question from three aspects.\n    \n**First, although foundation models have strong generalization ability, it still obtains suboptimal performance on some tasks.** For example, the original paper on CLIP (see Fig. 5) [1] points out that zero-shot CLIP is worse than ResNet-50 on several specialized, complex tasks such as satellite image classification (EuroSAT and RESISC45) and lymph node tumour detection (PatchCamelyon). On the other hand, many existing moderate-size pre-trained models can do some domain-specific tasks very well. Our EMMS is complementary to foundation models, serving as a tool to select an appropriate pre-trained model for target tasks.\n\n**Second, foundation models have amounts of parameters which are computationally expensive and require significant computational resources to train and deploy.** In certain scenarios, such as on mobile devices where computational resources are limited, conventional models that are smaller and faster to train can be more practical and cost-effective.\n\n**Third, model selection is a well-defined challenging task in transfer learning [2,3,4,5], considering that different tasks may have specific requirements.** However, previous approaches [2,3,4,5] mainly focus on classification tasks. In this work, we extend the model selection to multimodal multitask scenarios by label embedding encoded from multiple foundation models, which essentially expands the applicability of the model selection technique. Similar to TaskMatrix.AI [6], we believe that EMMS could be a useful tool for task completion with the help of foundation models.\n \nWe hope these experiments help address the concerns of the reviewer. We are happy to run more experiments if the reviewer has any pieces of interest.\n\n[1] Alec Radford, et al. Learning Transferable Visual Models From Natural Language Supervision. In ICML 2021.\n[2] Kaichao You, et al. LogME: Practical assessment of pre-trained models for transfer learning. In ICML 2021.\n[3] Wenqi Shao, et al. Not all models are equal: Predicting model transferability in a self-challenging\n375 fisher space. In ECCV 2022.\n[4] Yandong Li, et al. Ranking neural checkpoints. In CVPR 2021.\n[5] Nan Ding, et al. PACtran: PAC Bayesian Metrics for Estimating the Transferability of Pre-trained Models to Classification Tasks. In ECCV 2022.\n[6] Yaobo Liang et al. TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIs."}, {"review_id": "MePq6SMLl9", "reviewer": "Reviewer_gNw5", "summary": "This paper focus on an under-explored problem of estimating neural network transferring capability without actually fine-tuning the multi-modal multi-task model on individual downstream tasks. The problem is well-motivated and is of great practical importance. The solution proposed in this paper is straightforward, which essentially is to encode the target label (in the text form) with a foundation model. The embedding of the label text is then treated as the target and a model's transferability can be estimated through a simple weighted linear regression, which is then solved by an alternating minimization algorithm. The paper experiments with 5 downstream tasks and 24 datasets, and show that the proposed estimation method (EMMS) is fast and effective on most tasks. ", "questions": "I'm curious if there are other metrics that can be used to compare the proposed methods with other approaches, e.g. other than $\\tau_w$ and wall-clock time. For the former as I stated in the weaknesses section, it seems that it only cares about relative orders of candidate pre-trained models; for the latter the wall-clock time is easily affected by multiple factors (e.g. multi-threading, concurrent execution of other tasks in multi-core environments, etc.) I would like to learn more about the rationale for choosing these metrics.\nAlso it seems in this paper only the linear mapping is considered and it is used as an assumption without further justification. This may need more clarifications as one can easily come up with alternative approaches for estimate the mapping from features to label embeddings.", "limitations": "This paper has a dedicated section to discuss the limitations and I agree with the authors that the quality of the foundation model (which is mainly responsible for the encoding of the label text) plays critical roles in the effectiveness. However I think there exists some other limitations which are also worth mentioning. Specifically, some important assumptions in this paper is in L164 and L168, where the label embedding is assumed as a linear mapping of the model feature. This assumption seems harsh and I think a more thorough discussion about the assumption will be useful. There is no obvious potential negative societal impact with this paper.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The task is practically important and under-explored in previous work. As the authors mentioned, the previous estimation is usually limited by the classification tasks and the proposed work is more flexible as it directly encodes the label as a text sequence, therefore it can be used for multiple different types of tasks.\n2. The proposed method is intuitive and straightforward, and the algorithms for solving the problem are clear. The theoretical proof looks reasonable although I didn't carefully check all the details in the equations.\n3. The approximation with algorithm significantly speed up the computation when estimating the performance.\n4. This paper performs extensive experimentations by fine-tuning the expensive models on many downstream tasks (to obtain ground truth) and the proposed approach shows superior performance over other methods with regard to correlation and the wall clock speed.", "weaknesses": "1. The Figure 1 bottom is not very informative -- the radar chart is most effective for showing multivariate data with quantitative variables, but in this setting the variables are all binary (applicable/inapplicable), therefore it's actually carrying limited information. Probably replace the figure with a table of checkbox to illustrate the advantage of the proposed approach.\n2. As I mentioned in the limitations, some important assumptions are not validated in the paper. Specifically the key assumption that directly leads to the weighted linear square regression problem and alternating minimization algorithm is the linear mapping from model feature space to the F-label space. More justifications about the assumptions are needed.\n3. I'm also concerned about the choice of evaluation metric of weighted Kendall's $\\tau_w$. If I understand correctly, as a ranking metric it focus more on the relative orders of the elements in the rank, but ignores the actual number. In other words for A>B>C>D, it doesn't care whether A is just marginally higher than B or is much higher than B. It seems wasteful since we already obtain the ground truth downstream task model performance through expensive fine-tuning, but with the $\\tau_w$ metric only the relative orders of the ground truths are useful. Therefore I'm wondering if it's possible to directly estimate the actual performance. I might be wrong or missed some important considerations so any clarifications or rationales are appreciated.\n4. Some minor issues: in the tables with actual results, some numbers are bolded as the best performance, but in many cases there is a tie between two numbers in the table. Probably it's better to highlight both.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and valuable suggestions. We address the reviewer's concern as follows.\n    \n**Q1:** \"The Figure 1 bottom is not very informative ...\"\n\n**A1:** Good advice. We have redrawn a table of checkboxes to illustrate the ability of EMMS. Please see Fig.A and Table A in the attached PDF for more details.\n\n**Q2:** \"... need justification for the assumption of linear mapping ...\"\n\n**A2:** We are sorry for the insufficient justifications. Here we provide more details about the linear assumption. Specifically, EMMS assumes that the true label embedding $z$ is a linear mapping of the model feature with Gaussian noise. This assumption is commonly used in recent methods. For example, LogME [1] assumes that: $z \\leftarrow N(w^T\\hat{x},\\beta^{-1})$, which implies that there is a linear mapping from the model feature space to the label space. PACTran [2] also has a similar setting. \n\nThe difference is that LogME takes a one-hot label as the true label embedding, which limits its applicability. But our EMMS treat the label embedding $z$ as a hidden variable. And F-Labels $\\{z_k\\}_{k=1}^K$ obtained from different foundation models are assumed to be noisy oracles of true label embedding $z$. Since labels in many tasks can be easily encoded into F-Lables, our EMMS can be used as a multitask model selector.\n\nThe linear assumption is reasonable in image and text classification tasks because a linear classifier is usually used when the pre-trained model is transferred to a target task. For tasks of VQA and image captioning, the label in these tasks can be viewed as pure text [3]. By common practice [4,5], these tasks can also be essentially treated as a multi-label classification (vocabulary-based classification). Hence, a linear assumption is also reasonable for these tasks. For the task of referring to image comprehension, we still encode the bounding box label as embedding following [2]. We find that a linear assumption still works well, although the task head is a transformer encoder.\n\nIt is noteworthy that the computation of previous model selection methods is computationally expensive if we direct employ them with vocabulary-based one-hot multiple labels due to the large vocabulary size. F-Labels in EMMS can be deemed as a compressed representation of vocabulary-based one-hot multiple labels as indicated in Fig.3. Due to the simplicity of a linear assumption and informative F-Labels, EMMS can quickly measure the transferability of numerous pre-trained models,  making it possible to be a fast and accurate transferability assessment.\n\n[1] Kaichao You, et al. LogME. In ICML 2021.\n[2] Nan Ding, et al. PACtran. In ECCV 2022.\n[3] Peng Wang, et al. OFA. In ICML 2022.\n[4] Antol et al. VQA. In ICCV 2015.\n[5] Goyal, Y., et al. Making the V in VQA matter. In: CVPR 2017.\n\n**Q3:** \"I'm concerned about the choice of evaluation metric of weighted Kendall's ... if it's possible to estimate the actual performance.\"\n\n**A3:** Model selection is a well-defined problem in transfer learning, which aims to rank pre-trained models and select the top-performing model for the target task. Model selection techniques have been used extensively in classification tasks, such as LEEP, NLEEP, LogME, TransRate, PACTran, SFDA, and so on. The most common metric to measure the performance of a model selection method is the weighted Kendall's tau, i.e. $\\tau_w$. Eqn. (15) of the Appendix gives the definition of $\\tau_w$. In principle, a larger $\\tau_w$ implies the transferability metric can rank pre-trained models better. And if a metric can rank top-performing models better, $\\tau_w$ would also be larger. We also use other measurements, such as Person's coefficient, to assess the performance of transferability metrics in Table 9 of Sec. D in Appendix, where we can see that EMMS still outperforms LogME and other baselines under various measures.\n\nIn addition, we agree that directly predicting the performance of a pre-trained model on the target task can be practical. But it is more challenging than model selection. Recent work [6] tackles this problem by proposing a benchmark consisting of ground-truth evaluations of 35 pre-trained models and 23 datasets. However, it still focuses on classification tasks, and it is unknown how well the prediction technique can be generalized to datasets in the training set. How to design a method to predict the performance on various target tasks requires ongoing efforts.\n\n[6] Orr Zohar et al. LOVM: Language-Only Vision Model Selection.\n\n\n**Q4:** \"Some minor issues: in the tables with actual results, some numbers are bolded as the best performance, but in many cases, there is a tie between two numbers in the table. Probably it's better to highlight both.\"\n\n**A4:** Thank you for the suggestion. We will highlight all the best results in the final version.\n\n\n**Q5:** \"For the latter, the wall-clock time is easily affected by multiple factors (e.g. multi-threading, concurrent execution of other tasks in multi-core environments, etc.\"\n\n**A5:** For the computation complexity, we measure the time for each metric on the same CPU device (AMD EPYC 7H12 with 64-Core Processor) three times and take the average as the final result. To avoid the influence of multi-threading, concurrent execution. We run each model selection method on a single dataset for a single testing case. The code for counting wall-clock time is available at https://github.com/anonymous123654/AnonymousEMMS. In addition, we also compare the theoretical computational complexity between EMMS and LogME in Table C of General Response Q2/A2. We can see that our EMMS not only has lower computation complexity but also enables fully vectorized computation. Please check more details in [General Response](https://openreview.net/forum?id=2ep5PXEZiw&noteId=kK4DhbM069).\n\nWe hope our responses help address the concerns of the reviewer. We are happy to run more experiments if the reviewer has any pieces of interest."}, {"review_id": "rxHsFLXxmy", "reviewer": "Reviewer_4U2K", "summary": "This paper proposed an efficient multi-task model selector (EMMS) to address the inapplicability in a multi-modal multi-task scenario. Specifically, the proposed method achieves a new state-of-the-art in performance and speedup through the incorporation of design elements such as the F-label, Weighted Linear Square Regression, Fast Computation by Alternating Minimization.", "questions": "na", "limitations": "na", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation for addressing “a unified representation to represent diverse label formats” is clearly presented and validated. And use foundation model is an impressive way.\n2.\tThe detailed derivation and experiment in this paper are comprehensive, providing strong evidence of the validity of the proposed model.", "weaknesses": "1.\tIt is recommended to include a complete demo in the code address of the paper, rather than a short python file.\n2.\tThe phrase \"5.13×, 6.29×, 3.59×, 6.19×, and 5.66× speedup in wall-clock time\" appears to describe the speed efficiency of the EMMS (One) method. Could you clarify why EMMS's performance varies, being faster than LogME in some instances and slower in others, as indicated in Table 1?\n3.\tIn Table 1, there seems to be some ambiguity about the effect of F-label on transferability assessment, since the perforamnce of the EMMS(one) and EMMS is identical in some rows.\n4.\tIt is recommended that the paper include a broader array of foundational models for F-label. Is CLIP currently the model with the highest effects?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the useful and helpful comments on our manuscript.  We address the reviewer's concern below.\n\n**Q1:** \"It is recommended to include a complete demo in the code.\"\n\n**A1:** Thanks for your suggestion. We have open-sourced complete code at https://github.com/anonymous123654/AnonymousEMMS.\n\n**Q2:** \"The phrase \"5.13×, ... in wall-clock time\" appears to ... EMMS (One) method. Could you clarify why EMMS's performance varies, being faster than LogME in some instances and slower in others, as indicated in Table 1?\"\n\n**A2:** It is true that we report the results (both $\\tau_w$ and wall-clock time) of EMMS(one) in Table 1 for classification tasks. EMMS(one) is a variant of EMMS which replaces the F-Label with a one-hot label. Hence EMMS(one) can only be used in classification tasks. In image classification, we prefer to use EMMS(one) rather than EMMS because EMMS(one) achieves a good trade-off between effectiveness and efficiency. To see this, we list the computational complexity of LogME, EMMS(one), and EMMS in Table A. \n\nFrom Table A, we see that EMMS(one) have lower computational complexity than LogME. Moreover, our EMMS(one) allows for full vector computation and can be efficiently solved by existing scientific computation packages such as np.linalg.lstsq in our implementation. Nevertheless, LogME cannot be written in fully vectorized form because the optimization of model parameters of LogME are highly coupled.\n\nIn addition, EMMS usually has higher computation complexity because $D_2 \\gg C$. In some cases, when the number of categories $C$ and the iteration number $T$ are large, EMMS could be faster than LogME with vector computation. For example, we find that $C=397$ and $T=4.46$ on average over all models when LogME is convergent on the Sun397 dataset. It results in higher time complexity than LogME, as indicated in Table A. We further verify this by implementing LogME with $T=1$. As shown in Table B in the attached PDF, EMMS spends more time in calculating the transferability than LogME (T=1) on all datasets. However, LogME performs much worse than EMMS because it does not converge when $T=1$.\n\n**Table A.** The comparison of computational complexity between LogME, EMMS(one), and EMMS in image classification. We denote model feature $X \\in R^{N \\times D_1}$, F-labels $Z\\in R^{N\\times D_2\\times K}$, and one-hot label $Y\\in R^{N\\times C}$ with $N \\approx 10^4$, $D_1 \\approx 10^3$, $D_2 \\approx 10^3$, $K=3$, and $C\\approx 10^2$. Moreover, $T$ denotes the iteration number of the inner loop in LogME with $T \\approx 3$.    \n| Method                             | LogME                             | EMMS（One）                       | EMMS                                                         |\n| --------------------------------- | --------------------------------- | --------------------------------- | ------------------------------------------------------------ |\n| **Complexity** | $3TCD_1^2 + (2T+1)NCD_1 + D_1^3 + ND_1^2$ | $CD_1^2 + NCD_1 + D_1^3 + ND_1^2$ | $ND_1^2 +2ND_1D_2 + D_1^3 + D_1^2D_2 + (K^2+K)(ND_2) + K^3 + K^2 +K\\log K$ |\n| **Simplified Complexity** |  $3TCD_1^2 + ND_1^2+ND_1C(2T+1)$ | $ND_1^2$  | $ND_1^2 + 2ND_1D_2$  |\n| **Vector Compuration** |  ✗ | ✔  | ✔  |\n\n\n**Q3:** \"Table 1, there seems to be some ambiguity about the effect of F-label on transferability assessment, since the performance of the EMMS(one) and EMMS is identical in some rows.\"\n\n**A3:** Thanks for the question. We claim that F-Label is the key ingredient of our EMMS. We demonstrate the effectiveness of F-Label in the following.\n\nF-Label has evident advantages over one-hot labels from our experiments.  First, from Table 1 of the main text, EMMS (with F-Label) outperforms EMMS(One) on 5 datasets but performs worse than EMMS(One) on only 1 dataset. Moreover, EMMS (with F-Label) has higher $\\tau_w$ than EMMS(one) on average over 11 datasets in total. Second, we have compared the effectiveness of F-Label and one-hot label when selecting pre-trained CNN-based models in Table 10 of the Appendix. We see that EMMS (with F-Label) outperforms EMMS(one) on 7/11 datasets when a single F-Label is used and outperforms EMMS(one) on 9/11 datasets when multiple F-Labels are used. These results demonstrate the effectiveness of F-Labels.\n\nIn addition, we also show that F-Label encodes richer semantic information than one-hot labels. Hence, EMMS has an advantage in encoding semantic labels with a large category size in image classification. For instance, EMMS outperforms EMMS(one), such as CF100 (EMMS 0.736 v.s. EMMS(one) 0.745), Food (EMMS 0.579 v.s. EMMS(one) 0.673) and Sun397 (EMMS 0.592 v.s. EMMS(one) 0.619) in Table 1 of the main text. Besides, F-Label unifies diverse forms of labels in different tasks and facilitates multitask model selection. As shown in Table 2-5 of the main text, EMMS is a good model selector in image captioning, VQA, TQA, and referring expression comprehension.\n    \n**Q4:** \"It is recommended that the paper include a broader array of foundational models for F-label. Is CLIP currently the model with the highest effects?\"\n\n**A4:**  We have investigated the effect of different foundation models, including three language foundation models and three multimodal foundation models. Please see more details in [General Response Q1/A1](https://openreview.net/forum?id=2ep5PXEZiw&noteId=kK4DhbM069).\n\nWe hope our responses help address the concerns of the reviewer. We are happy to run more experiments if the reviewer has any pieces of interest."}]

Author rebuttal:
# General Response
We thank the reviewers for their detailed reviews and thoughtful suggestions on our work. 

In general, two main concerns are raised, including (1) the effect of using a single foundation model and (2) the computational complexity of EMMS. To address the reviewers’ concerns, we try our best to perform additional experiments.

**Q1: The effect of using a single foundation model to extract F-Label. Is CLIP the best model?**

**A1:** We investigate the effect of using a single foundation model. We conduct experiments on image classification and image captioning. We consider EMMS with the single foundation model including language foundation model (1) GPT-2, (2) BERT, (3) RoBerta, and multimodal foundation model (4) CLIP, (5) FLAVA, and (6) AltCLIP. For comparison, we include the result of our EMMS with default setting (K=3, i.e. CLIP, BERT, and GPT-2) and the result of previous state-of-the-art methods obtained from LogME, NLEEP and TransRate. The results are reported in Table A and Table B.

We have several observations. (1) Different downstream tasks prefer  F-Labels obtained from different foundation models. No single foundation model is dominant in all target tasks. In particular, CLIP is not the best model for extracting F-Label. (2) For image classification, both language and multimodal foundation models are competent for acquiring F-Labels. For image captioning, multimodal foundation models are more appropriate for extracting F-Labels than language foundation models. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.



**Table A**: The effect of the single foundation model in image classification.
| Method   | Aircraft | Caltech | Cars      | CF10      | CF100     | DTD   | Flowers | Food      | Pets  | Sun397    | Voc2007   | Average   | Sota/All   |
| -------- | -------- | ------ | --------- | --------- | --------- | ----- | ------- | --------- | ----- | --------- | --------- | --------- | --------- |
| Previous SOTA | 0.299    | 0.412  | 0.693     | **0.741**     | 0.736     | **0.621** | **0.655**   | 0.580     | 0.707 | **0.619**     | 0.651     | 0.610    | 4/11     |
| (1) GPT2     | **0.481**    | **0.463**  | 0.448     | 0.652     | **0.745**     | **0.621** | 0.562   | 0.652     | **0.740** | 0.616     | **0.730**     | 0.610     | 6/11     |
| (2) Bert     | **0.481**    | 0.444  | 0.458     | 0.718     | **0.745**     | **0.621** | 0.562   | 0.592     | **0.740** | 0.616     | **0.730**     | 0.609     | 5/11     |
| (3) RoBerta  | 0.448    | 0.444  | 0.507     | 0.701     | **0.745**     | 0.608 | 0.562   | 0.580     | **0.740** | 0.574     | **0.730**     | 0.604     | 3/11     |
| (4) CLIP     | **0.481**    | 0.444  | 0.496     | 0.608     | 0.720     | **0.621** | 0.562   | 0.558     | **0.740** | 0.616     | 0.706     |  0.595     | 3/11     |
|(5) FLAVA     | **0.481** | 0.444 | 0.508     | **0.741** | **0.745** | **0.621** | 0.562 | 0.652     | **0.740** | 0.574     | 0.706     |  0.615    |  5/11    |
|(6) AltCLIP   | **0.481** | 0.444 | 0.437     | **0.741** | **0.745** | **0.621** | 0.562 | 0.580     | **0.740** | 0.595 | **0.730** |  0.607    |  6/11    |
| **EMMS (ours)**     | **0.481**    | 0.444  | **0.706** | 0.718 | **0.745** | **0.621** | 0.562   | **0.673** | **0.740** | **0.619** | **0.730** | **0.639** | 8/11 |

**Table B.** The effect of the single foundation model on EMMS in image captioning.
| Method        | Flickr8k  | Flickr30k | RSICD | flickr10kH | flickr10kR | Average   | Sota/All   |
| ------------- | --------- | --------- | ----- | ---------- | ---------- | ---------- | ---------- |
| LogME（CLIP） | 0.530     | 0.393     | 0.618 | 0.764    | 0.634      | 0.588    |   0/5    |
| (1) Gpt2          | 0.566     | 0.393     | 0.431 | 0.715      | 0.618      | 0.545      |  0/5     |
| (2) Bert          | 0.395     | 0.319     | 0.448 | **0.802**      | **0.711**      | 0.535      |  2/5    |
| (3) Roberta  | 0.346     | 0.111     | 0.587 | 0.571      | 0.566      | 0.436      | 0/5     |
| (4) CLIP      | 0.510     | 0.448     | **0.704** | **0.802**      | 0.678      | 0.628      | 2/5      |
| (5) FLAVA   | 0.463  | 0.382   | 0.693 | 0.704   | 0.678   | 0.584   | 0/5 |
| (6) AltCLIP    | 0.453     | 0.448     | 0.623     | **0.802**      | 0.678      | 0.601   | 1/5 |
| EMMS          | **0.660** | **0.504** | **0.704** | **0.802**      | 0.678      | **0.670**      | **4/5**|

**Q2: The computational complexity of EMMS.**

**A2:** We compare the computational complexity between LogME and EMMS in Table C. We see that EMMS has lower computation complexity than LogME because LogME needs several iterations (T=3 on average) to converge. Moreover, EMMS allows for full vector computation and can be quickly solved by existing packages such as numpy.linalg.lstsq. But LogME cannot be written in fully vectorized form as the model parameters in LogME are highly coupled. Hence, LogME needs to be executed in a while loop.

**Table C.** The comparison of computational complexity between LogME and EMMS. LogME is fed with F-Label. We denote model feature $X \in R^{N \times D_1}$ and F-labels $Z\in R^{N\times D_2\times K}$ with $N \approx 10^4$, $D_1 \approx 10^3$, $D_2=1024$, $K=3$, and $C\approx 10^2$. Moreover, $T\approx 3$ denotes the iteration number of LogME.
| Method                             | LogME                             |               EMMS                                                         |
| --------------------------------- | --------------------------------- | ------------------------------------------------------------ |
| **Complexity** | $3TD_1^2D_2+(2T+1)ND_1D_2+D_1^3+ND_1^2$  | $ND_1^2+2ND_1D_2+D_1^3+D_1^2D_2+(K^2+K)(ND_2)+K^3+K^2+K\log K$ |
| **Simplified Complexity** |  $3TD_1^2D_2+ND_1^2+ND_1D_2(2T+1)$  | $ND_1^2+2ND_1D_2$  |
| **Vector Compuration** |  ✗   | ✔  |

We are happy to run more experiments if the reviewer has any pieces of interest.

Meta-review: The paper addresses the challenge of predicting the performance of pre-trained neural networks on various multi-modal tasks without the need for fine-tuning. The authors present the Efficient Multi-task Model Selector (EMMS) which employs foundation models to unify diverse label formats into a cohesive label embedding. Main concerns are the complexity of the method and the reliance on foundation model capabilities. After discussion, main concerns are addressed. Given the general quality of the paper and the feedback, I recommend acceptance.
Decision: Accept (poster)