Model Spider: Learning to Rank Pre-Trained Models Efficiently
Authors: Yi-Kai Zhang
Ting-Ji Huang
Yao-Xiang Ding
De-Chuan Zhan
Han-Jia Ye
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Pre-trained Model Ranking
Transfer Learning
TL;DR: We propose Model Spider to learn how to tokenize both Pre-Trained Models (PTMs) and tasks. Model Spider accelerates selecting a helpful PTM from visual models or Large Language Models (LLMs) for a downstream task and improves the PTM ranking quality.

Abstract: Figuring out which Pre-Trained Model (PTM) from a model zoo fits the target task is essential to take advantage of plentiful model resources. With the availability of numerous heterogeneous PTMs from diverse fields, efficiently selecting the most suitable one is challenging due to the time-consuming costs of carrying out forward or backward passes over all PTMs. In this paper, we propose Model Spider, which tokenizes both PTMs and tasks by summarizing their characteristics into vectors to enable efficient PTM selection. By leveraging the approximated performance of PTMs on a separate set of training tasks, Model Spider learns to construct representation and measure the fitness score between a model-task pair via their representation. The ability to rank relevant PTMs higher than others generalizes to new tasks. With the top-ranked PTM candidates, we further learn to enrich task repr. with their PTM-specific semantics to re-rank the PTMs for better selection. Model Spider balances efficiency and selection ability, making PTM selection like a spider preying on a web. Model Spider exhibits promising performance across diverse model zoos, including visual models and Large Language Models (LLMs). Code is available at https://github.com/zhangyikaii/Model-Spider.

MODEL SPIDER: Learning to Rank
Pre-Trained Models Efficiently
Yi-Kai Zhang', Ting-Ji Huang', Yao-Xiang Ding', De-Chuan Zhan', Han-Jia Yel.es National Key Laboratory for Novel Software Technology. Nanjing University, China 2State Key Lab of CAD & CG, Zhejiang University "iiaanyk,huangge zhandc, y jehj amladda nju. edu. en yxdingezju.edi edu.
Abstract
Figuring out which Pre-Trained Model (PTM) from a model zoo fits the target task is essential to take advantage of plentiful model resources. With the availability numerous heterogeneous PTMs from diverse fields, efficiently selecting the most suitable one is challenging due to the time-consuming costs of carrying out forward backward passes over all PTMs. In this paper, we propose MODEL SPIDER, which tokenizes both PTMs and tasks by summarizing their characteristics into vec- tors to enable efficient PTM selection. By leveraging the approximated performance PTMs on a separate set training tasks, MODEL SPIDER learns to construct representation and measure the fitness score between a model-task pair via their representation. The ability to rank relevant PTMs higher than others generalizes to tasks. With the top-ranked PTM candidates, with their PTM-specific semantics to re-rank the PTMs for better selection. MODEL SPIDER balances efficiency and selection ability, making PTM selection like spider preying on a web. MODEL SPIDER exhibits promising performance across diverse model zoos, including visual models and Large Language Models (LLMs). SSider
further learn enrich task repr.
Code is available at https:// //tp:/b//.ttttt./// com
Introduction
Fine-tuning Pre-Traine Trained Models (PTMs) on downstream tasks has shown remarkable improvements in various fields [35 26 75
making "pre-training - fine-tuning" the de-facto paradigm in many real-world applications. A model zoo contains diverse PTMs in their architectures and func- tionalities [2]. but randomly selected PTM makes their helpfulness for particular downstream task vary unpredictably [80 77 102). One important step to take advantage of PTM resources is to identify the most helpful PTM in model zoo estimating and ranking the transferabilities of PTMs with the downstream task's 's data accurately and efficiently.
42 16,
Which PTM is the most helpful? A direct answer is to enumerate all PTMs and evaluate the performance of their corresponding fine-tuned models. However, the high computational cost of the backward steps in fine-tuning makes this solution impractical. Some existing methods estimate proxies of transferability with only forward passes based on the target task's 's features extracted by
25 [3]. Nowadays, public model zoo often contains hundreds and thousands of PTMs [104]. Then, the computational burden of forward passes will be amplified, let alone for the time-consuming forward passes of some complicated PTMs. Therefore, the efficiency of searching helpful PTMS and estimating the transferability should be further emphasized.
PTMs 9 97 66. (55] [13 27
In this paper, propose MODEL SPIDER, the SPecification InDuced Expression and Ranking of PTMs, for accurate and efficient PTM selection. In detail, we tokenize all PTMs and tasks into vectors that capture their general properties and their relationship with each other. For example,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Transferaiility Ranking
Transferability Ranking
Similarity Measure
Correlation (t.)
0.7
LogME
Fectovee Light
Pacterr?
0.a
2-
Leamed Repc
A--LEP
Forwardlesdh Pre oodkedd Mode!!
Zoo
Tasks
Tasks
Time (muttiple of acceleration)
(a) Forward Corward-Based and Representation,Speeffettt Based Model Selection
(b) Selection ability & Efficiency Comparison
Figure 1: (a) Two strategies for PTM selection. Related works utilize forward-bass features and corresponding proxies on the target dataset to evaluate transfcrability. The prrrrseettatio/spccffcitt based approach with learned model-task pair reduces the requirement for forwarding pass each PTM. (b) The average efficiency (wall-clock time) vs performance (correlation Tw. the higher, the better) comparison of PTM selection. The circle sizes indicate the memory footprint. Red circles MODEL SPIDER with different values the number of PTM--pecific features k while others are comparison methods. MODEL SPIDER balances efficiency and accuracy well.
two models pre-trained on NABirds 37 and Caltech-UCSD Birds [00] datasets may have similar abilities in bird recognition. The comprehension abilities of two models pre- trained XSum [64] dataset, Ax-b, and Ax-g datasets of SuperGLUE benchmark [0I] may also be mutually transferable. We then associate them with similar representation. Then the transferability from PTM to task could be approximated by the distance of their repr. without requiring per-PTM forward pass over the downstream task. The success of MODEL SPIDER depends two key factors. First, how do we obtain representation for tasks and PTMs? The representation of the most helpful PTM should be close to the task one w.r.t. some similarity measures. Then, will general task repr. weaken the selection ability since it may ignore specific characteristics of a PTM?
In MODEL SPIDER, learn construct representation with general encoder and measure the similarity between them with a Transformer module [98] in a supervised learning manner. We estimate the rankings of PTMs in the model zoo for some historical tasks using rank aggregation. By leveraging the approximated supervision, we pull task representation close to the top-ranked PTM repr. and push unhelpful PTM repr. away based on the transformer-measuee expect that the ability to tokenize and measure similarity could be generalized to unseen tasks. The difference between MODEL SPIDER'S representation- PTM selection with forward-based strategy is illustrated in Figure
similarity. We
The representation generated by general encoders significantly reduces the PTM search time and improves the search performance. If the budget allows, we can extract features the downstream task by carrying out forward passes over part of (the top-k ranked) PTMs, revealing the specific relationship between PTMs and the task. We equip our MODEL SPIDER with the ability to incorporate PTM--specffic representation, which re-ranks the PTMs and further improves the selection results. In summary, MODEL. SPIDER suitable for different budget requirements, where the general and task-specific repr. makes flexible trade-off between efficiency and accuracy, given various forward passes. Figure iillustrates a comparison of PTM selection methods w.z.. both efficiency and accuracy, Our contributions are
We propose a novel approach MODEL SPIDER to tokenize tasks and PTMs, which is able to rank PTMs in model zoo given downstream task efficiently and accurately.
MODEL SPIDER learns to tokenize and rank PTMs on separate training set of tasks, and it can
incorporate task-specific forward results of some PTMs when resource budgets allow.
The experiments demonstrate that MODEL. SPIDER effectively ranks P'T's and achieves significant improvements on the visual models and the Large Language Models (LLMs).
2
Related Works
Efficient PTM Search with Transferability Assessment. Whether selected PTM is helpful could be formulated as the problem measuring transferability from source data pre-training the PTM to the target downstream task 41 4 78). The current evaluation of transferability relies

---

forward pass of the PTM on the target task, which generates the PTM PTM-specific features on the target
task. For example, NCE 197. LEEP [6]. LogME TE 114). PACTran [27, and TransRate [39 estimate negative conditional entropy. log expectation, marginalized likelihood, PAC-Bayesian bound, mutual information to obtain proxy metric of transferability, respectively. Several extensions including V-LEEP divergence transition matrix to approximate the transferred log- legliikeiiood, and [25] 71
(55] with Gaussian mixture model on top of PTM features, H-Score 9 utilizing
84] exploring
correlations between categories of target task. Auxiliary information such as source clues 66 93 and
gradients of PTMs when back propagating with few steps 85 74 are also investigated. Although
the transferability assessment methods avoid the time-consune consuming fine-tuning, the forward costs over PTMs also become heavier given diverse and complicated pre-trained model zoos.
Relatedness of Task. Whether a PTM gains improvements after the-tuning on the downstream task has been verified to depend on the relatedness between tasks both theoretically [00 60)
and empirically [102 58 112]. The relatedness could be measured through various ways, such as
fully fine-tuning [115. task vectors 2 example-based graphs 48
29 86], representation-level
similarities 330 3, and human prior knowledge [4 [6.. Instead of utilizing pre-defined strategy
to measure the relatedness, MODEL SPIDER constructs the representation of PTMs/tasks in vector
forms and learns similarity between them historical tasks.
Learning to rank predicts the orders of objects usually with a score function 43 and the experience on a training set could be generalized to unseen data [3] 63) Additional learned metrics or embeddings further improve the ranking ability 62 L5] The task relatedness can also be modeled a baaning-to-rant, problem, where the preference over one PTM over another could be learned from historical rankings of PTMSs However, obtaining the supervision on the training set requires complete fine-tuning over a large number of historical tasks, which either come from a time-consuming transfer learning experience [103] or the output from some specially selected transferability assessment methods (28). We propose strong and efficient approximation of the PTM ranking supervision on the training set tasks, and a novel representation-bee based similarity is applied.
Preliminary
We describe the PTM selection problem by assuming all PTMs are classifiers, and the description could be easily extended to PTMs for other tasks, e.g. regression. Then we discuss several solutions.
3.1 Selecting PTMs from Model Zoo
Consider have target classification task T = (xi, Di) 1:) with N labeled examples, where the label y; of each instance 2; comes from one of the Cy classes. Instead of learning on T directly,
we assume there is a model zoo M = {f = W., Φm p containing M PTMs. A PTM fm could be decomposed into two components. Φm is the feature extraction network producing dmn- dimensional features. Wm E Rd XCC. is the top-layer classifier which maps d,m -dm-dimnnio feature to the confidence score over Cm classes PTMMs in M are trained on source data across various domains. Their feature extractors Φm have diverse architectures, and the corresponding classifiers are pre-trained for different sets of objects. In other words, dm and C.,' may differ for a certain pair of m and m'' A widely-used way to take advantage of PTM f' Wm Φm in the target task is to fine-tune the feature extractor together with randomly initialized classifier over T. In detail, we minimize the following objective
f = W o arg f=Woo
Σ e(W ((xx), Vi
pm))
(I)
where & is initialized with om. The fine-tuned f makes prediction with arg max (cr v & (x). [C] = [1...., C} and w. is the cth column of W. Then, we can rank the helpfulness of PTMs based on the performance of their fine-tuned models. other words, we obtain fm following Equation based on the math PTM fmx then calculate the averaged accuracy when predicting over unseen test set of T (the higher, the better), i.e.,
L T
E
[ arg max fmn (x)))))
(2)
'e omit the bias term for simplicity.
3

---

E is also named as the transferability, measuring if the feature extractor 6m in a PTM could be transferred well to the target task with fine-tuning 197 39
I(.) the indicator function, which
outputs 1 if the condition is satisfied. Given Ł = {Leb... +7 m==1 i.e., the transferability for all PTMs, then we can obtain the ground-truth ranking of all PTMs the model zoo for task and select the top-ranked one. In the PTM selection problem, the goal is to estimate the ranking of all
M
PTMs for task T using Ły ff -m- The evaluation criterion is the similarity between the
predicted ε and the ground-truth By, typically measured by weighted Kendall's Tw [5]. We omit the subscript T when it is clear from the context.
3.2 Efficiency Matters in PTM Selection
One direct solution to PTM selection is approximating the ground truth ty by fine-tuning all the PTMs over T, where validation set should be split from T to estimate Eqqaaion 2| Since fine-tuning PTM contains multiple forward and backward passes, the computation burden is astronomical.
forward pass of certain PTM's extractor Φ... over T generates the features Φ {( (ar4)
Rd (=..):)TT. which is lightweight compared with the backward step. The feature reveals how examples in T are distributed from the selected PTM's view, and a more discriminative feature may have a higher transfer potential. As mentioned in section 2] the existing transferability assessment methods estimate -T. based on the PTM-specifiic feature h and target labels {y/ } [66] [13] 55 [14. Precise estimation requires large N, which means we need to collect enough examples to identify the most helpful PTMs from model zoo.
While the previous forward-based transferability assessment methods reduce the time cost, selecting among M PTMs in the model zoo multiplies the forward cost M times, making the estimation of f computationally expensive. Moreover, since forward passes for complicated PTMs take longer, selecting PTM efficiently, especially given large model zoo, is crucial.
MODEL SPIDER
In MODEL SPIDER, we propose to tokenize PTMs and tasks regardless of their complexity, allowing us to efficiently calculate their relatedness based certain similarity measure over their representation, which capture general properties and serve as specification of model or task, demonstrating which kinds of tasks a model performs well on or what kind of models a task requires. In this section, we first introduce the process of obtaining repr. by learning from a training set of tasks, and the ability to rank PTMs could be generalized to downstream tasks. We then describe the encoder, the similarity measure, and an efficient way to generate supervision during representation learning. Finally, we discuss how MODEL SPIDER can be flexible in incorporating forward pass results of top-ranked PTMs to further improve the representation' 's semantics and the ranking's 's quality.
4.1 Learning to Rank PTMs with Representation
In MODEL. SPIDER, learn the model repr. (0m JM task repr. fa (T), and the similarity measure sim(s.) in a supervised learning manner based on a separate training set D. The training set does not contain overlapping classes with the downstream task T.
Specifically, we randomly sample training tasks (T.} from D. For a given training task T, we assume that we can obtain the ground-truth ranking ty, (t&m., -T, m== over the M PTMs, indicating the helpfulness of each PTM. We will discuss the details of obtaining the supervision ty; later. We then select PTMs for T; based the similarity between the task repr. çu (Tx) and those M PTM repr. {0, 1M We expect the higher the similarity, the more helpful a PTM is for the given task. We use ④ to denote all learnable parameters and optimize O with ranking loss, which minimizes the discrepancy between the rank ty. predicted by the similarity function and the ground-truth t7,:
Lrank (fr, [sim(0m. / (Ti)) Ar tT..) (T
min
TOND
Given t. E R.M, we use an operator dsc(-) to index the elements of in a descending order, i.e. V m < L, we have tdeo(m) 2 Edsc(()) dsc (m) is exactly the index of the PTM with mith largest

---

Model Beprs.
Model Space
Model- Similarity Measure
Recatmee-ssaee sased
4
sind, repe
task
(ximm
A
Task Repr.
project to
Aunkong
Geowrn Extsedor
PTM Specific Task
Approximated Supervision.
on faniaaalet Stuts
frank
B
Reninng
[Optional]
Loght leriag
An illustration of MODEL SPIDER. The middle part (b) shows the workflow of MODEL SPIDER, which involves tokenizing both PTMs and tasks into shared space. Plot (c) demonstrates how the model-task similarity calculated based on the representation helps rank PTMs for given task. In plot (a), when the budget allows, MODEL SPIDER can take advantage of PTM-specific features obtained by performing forward passes of the top-k ranked PTMs on some selected tasks. This improves the quality of task repr. as well as the PTM ranking.
Figure 2:
ground-truth score. Based on this, we use the following ranking loss:
M
exp (Lise((o))
Flank(,,))
log
(4)
E exp (Lase(t)
Equation aims make the whole order of the predicted br; similar the ground-truth LT:- So the similarity between the task repr. and that of a higher-ranked PTM indicated by ly, should be larger than the similarity with lower-ranked PTM representation. The underlying intuition is that if PTM performs well on certain tasks, it is likely to generalize its ability to related tasks. For example, if a PTM excels at bird recognition, it may effectively recognize other flying animals.
For a downstream task T, we generate its task repr. with (T)).. and identify the close PTM ones with the learned -). Objective Equation ObjectivvEEquation 3 also works when the number of examples in task
small. By learning to rank PTMs for sampledfew-shor few-shot tasks, MODEL SPIDER can rank helpful models even with limited training data. We will show this ability of MODEL SPIDER secscciion
4.2 Model and Task Representation for PTM Selection
We encode the general characteristics of tasks and PTMs via two types of representation.
Model Representation. Given a model zoo with M PTMs, we associate PTM fm with form 0m E Rd encoding rich semantics about the aspects in which fm excels. Models pre-trained from related datasets or those with similar functionalities are expected to have similar representation.
Task Representation. A -C-sslas task {Zx, y/) | contains would like to tokenize a task with a mapping (u()). which outputs a set of vectors μ (T) RdxCr one for each class. We implement f with one additional frozen encoder & with an equivalent parameter magnitude as the PTMS in the model zoo. to pre-trained by self-supervised learning methods [17 33 (53 and captures the semantics of a broad range of classes. In detail, we extract the features of all instances in the task T and take the class centers as the task repr.:
of instances and labels. We
(T)
(bb (w:) [y:
c)]
(5)
[
oE(C)
The task repr. expresses the characteristies of task, e.g., those tasks with semantically similar classes may have similar sets of representation.
Model-Task Similarity. The helpfulness of PTM w.r.t. task, i.e., the transferability score, could be estimated based on the similarity of the model-task pairs - sim μ (T)), and the PTM selection is complemented by embedding the model and tasks into space and then identifying close PTM repr. for task. In MODEL SPIDER, the sim(:-- is implemented with one-layer Transformer 198). a self-attention module that enables various inputs. The Transformer consists of alternating layers of multi head self-an olffaattenionn multi-layer perceptron, and layer norm blocks. We set the input of the Transformer as the union set of model and task repr. 1m. / (T) e (1+())
5

---

then the similarity f -T between model and task ones is:
sim 14 (T)) FC (transformer (z) [0]
(6)
where [0] is the first output of the Transformer, i.e., the corresponding output of the model representa- tion. We add Fully Connected (FC) layer project the intermediate result to scalar. Learnable
parameters O including (0m) J FC, and weights of the Transformer, are trained via objective
in Equation 3
4.3
Accelerating Training for MODEL SPIDER
The training of MODEL SPIDER in Equation 3 requires a large number of (task Tú PTM ranking (T) pairs. Although we could collect enough data for each task, obtaining the ground-truth PTMs rankings, i.e., the helpfulness order of PTMs for each task, computationally expensive. addition, using some proxies of tr, may weaken the ability of the MODEL SPIDER. We propose a closer approximation of the ground -round-truthh truth tr,, which efficiently supervises sampled tasks from D.
Approximated Training Supervision. We take advantage of the fact that existing PTM selection methods rely the PTMM-peccfic features T7 estimate the transferability score w.r.t. T; and produce diverse scores. In other words, PTM will be placed in different positions based on the scores provided by various methods such as NCE 197), LEEP [66), and LogME [13(114] Based on their "relatively good but diverse" ranking results, an intuitive approach to estimate the ground-truth by; is ensemble their multiple ranking results into a stronger single order.
Given (t........... multiple predicted rankings over M PTMs for sampled task Tr, i.e., the order sorted by the estimations of transferability via various methods, we take advantage of Copeland's 's ag- gregation method 77 [22 to ensemble the orders: ty, = ( | = RankAgg ((t), .).. Copeland's aggregation compares each pair of ranking candidates and considers all preferences to determine which of the two is more preferred. The output ł7, acts as a good estimation of the ground-truth supervision tr,. The aggregated टT. more accurate than a particular transferability assessment method, which improves the quality of the supervision in ranking loss in Equation 4
Sampling Tasks for Training. We assume that the training data DD contains large number of classes with sufficient data. To sample tasks for training, we randomly select a set of classes from D and choose a subset of their corresponding examples. Benefiting from the supervision estimation approach RankAgy, we are able to obtain the aggregated ranking for any sampled task.
Training Complexity. The training phase in MODEL SPIDER is efficient. First, pre-extract LM features ( mm=1 for D with all PTMs in advance. Then only the computational burden of base transfcrability assessment methods, rank aggregation methods, and the optimization of top-layer parameters are involved. Furthermore, training tasks with the same set of classes share the same ły.-
4.4 Re-ranking with
Trade-off'
The learnable model representation captures the PTM's empirical performance on various fields of training tasks, which decouples the task repr. from the PTM. Each model repr. implicitly expresses the field in which the PTM excels, the PTM selection only requires a task repr. express the field in which the task is. In contrast to the general task repr. (T.)),, PTMM-ppecfiic features T%) for a subset of PTMS provide rich clues about how those PTTs fit the target examples, which are also used in related transferability assessment approaches (1). We claim that given specific features with a subset of PTMs when the budget is available, our MODEL SPIDER can re-rank the estimated PTM order and further improve performance.
Specifically, extract the PTM--pecifii task repr. pm (T) E Rd x Or with the specific features T of the mth PTM as Equation 5] To take account of different values of dm due to the heterogeneity of PTMs, we learn a projection P E dR xd for the mith PTM to align the dimensionality of pm (T) with the model representation. We then replace the general task repr. / (T) via the specific one P pm (T) when calculating the similarity with the repr. Om of the mth PTM. The specific task repr. may facilitate obtaining more accurate estimations. During the training process, we dynamically select partial set of PTMs and incorporate the specific repr. into the sampled tasks. Thus, the same Transformer module in Equation 6|can deal with the new type of representation. To differentiate the general and specific representation, we learn two additional d-dimensional embeddings prompts.

---

Table Performance comparisons of baseline approaches and MODEL SPIDER on model zoo with 10 PTMs [13]. We measure the performance with Kendall's 45 weighted Tw. The downstream tasks from diverse fields (8 datasets) are evaluated in a standard manner (all training examples) and few-shot manner (10 examples per class and 30 trials). Specific features of top-3 ranked PTMs are used MODEL SPIDER. We denote the best-perforning results in bold.
Downstream Target Dataset
Method
Mean
Aireraft Ca Cars CIFARIO CIFAR100 DTD
Pets
SUN397
Standard Evaluation
H-Score
0.328
0.738
0.616
0.797
0.784
0.395
0.610
0.918
0.648
NCE D7
0.501
0.752
0.771
0.694
0.617
0.403
0.696
0.892
0.666
LEEP 666
0.244
0.014
0.704
0.601
0.620
-0.111
0.680
0.509
0.408
W-LEEP [55
-0.725
0.599
0.622
0.768
0.776
0.074
0.787
0.730
0.454
LogME 113
0.540
0.666
0.677
0.802
0.798
0.429
0.628
0.870
0.676
PACTran
27
0.031
0.200
0.665
0.717
0.620
-0.236
0.616
0.565
0.397
OTCE 9
-0.241
-0.011
-0.157
0.569
0.573
-0.165
0.402
0.218
0.149
LFC 25
0.279
-0.165
0.243
0.346
0.418
-0.722
0.215
-0.344
0.034
GBC
-0.744
-0.055
-0.265
0.758
0.544
-0.102
0.163
0.457
0.095
MODEL SPIDER
0.506
0.761
0.785
0.909
1.000
0.695
0.788
0.954
0.800
Few-Shot Evaluation (10-example per class)
H-Score
-0.014
0.078
0.375
0.018
0.005
-0.028
-0.006
0.853
0.160
NCE
0,273
0.534
0.597
0.267
0.232
0.362
0.352
0.793
0.426
LEEP 66
0.069
-0.038
0.476
0.530
0.471
-0.111
0.567
0.468
0.304
W-LEEP 55
-0.559
0.476
0.743
0.515
0.707
0.027
0.713
0.812
0.429
LogME 1
0.341
0.453
0.497
0.718
0.698
0.407
0.657
0.817
0.574
PACTran 27
0.136
0.262
0.484
0.631
0.614
-0.227
0.701
0.477
0.385
OТСE 99
-0.316
-0.050
-0.127
0.515
0.505
-0.168
0.406
0.210
0.123
LFC 25
0.226
-0.226
-0.235
0.330
0.271
-0.669
-0.059
-0.151
-0.064
MODEL SPIDER
0.382
0.711
0.727
0.870
0.977
0.686
0.717
0.933
0.750
The prompts are added to the input repr., allowing the transformer to utilize represented-ee -type context for better ranking process. Notably, Pm (T) depends on Φ7 and the pre-extracted PTMM-seecific features for all training tasks make the construction of these specific representation efficient.
4.5 A Brief Summary of MODEL SPIDER
MODEL SPIDER learns to rank PTMs based on the model-task pair, balancing efficiency and accuracy. During the training, sample tasks where PTM representation and transformer- based similarity are learned. In particular, to enable the model-task similarity to incorporate PTM-specific features, replace some of the inputs to the transformer with enriched representations. We pre-extract PTM-specific features for all training tasks, and then the estimated ground-truth and the specific repr. could be constructed efficiently. During deployment, search with a general representation. Then we carry out forward passes over the target task only for top-k ranked PTMs, where the obtained PTM-specific task repr. will re-rank the PTMs by taking the distributed examples with PTM's features into account.
first employ coarse- -raiieed PTT
5 Experiments
We evaluate MODEL SPIDER on three benchmarks: the PTM zoo comprising heterogeneous models from the single-source, multi-source datasets, or composed of large language models. We analyze the influence of key components in MODEL SPI- DER and visualize the ability of PTM using spider charts based on the learned representation.
Table 1: Performance comparison of regression- conducted approaches with the same model zoo and weighted Tu measurement as Table The downstream task is dSprites and UTKFaae.
Methods for Regression Tasks
Dataset
H-Score LogME GBC
Ours
dSprites 0.106
0.612
-0.283
0.679
UTKFace 0.075
-0.156
0.052
0.364
5.1 Evaluation on Single-Source Model Zoo
Setups. We follow 113 and construct model zoo with PTMs pre-trained on ImageNet across five architecture families, í.e. Inception (88), ResNet [35, DenseNet [38], MobileNet (83]

---

(S.066
(CE(-0
(E26(-0611)
(-0,00))
(-00)) PAC (-0.60))
(FL1--0177)
Ours (0.353)
H-Seore (UL35))
(05
LEEP (0.200)
VALEXP(0.2550
L0.5E
PACTram (0.237)
(0.20)
(B.444
Ours (9.549)
H-Score (0.60)
(611().361)
(-.73
67)
PACTram (0.
6.5
676
616 615 618
11.01 00 1131
Dxsecee:011
Figure 3: Visualizations when selecting PTMs from multi-source heterogeneous model zoo (w/ 42 PTMs) on three downstream datasets. Rows represent approaches, and columns represent datasets. Correlations (Tw)) shown above each subfigure. The horizontal axis denotes transferred accuracy (w/ tueeetuning,, while the vertical axis is the output ranking score. The PTM architectures are drawn in red, yellow, and green. The bold line and the gray area show the fitted straight line and the confidence interval for all PTMs. The strong linear correlation suggests superior performance.
and MNASNet [90] We evaluate various methods on downstream datasets, i.e. Aireraft [59] Caltech101 (32) Cars (47), CIFARIO [9]. CIFAR 100 49), DTD [19], Pet (73). and SUN397 107
for classification, UTKFace
118 and dSprites 61 for regression.
Baselines. There are three groups of comparison methods. First are creating a proxy between PTM- specific features and downstream labels, such as H-Score (. NCE (97). LEEP [66]. .VLLEPP (55)
LogME [133. and PACTran (27). The second are based on the downstream inter-categories features like OTCE [93], Label-Feature Correlation (LFC) (25), and GBC 77.. Following (66) and (113), we
cquivalently modify NCE and H-Score to the general model selection application.
Evaluations. For the standard evaluation, follow the official train-test split of each downstream dataset and utilize all the training samples. In few-shot evaluation, we consider if MODEL SPIDER select useful models with limited labeled examples under privacy and resource constraints. We sample 10 examples per class from the training set as a "probe set" and report the average results over 30 trials. The full results, along with 95% confidence intervals, are presented in the appendix.
Training Details of MODEL SPIDER. We implement the with the pre-trained Swin-B [57 53
to extract the task representation. MODEL SPIDER is trained 832 sampled tasks from the mix of 6 datasets, i.e., Au [36], OfficeHome [99), PACS [54], SmalINORB (51), STL1O 22 and VLCS 31. MODEL SPIDER utilizes specific features from the top-3 ranked PTMs (out of 10) for downstream tasks, resulting in a 3-4 times speedup.
Results of Standard and Few-Shot Evaluation. For the standard evaluation shown in Table 2 and Table MODEL. SPIDER outperforms other baselines across datasets, except for Aircraft, which ranks top-2. It also demonstrates superior stability and outperforms all the existing approaches in few-shot scenarios, as displayed in the lower part Table 2 Consistently ranking and selecting the correct PTMs, MODEL. SPIDER achieves the highest mean performance among all methods.
5.2 Evaluation on Multi-Source Model Zoo
We construct a large model zoo where 42 heterogeneous PTMS are pre-trained from multiple datasets.
Setups. PTMs with 3 similar magnitude architectures, i.e., Inception V3, ResNet 50, and NenseNet
201, are pre-trained on 14 datasets, including animals [37 46 general and 3D objects 51
49 47
14], plants [68], scene-basee based [107), remote sensing [106] 18 36 and multi-domain recognition 54]
We evaluate the ability of PTM selection on Aircraft 59 DTD 19 and Pet datasets.
Training Details. We use the same task representation extractor as in suusecction 5.1 with 4352 training tasks sampled from the mix of the above datasets for pre-training the model zoo.
Analysis of Multi-Source Model Zoo. With many PTMs in the model zoo, we first set h and select PTMs based on general representation. We visualize the results in Figure 3 with each
8

---

Table 3: Top-1 ranked Large Language Model (LLM) performance comparisons against L.M evaluation results 194 116 96 199 69,, which includes 2 directly baselines and our MODEL SPIDER, ranking on a pre-trained model zoo of 9 LLMs. The 10 downstream tasks are construct based on the OpenCompass 23 benchmark from 5 diverse fields as examination, language, knowledge, We denote the best-performing results bold.
understanding, reasoning.
Downstream Target Dataset
Method
Mean
Exam.
Language Knowledge Understand. Reason.
LLM Evaluations
Alpaca-7B [94]
24.30
67.20
41.95
33.30
51.70
43.69
Cha6GLM2-6B [16
39.00
67.30
44.35
40.25
68,67
51.91
LLaMA2-7B 7B [96]
31.30
67.40
55.90
40.30
52.93
49.57
Vicuna-7B [119]
29.10
66.70
49.45
34.70
52.67
46.52
ChatGPT 69
39.90
60.90
57.10
55.40
69.90
56.64
Top-1 Results of LLM Ranking Methods, Selected by
Self-assessed Confidence
34.60
67.40
45.10
37.45
62.60
49.43
Perf. on Similar Tasks
29.10
67.20
44.35
53.45
63.03
51.43
MODEL SPIDER
41.30
67.65
55.90
56.80
70.07
58.34
subfigure showing the transferred accuracy using the selected PTM with tine-tuning and the predicted ranking score. A petter-performing method will show a more obvious linear correlation. The results demonstrate that MODEL SPIDER achieves the optimum in all three datasets. Furthermore. a visualization of efficiency, the averaged performance over all datasets, and model size on this benchmark with standard evaluation is shown in Figure The different configurations of k balance the efficiency and performance in PTM selection, which "envelope" the results of other methods. These results confirm that MODEL SPIDER performs well in complex scenarios, highlighting its ability to select heterogeneous PTMs in a large model zoo.
5.3 Evaluation on Zoo of Large Language Models
We introduce 9 open-source Large Language Models (LLMs) to construct our LLM zoo and deploy the MODEL SPIDER framework. We conduct comparative analysis of the performance of the selected top-1 model against ChatGPT 669
Setups. The LLM involves Alpaca-7B 94
Baichuan-7B [109. Baichuan2-7B [T09]
ChatGLM2-6B [166,, InternLM-7B [95]. LLaMA2-7B 196), Vicuna-7B [119) Qwen-7B g and its chat fine-tuned version. We assess their zero-shot performance on diverse target tasks using the OpenCompass 23 LLM evaluation benchmark. We then focus on unseen tasks from the examination language, knowledge, understanding, and reasoning datasets as the target tasks. For more details, please see appendix (subsection B.3] We report the performance of the top-1 model recommended by each LLM ranking method and compare it with existing LLM evaluation results.
Training Details. For task representation, we employ a general Sentence-T5 [67 to obtain task representation. We extract answers from 10 instruction samples We initialize the corresponding model repr. to encode the capabilities of LLMs on instruction data.
representative task for dataset.
Analysis of Ranking the Zoo of LLMs. Given that LLMs are computationally intensive in PTMs, we rank LLMs based on their general task representation. Intuitive methods for LLM ranking, like proxy measures relying self-assessed confidence scores from generated answers or few-shot tasks in related domains, often fall short in assessing target task performance. The results indicate that while ChatGPT-3.5 demonstrates impressive performance in terms of universal performance across all diverse target tasks, as shown in Table 3 being 56.64, the top-1 ranked of MODEL SPIDER can surpass ChatGPT when efficiently choosing the appropriate LLM for each specific task. Our method achieves the average best and excels in the 4 out of 5 major fields of target tasks.
5.4 Ablation Studies
We analyze the properties of MODEL SPIDER on some downstream datasets, following the evaluation of single-source model zoo in subsection 5.1
Will RankAgg provide more accurate ground-truth during training? As discussed in subsec MODEL SPIDER is trained on historical tasks and we utilize RankAgg to approximate
tion 4.3

---

Medical
Arinad'
Medical
Animals
1.0
Correlation (ta)
0.8-
0.5
E64087
0.6-
3D Objec
Remoto Serving
3D Object
Remoto
0.4
Semsing
dittt Ca
0.2-
0.0
Soanc
Birds
Scrne
Rink
Number of PTM-specifie Features
Pre-Trained un AID
Pre-Trained on NABirds
(a) Number of Used Features vs. Correlation. (b) Spider chart of which semantic aspects the PTM excels in.
Figure 4: (a): The ablation analysis of how the ranking correlation changes (Y-axis) with more PTM-specific features (X-axis). (b): Visualization of the PTM's ability 6 major semantic clusters of datasets with spider chart. The score on the vertex of the spider chart is the averaged similarities between a PTM and the task representation in the cluster. The higher the vertex value, the better PTM would perform on that kind of task.
accuracy ranking. We investigate if this approximation offers better supervision and if using previous model selection methods like H-Score or LogME without aggregation sufficient. The results in 4inccluee CIFAR1O 10 and averaged results over eight classification datasets. It is evident that RankAgg provides stronger supervision during MODEL SPIDER'S 's training.
Will more PTM--peccficc features help? As mentioned in subsection 4.4 MODEL SPIDER is able to incorporate PTM-specific features the forward pass of PTM over the downstream task to improve the ranking scores. When no specific features (k = 0) exist, we use the general representation to rank PTMs (most efficient). In Figure 4 (a), we show that Tw increases when MODEL SPIDER receives more PTM-specific features. It balances the efficiency and accuracy trade-off.
5.5 Interpreting MODEL SPIDER by Spider Chart
An interesting by-product of MODEL SPIDER that can visualize the ability of PTM with a spider chart, which demonstrates which fields the PTM is good at. We cluster the datasets in our multi-source model zoo into six major groups. Then, we approximate a PTM's ability on the six types of tasks with the averaged sim- ilarity between PTM to the tasks the cluster. The larger the similarity, the better the PTM performs that task. In Figure (b), we find PTM pre-trained on AID [166] dataset works well on medical and remote sensing tasks, and a PTM pre-trained on NABirds (37 dataset shows strong ability on birds and
Table 4: The weighted Tar of MODEL SPI- DER variants when the training supervi- sion is approximated by different methods. "Mean" denotes the averaged performance over 8 downsteam datasets in Table 2]
Method
CIFARI0
Mean
w/ H-Score 9
0.386
0.642
w/ LogME 113
0.695
0.689
w/ RankAgg (Ours)
0.845
0.765
animal recognition. The spider charts provide valuable insights into PTM capabilities and assist in PTM recommendations for specific application scenarios.
Conclusion
The proposed MODEL SPIDER learns to rank PTMs for existing tasks and can generalize the model selection ability to unseen tasks, even with few-shot examples, and is applicable to both visual and large language models (LLMs). The two-stage pipeline in MODEL SPIDER enables it to fit the resources adaptively. A task is matched with PTMs efficiently based on their task-agnostic representation if resource is limited. While there is a sufficient resource budget, limited forward passes are carried out over the candidates of top-ranked PTMs, which re-ranks candidates via incorporating the detailed fitness between the task and the selected PTMs. The learned representations help construct a spider chart for each task, illustrating its relevance with all PTMs. The representation for models and tasks acts as a kind of specification that matches the main design in Learnware [12] 22
34 92
10

---

Acknowledgments. This work is partially supported by the National Key R&D Program of China (2022ZD01 14805). NSFC (62250069. 62376118, 62006112, 62206245), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology 2021-020, Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
[1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. Tensorflow: system for large-scale machine learning. In OSDI, 2016.
[2] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji. Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In ICCV, 2019.
[3] Enric Boix Adserà. Hannah Lawrence, George Stepaniants, and Philippe Rigollet. GULP: prediction- based metric between representations. In NeurlPS, 2022.
[4] Andrea Agostinelli, Michal Pándy, Jasper R. Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metries evaluations? In ECCV, 2022.
[5] Nir Ailon and Mchryar Mohri. Preference-based learning to rank. Machine Learning, 80(2-3), 2010.
[6] David Alvarez-Melis and Nicolò Fusi. Geometric dataset distances via optimal transport. In Neur!PS, 2020.
[7] Ann Arbor. A reasonable social welfare function. Seminar on Applications of Mathematics to Social Sciences, 1951.
[8] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang. Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang. Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang. Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309.1660 2023.
[9] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An ceemmation--hhoreete approach to transferability in task transfer learning. In ICIP, 2019.
[10] Shai Ben-David and Reba Schuller. Exploiting task relatedness for multiple task learning. In COLT, 2003.
[11] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira, Analysis of representations for domain adaptation. NIPS, 2006.
[12) Steiner Benoit, DeVito Zachary, Chintala Soumith, Gross Sam, Paszke Adam, Massa Francisco, Lerer Adam, Chanan Gregory, Lin Zeming, Yang Edward, Desmaison Alban, Tejani Alykhan, Kopf Andreas, Bradbury James, Antiga Luca, Raison Martin, Gimelshein Natalia, Chilamkurthy Sasank, Killeen Trevor, Fang Lu,, and Bai Junjie. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[13] Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning. In NewrlPS 2021, 2021.
[14] Lukas Bossard, Matthiuu Guillaumin, and Lue Van Gool. Food-101-mining discriminative components with random forests. In ECCV, 2014.
[15] Fatih Çakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff. Deep metric learning rank. In CVPR, 2019.
[16] Wei-Lun Chao, Hin-Jia Yc, De-Chuan Zhan, Mark E. Campbell, and Kilian Q. Weinberger. Revisiting meta-learning as supervised learning. CoRR, abs/2002.00573, 2020.
[17] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML, 2020.

---

[18] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of IELE, 105(10), 2017.
[19] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. CVPR, 2014.
[20] Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty natural yes/no questions. In NAACL-HLT, 2019.
[21] Jonathan Clark, Jennimaria Palomaki, Vitaly Nikolaev, Eunsol Choi, Dan Garrette, Michael Collins, and Tom Kwiatkowski. Tydi QA: A benchmark for information-seeking question answering in typologically diverse languages. Trans. Assoc. Comput. Linguistics, 8, 2020.
[22] Adam Coates, Andrew Y. Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature leaming. In AISTATS, 2011.
[23] OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. httpa://github: httpa////ttttub...o/aeeeee//pppp hcpppeecccccaaaspppcccccaaaaa 2023.
[24] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. Machine learning challenges workshop. Springer, 2005.
[25] Aditya Deshpande, Alessandro Achille, Avinash Ravichandran. Hao Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. A linearized framework and a new benchmark for model selection for fine-tuning. CoRR, abs/2102.00084, 2021.
[26] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[27] Nan Ding, Xi Chen, Tomer Levinhoim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models classification tasks. In ECCV, 2022.
[28] Yao-Xiang Ding. Xi-Zhu Wu, Kun Zhou, and Zhi-Hua Zhou. Pre-trained model reusability evaluation for small-data transfer learning. In NeurlPS, 2022.
[29] Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy transfer learning. CVPR, 2019.
[30] Kshitij Dwivedi, Jiahui Huang, Radoslaw Martin Cichy, and Gemma Roig. Duality diagram similarity: generic framework for initialization selection in task transfer learning. In ECCV, 2020.
[31] Chen Fang, Ye Xu, and Daniel N. Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV, 2013.
[32] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categorics. In CVPR Workshops, 2004.
[33] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bi- lal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, 2020.
[34] Lan-Zhe Guo, Zhi Zhou, Yu-Feng Li, and Zhi-Hua Zhou, Identifying useful learnwares for heterogeneous label spaces. In ICML, volume 202 of Proceedings of Machine Learning Research, 2023.
[35] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[36] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land and land cover classification. IEEE Journal of Selected Topics Applied Earth Observations and Remote Sensing, 2019.
[37] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge J. Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, 2015.
[38] Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. In CVPR, 2017.
12

---

[39] Long- Kai Huang, Junzhou Huang, Yu Rong. Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In ICML, 2022.
[40] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan 7hang. Tangjun Su, Junteng Liu, Chuancheng Lx, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. CoRR, abs/2335.08322 2023.
[41] Shibal Ibrahim, Natalia Ponomareva, and Rahul Mazumder. Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance. CoRR, abs/2110.06893, 2021.
[42] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[43] Thorsten Joachims. Optimizing search engines using clickthrough data, In SIGKDD, 2002.
[44] Brendan Jou and Shih-Fu Chang. Deep cross residual learning for multitask visual recognition. In ACM
MM, 2016.
[45] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2), 1938.
[46] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li, Novel dataset for fine- grained image categorization: Stanford dogs. In CVPR workshop on FGVC, volume 2, 2011.
[47] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- ine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), 2013.
[48] Nikolaus Kriegeskorte. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2008
[49] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[50] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova. Llion Jones, Matthew Kelcey, Ming-Wei& Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: a benehmark for question answering rescarch. Trans. Assoc. Comput. Linguistics, 7, 2019.
[51] Yann LeCun, Fu Jie Huang, and Léon Bottou, Learning methods for generic object recognition with invariance to pose and lighting. In CVPR, 2004,
[52] Hector Levesque, Emnstt Davis, and Leora Morgenstemn. The winograd schema challenge. KR, 2012.
[53]
Chunyuan Li, Jianwei Yang, Pengchuan Zhang. Mei Gao, Bin Xiao, Xiyang Dai. Lu Yuan, and Jianfeng Gao. Efficient supervide vision transformers for representation learning. ICLR, 2022.
[54] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
[55] Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In CVPR, 2021.
[56] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong. Hongyu Ren, Huaxiu Yao, Jue Wang. Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksckgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji. Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Ieard, Tianyi Zhang, Vishray Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR, abs/2211 1,011 10, 2022.
[57] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transtormer using shifted windows. ICCV, 2021.
[58] Su Lu, Han-Jia Ye, and De-Chuan Zhan. Tailoring embedding function heterogeneous shw-shot tasks by global and local feature adaptors. In AAAI, 2021.
[59] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, abs/1306.5151, 2013.
13

---

[60] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.
[61] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://thtb..oomm/mmmi////pppss
2017.
[62] Brian McFee and Gert R. G. Lanckriet. Metric learning to rank. In ICML, 2010.
[63] Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning- Adaptive computation and machine learning. 2012.
[64] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, In EMNLP, 2018.
[65] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[66] Cuong V Nguyen, Tal Hassner, Cedric Archambeau, and Matthias Seeger. Leep: A new measure evaluate transferability of learned representations. JCML, 2020.
[67] Jianmo Ni, Gustavo Hernández Abrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. Sentence-15: Scalable sentence encoders from trentrained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages Association for Computational Linguistics, 2022.
[68] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In ICVGIP, 2008.
[69] OpenAL. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
[70] Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 2009.
[71] Michal Pándy, Andrea Agostinelli, Jasper R. R. Uijlings, Vittorio Ferrari, and Thomas Mensink. Transfer- ability estimation using bhattacharyya class separability. In CVPR, 2022.
[72] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Feránndez. The LAMBADA dataset: Word prediction requiring a broad discourse context. ACL. 2016.
[73] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
[74] Huiyan Qi, Lechao Cheng, Jingjing Chen, Yue Yu, Zunlei Feng, and Yu-Gang Jiang. Transferability estimation based on principal gradient expectation. CoRR, ftls/2211.1622 2022.
[75] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In JCML, 2021.
[76] Rajecv Ranjan, Vishal M. Patel, and Rama Chellappa. Hyperface: A deep multi-task learning framework for face delection, landmark localization, pose estimation, and gender recognition. IEEE Transactions on Pattern Analyxis and Machine Intelligence, 41(1), 2019.
[77] -ylvestre-Alvise Rebufli, Alexander Kolesnikov. Georg Sperl, and Christoph H. Lampert. icarl: Incre- mental classifier and representation learning. CVPR, 2017.
[78] Cédric Renggli, André Susano Pinto, Luka Rimanic, Joan Puigcerver, Carlos Riquelme, Ce Zhang, and Mario Lucic. Which model to transfer? finding the needle the growing haystack. In CVPR, 2022.
[79] Mclissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium, 2011.
[80] Michael T Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G Dietterich. To transfer or not to transfer. In NIPS Workshop on Trunsfer Learning, volume 898, 2005.
[81] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 2015.
14

---

[82] Saari, Donald G., and Vincent R. Merlin. The copeland method: L: Relationships and the dictionary. Economic Theory, 8(1). 1996.
[83] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chies Chen. Mo- bileNetV2: inverted residuals and linear bottlenecks. In CVPR, 2018.
[84] Wenqi Shao. Xun Zhao, Yixiao Ge, Zhaoyang Zhang. Lei Yang. Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in self-challenging fisher space. In ECCV, 2022.
[85] Jie Song, Yixin Chen, Xinehao Wang, Chengehao Shen, and Mingli Song. Deep model transferability from attribution maps. In NeurlPS, 2019.
[86] Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, and Mingli Song. Depara: Deep attribution graph for deep knowledge transferabilityy In CVPR, 2020.
[87] Kai Sun, Dian Yu, Dong Yu, and Claire Cardic. Investigating prior knowledge for challenging chinese machine reading comprchension. Trams. Assoc. Comput. Linguistics, 8, 2020.
[88] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zhigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In CVPR, 2016.
[89] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. NAACL-HLT, 2019.
[90] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quac V. aaaform-wwwre aware neural architecture search for mobile. In CVPR, 2019.
Le. Mnasnet: Platform
[91] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi- Hua Zhou. Towards cnabling learwaae to handle heteroge- neous feature spaces. Machine Learning, 2022.
[92] Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou, Handling learwwares developed from heteroge- neous feature spaces without auxiliary data. In IJCAI, 2023.
[93] Yang Tan, Yang Li, and Shao-Lun Huang. OTCE: A transferability metric for cross-domain cross-task representations. In CVPR, 2021.
[94] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An foslluction-folloiing Ilama model. https://gitthb com/tatsu-e 1ab/ www.ab/staffrddddaaaeeeea plccaa 2023.
[95] IntemLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https://githut com/ InternLM/ InternLM 2023.
[96] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi. Yasmine Babaci. Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava. Shruti Bhosale, Dan Bikel, Lukas Blecher. Cristian Canton Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marein Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
[97] Anh Tuan Tran, Cuong Nguyen, and Tal Hassner. Transferability and hardness of supervised classifica- tion tasks. In /CCV, 2019.
[98] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[99] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[100] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech- Birds-200-200 Dataset. Technical Report CNS-TR-2011-4 California Institute of Technology, 2011.
15

---

[101] Alex Wang, Yada Pruksachatkun. Nikita Nangia, Amanpreet Singh. Julian Michael, Felix Hill. Omer Levy. and Samuel R. Bowman. Superglue: A stickier benchmark for general- language understanding systems. NearIPS, 2019.
[102] Zirui Wang, Zihang Dai, Barnaáás Póezos, and Jaime Carbonell. Characterizing and avoiding negative transfer. In CVPR, 2019.
[103] Ying Wei, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning to transfer. In ICML, 2018
[104] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ee natural language processing, In EMNLP, 2020.
[105] Fen Xia, Tie- Yan Liu, Jue Wang, Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In ICML, volume 307, 2008.
[106] Gui GuinSong Xia, Jingwen Hu, Fan Hu, Baoguang Shi. Xiang Bai, Yanfei Zhong, and Liangpei Zhang. Aid: benchmark dataset for performance evaluation acrial scene classification. CoRR, /00/1600005660 2016.
[107] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[108] Liang Xu, Hai Hu, Xuanwei Zhang. Li, Chenjie Cao. Yudong Li, Yechen Xu, Kai Sun, Dian Yu. Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng. Rongzhao Wang. Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang. He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUB: A chinese language understanding evaluation benchmark. In COLING, 2020.
[109]
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang. Fei Deng, Feng Wang. Feng Liu, Guangwei Ai, Guosheng Dong. Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji. Jian Xie, Jun Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuchai Pan, Yanjun Shen, Yiding Wang. Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. Baichuan 2: Open large-scale language models, 2023.
[110] Yang Yang, Han-Jia Ye, De-Chuan Zhan, and Yuan Jiang. Auxiliary information regularized machine for multiple modality feature learning. In IJCAI, 2015.
[111] Han-Jia Ye, De-Chuan Zhan, Nan Li, and Yuan Jiang. Learning multiple local metries: Glohal considera- tion helps. IEEE Trans. Pottern Anal. Mach. Intell., 42(7), 2020.
[112] Han-Jia Ye, De-Chuan Zhan, Yuan Jiang, and Zui-Hua Zhou. Heterogeneous few-shot model rectification with semantic mapping. IEEE Trans. Pattern Anal. Mach. Intell., 43(11), 2021.
[113] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In ICML, 2021.
[114] Kaichao You, Yong Liu, Ziyang Zhang, Jianmin Wang, Michael Jordan, and Mingsheng Long. Ranking and tuning trentrained models: A new paradigm for exploiting model hubs. Journal of Machine Learning Research, 23, 2022.
[115] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
[116] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. G1m-130b: An open bilingual pre-trained model. arXiv preprint arX1v:2210.02414, 2022.
[117] Xiaotian Zhang, Chunyang Li. Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on GAOKAO benchmark. CoRR, abs/2305. 12474, 2023.
[118] Zhifei Zhang, Yang Song. and Hairong Qi. Age progression/regression by conditional adversarial autoencoder. In CVPR, 2017.
16

---

[119] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin. Zhuohan Li. Dacheng Li. Eric. P Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mi-bench and chatbot arena, 2023.
[120] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu. Yanlin Wang. Amin Saied, Weizhu Chen, and Nan Duan. Agieval: human-centrie benchmark for evaluating foundation models. CoRR. abs/2304.06364, 2023.
[121] Zhi-Ho Zhi-HuaBae Zhou. Leamware: on the future machine learning. Frontiers Computer Science, 10(4), 2016.
[122] Zhi-Hua Zhou and Zhi-Hao Tan. Learwware: Small models do big. CoRR, abs/2210.03647, 2022.
17

---

Supplementary Material
We provide details omitted in the main paper.
Appendix A Workflow of MODEL SPIDER, encompassing the construction of model-task repr., training, and testing, with the "how to" and "answer" format.
Appendix B Experimental setups and implementation details MODEL SPIDER, especially
the two types of pre-training model zoos utilized in the experimental section.
Appendix C
Additional experimental results conducted along different dimensions of robustness analysis.
Appendix D
Additional datasets descriptions and other details mentioned in the main text.
Appendix
Discussions and future exploration of MODEL SPIDER.
A Details and Discussions of MODEL SPIDER
In the method section of the main text, elucidate the comprehensive workflow for training and testing the deployment of MODEL SPIDER. This process encompasses three main steps, including (I) the extraction of task repr., (2) the extraction of model repr., and (3) the construction of a training scheme that assesses the ranking of matching between model-task repr., thereby establishing the ground-truth rank of the model zoo for given task. Once these three steps have been accomplished, the subsequent phase entails training the MODEL SPIDER by leveraging the extracted repr. in conjunction with the ranked ground truth information.
In essence, the testing and deployment strategy employed by the MODEL SPIDER framework epitomizes balance between flexibility and efficiency. By employing fixed feature extractor vi to acquire repr. pertaining to downstream target tasks, the trained MODEL SPIDER undergoes a singular inference pass, generating output quantifying the similarity between each model and the downstream task representation. It then accomplishes the task of ranking the PTMs.
In the forthcoming sections, we elaborate on the details in the form of "how to do if" questions. The training process of MODEL SPIDER is illustrated in Algorithm [ while the sampling procedure for training tasks is elaborated in detail in subsecttion A.2 Additionally, in subsecttoon A.6, expound upon the training strategy of PTM-Specific task representation. Analogously, the testing process of MODEL SPIDER is presented in Algorithm 2 and in subsection A.7] we provide a comprehensive exposition of the entire deployment workflow for ranking pre-trained models.
A.1 How to construct model representations and task ones
This section supplements the details of subsection 4.2 and subsection 4.4 i.e., the construction of the model-task repr., including the enriched PTM-specifi ones.
PTM representation. The dimension of PTM repr., i.e., the d of 0 E Rd is implemented as 1024. a learnable parameter that optimized with the training process.
Task representation. The y implemented by pre-trained Swin-B-based EsVIT [37 (linked self-supervised learning on the ImageNet-1K IK [8] with batch size 512. our experiments, this encoder acts as a wide-field feature extractor and is fixed without updating. The shape of task repr. 4u (T) E ædxCr varies with the number of categories of downstream tasks. As mentioned in Isubsection 4.4) task reprs. enriched by the PTM--speifiic features obtained through the forward pass of PTM. We another fully connected layer to project the PTM-specific feature to align with the model representation.
https:///ithuu.
A.2 How to sample the training tasks of MODEL SPIDER
We sample tasks for training MODEL SPIDER from additional datasets that are disjoint from the downstream tasks. These additional datasets possess notable differences and encompass diverse domains. Notably, MODEL SPIDER does not require substantial additional data for training. We sample the training tasks from diverse pool of datasets. The number and size of the mixed datasets controlled within certain range. For more details, please see Appendixx B
18

---

NCE:
LEEP:
RankAgg
PTM-1
PTM-2
PTM-3
PTM-4
LogME:
Figure 5: An illustration of the rank aggregation approach to ensemble the ranking of PTMs relying on diverse transferability assessment methods (three methods depicted in the figure). The PTMs that outperform more other PTMs should be placed ahead.
A.3 How to see the relationship between RankAgg and MODEL SPIDER
We claim that RankAgg proposed by cannot be considered direct baseline method. Firstly, RankAgg involves a substantial computational overhead when used as a stand-alone method for ranking PTMs. This is primarily due the time and memory requirements of computing the base selection methods. Using Rank Agg directly as a baseline would introduce a significant computational burden. However, introduce Rank A Ag in the training part of MODEL. SPIDER. It is more efficient compared to full parameter fine-tuning.
approximate ground-truth method for pre-computing
Actually, MODEL SPIDER aims to demonstrate its broad generalization capacity by leveraging RankAgg to process an independent set of mixed data that has no overlap with the test data. This independent evaluation showcases the effectiveness of MODEL SPIDER in real-world scenario and emphasizes its ability to handle diverse data efficiently. RankAgg itself does not play a role during the test exccution of MODEL SPIDER.
A.4 How to efficiently approximate the training ground-truth of MODEL SPIDER
This section complements subsection 4.4, wherein the training and ranking of the model zoo across multiple datasets are discussed. However, obtaining the ranking for all historical tasks through brute force is computationally expensive. To mitigate this issue, we introduce rank aggregation method denoted as RankAgg, which serves an approximation of the ground truth ranking.
Existing PTM selection methods rely on the PTM--ppecffic features TTh estimate the transferability score. Different methods may have diverse score values - a PTM will be placed in different positions based on the scores provided by various methods. We empirically observe that some popular approaches such as NCE [97]. LEEP [66], and LogME [113] [14] show "good but diverse" PTM ranking orders, so an intuitive approach to improving the transferability estimation quality to ensemble their ranking results to a stronger single order.
As mentioned in subsection 4.3 given (2.,e..... (a)) as multiple rankings over the same set of M PTMs for a target task 7, i.e., the order sorted by the estimations of transferability via various methods, we take advantage of Copeland's aggregation method 7 82) to ensemble the orders.
i (1om-+7h
= RankAgg ((6,,
Copeland's ' aggregation compares each pair of ranking candidates and considers all preferences to determine which of the two more preferred as illustrated in Figure 5]
Taking model m, m' as an example, we define the majority relation to express the one-on-one dominance between these two models. Precisely, assuming that An approaches rank model above model m', i.e., fi.m ti with Am such f., while the remaining Amn' ones do the opposite. Note that Am + Am = A. The m >M m' just case Am > Am', and correspondingly m =M m indicates An Amt. In summary, define the aggregation score for model
PM i}
1 (i =M
8
where # (.) is the size of the set. The aggregation score for model is the number of others over which they have a majority preference plus half the number of models with which they have preference tie. In our implementation, aggregate the results of NCE, LEEP, LogME, and H-Score.
19

---

Algorithm The Training Part of the MODEL SPIDER
1: Input: fixed th; learnable parameters θ including model repr. {6... m= m=11 FC for projection, and
parameters of the transformer-based MODEL SPIDER
2: Sample training tasks {Ti} from the additional mixed datasets as in subsection A.2
3: Extract and save all task repr. U, {u (7:) with vb.
4: for all sampled task Ti do
5: for m 1 to M do
6: if the mth PTM-specific features is available (randomly holds) then Derive the PTM-specific task repr. as mentioned in subsection 4.4
7:
8:
f +T, = sime (0m, P.T. Pm (Ti))
9:
else
10:
Take model repr. Om and estimate the similarity of model-rrepressentatin pairs as Eq.
tΦ... +T: s0 (7i))
12:
end if end for
13:
14:
From above for, the estimation scores of MODEL SPIDER é is conducted.
15:
Calculate H-Score, NCE, LEEP, and LogME on Te-
16:
Aggregate on the results of existing approaches to obtain ground-truth & as in subsection 4.3
=
= RankAgg((L (...)....)))
18:
Optimize the parameters of MODEL SPIDER with ranking loss frank W.r.t. the ranking of t.
exp (tdse(()) iimm exp (fase(n)
Brank (t,
Σ
log
Compute Valrank and update corresponding parameters with the gradients
20: end for
21:
Output: learned O including {0m) FC, and parameters the MODEL SPIDER
RankAgg can become quite time-consuming when calculating PTM ranking scores for the entire dataset, mainly due the substantial overhead of computing the base selection methods. In our experimental setup, we integrate the RankAgg method as a module during the training phase, enabling us pre-compute the rankings for each task. The RankAgg may raise the computational burden if employed directly testing baseline. Therefore, we employ RankAgg for the sampled few-shor tasks to balance ranking accuracy with efficiency and only use the training part. Note that MODEL SPIDER Iearns based on the RankAgg results, but is deployed independently of it and other bascline methods. Since RankAgg summarizes the PTM generalization capability on differentiated tasks spanning multiple domains, our model derived from the pre-aggregated rankings can learn the PTM ranking ability on broader range of unseen tasks.
A.5 How to learn the similarity of model-task representation
This section elaborates on (sussection 4.1 i.e., the learning process of MODEL SPIDER, especially the Transformer based estimation. The Transformer-basee module of model-task similarity. The model-task repr. is concatenated as a sequence of features. The Transformer based module naturally fits and takes such input. Concretely, in operation, transformer (-) is formalized
transformer (z) z + a (Q.K,V=z)
ZW (=W*)
(9)
+ softmax
W
Vd
we apply linear projections on the query, key, and values using W9, W and W respectively. The similarity between prototypes is measured by the inner product in the transformed space, which
20

---

Algorithm 2 The Downstream Inference Part of MODEL SPIDER
Input:
target task T. fixed ié, learned θ
Obtain task repr. fu (7) with & Eq.
Estimate similarity of model-representtaion pairs as Eq.
= [f -7 sime (6 /a (T)) m=1
Select top-k PTMs via & = ( (f }
m=1-
Obtain the indexes in descending order via dsc (+).
for m dsc (1) to dsc (k)) do Re-construct enriched repr. pm (T), and update:
fo--T -T -sme (0m, P Mm (T))
end for
Output: Rank PTMs with = fi } m=1
M
results in larger weights of the attention head a. Here d is the size of every attention head. The output of the corresponding position of the model repr. is forwarding passed through a learnable MLP and then obtains the fitness estimated score of PTM selection.
The learnable parameters in MODEL SPIDER. To learn PTM ranker, optimize M model repr. (0m 1m the fully connected layer projection heads of the PTM-specific task repr. 47, (mentioned in subsection 4.4p and the transformer-based model-task similarity evaluator sim(:.), which the main mapping and estimation module (mentioned in subsection 4.2).
A.6 How to re-rank with PTM-specifii task representation
As described Subsection 4.4 of the main text, we initially extract generic features using a fixed and conduct with the invariant task repr. across all PTMs. These features are used to generate coarse-grainee ranking by comparing the similarity between each task and the model representation. However, this ranking solely based standardized task representation and does not account for the specific task-related information for each individual PTM.
Hence, we propose the re-ranking strategy specifically targeted at the top-k PTMs. During the testing phase, we leverage the coarse saarse-griined ranking and perform inference on the downstream task with these top-k PTMs. Such PTM-specific task repr. worked to update their similarity with the downstream task, as outlined in Algorithm Notably, in the third line of the algorithm, we conduct a re-ranking based the revised similarity scores obtained through this process.
A.7 How to deploy MODEL SPIDER for testing
For novel downstream task, we employ the generic feature extractor & to extract the task representa- tion. We then evaluate the similarity between each PTM in the model zoo and the given downstream task using the learned model repr. and a transformer- based MODEL SPIDER. resources available, can leverage the results from the previous round to enhance the ranking process. Specifically, we can select the top-k PTMs from the previous ranking, extract their features, and apply the re-ranking approach described in subsection A.6
computational
B Experimental Setups and Implementation Details
In this section, we introduce the experiment setups and implementation details, including constructing the pre-trained model zoo and training as well as deploying MODEL SPIDER.
21

---

B.1 Single-source heterogeneous model zoo
Construction of the model zoo. We follow 113 and construct a model zoo with 10 PTMs pre-
trained on ImageNet [8T] across 5 families of architectures available from PyTorch. Concretely,
they are Inception V1
88 Inception V3 [88] ResNet 50 (35), ResNet 101
(35. ResNet 152 [35,
DenseNet 121 338. DenseNet 169
[38]. DenseNet 201 (38] MobileNet V2 [83] and NASNet-A
Mobile [90]. The model zoo spans PTMs of multiple parameter quantities. These pre-training models cover most of the supervised pre-training models the researchers employ.
The downstream tasks, There are 9 downstream tasks from various fields, including Aircraft [59] Caltech101 [32) Cars [47), CIFAR1O [9], CIFAR 100 [49]. DTD [19], Pets [73], and SUN397 [07 for classification, UTKFace [18 and dSprites [6]] for regression. We use official train-test splits on each dataset and calculate the estimation scores for the baseline approaches on the training part.
Transferred accuracy ranking of PTMs (ground-truth) after fine-tuning downstream tasks. We follow You et al. [113] to obtain the ground-truth transferability score as well as the rankings t = [ JM m=1 (M- = 10) with careful grid-search of hyper-parameters. Specifically, we grid search the learning rates (7 learning rates from 10 to 10-4, logarithmically spaced) and weight decays (7 weight decays from 10-6 to 10-3 logarithmically spaced) to select the best hyper-parameter on the validation set and compute the accuracy on the downstream test set. The training and computation of such ground truth necessitates a substantial investment of over 1K GPU hours, imposing significant financial and computational burdens. Consequently, the feasibility of accomplishing this task within the constraints of training MODEL SPIDER is rendered unattainable.
Sampling details of training tasks. We sample the training tasks from a diverse pool of datasets. The datasets considered for sampling include EuroSAT, OfficeHome, PACS, SmalINORB, STL10, and VLCS. To ensure a representative training set, we randomly sample 832 tasks from all datasets. Each task is distributed across 2 to 4 mixed datasets and consists of 100 categories, and for each category, we randomly select 50 examples. In cases where the number of categories or examples to be sampled exceeds the specified limits, we select the maximum allowable value.
Discussions. This model zoo covers several classical structures commonly used in deep learning. The number of model parameters ranges widely, with large application potential. Still, there is also a situation where PTMs with larger scales tend perform better in classification tasks and regression ones, making certain rankings always better on some datasets.
B.2 Multi-source heterogeneous model zoo
Construction of the Model Zoo. As mentioned in the main text, we construct a large model zoo where 42 heterogeneous PTMs are pre-trained from multiple datasets in different domains, including
animals (37 46], general and 3D objects 32
47
14] plants [68], scene-based [107], remote
sensing [106 18 361 and multi-domain recognition [54] The concrete datasets are Caltech101 [2
Cars [47. CIFARIO 44). CIFARI00 [49], SUN397
[07], Dogs [46), EuroSAT [36), Flowers [68].
Food 14, NABirds B77 PACS [54) Resise45 18 SmallNORB [1 and SVHN 665 The models'
structures are 3 similar garameter-magninee
architectures, i.e., Inception V3
(88) ResNet 50 (35] and
DenseNet 201 (38) The setting of the multi-source heterogeneous model zoo includes significantly
more pre-trainin treiiinn data than the single-source heterogeneous one described above. We pre-train the models with 3 structures on 14 datasets mentioned above (3 x 14 = 42, initialized from the weights of the corresponding ImageNet pre-trained models).
The downstream tasks. We select 3 representative datasets as the downstream test tasks and conduct the PTM selection methods on them. Concretely, they are Aircraft (59], DTD [9 and Pets (7). As outlined in the following description, obtain the transferred fine-tuning accuracy (ground-truth) with an equivalent level of papperr-rameterss search strategies.
Transferred accuracy ranking (ground-truth). Similarly, we adopt downstream supervised learning with optimizing by cross entrroyy loss. We meticulously conduct a grid-search of hyper-parameters such as optimizers, learning rates, and weight decays (2 optimizers as SGD or Adam, 6 learning rates from x 10-2 to 10-1, and weight decay values from 5 x 10-4 to 10-5, batch size of 128, and the maximum epoch of 100). For the multi-domain dataset, like PACS (54) we set the test set to the same domain as the training set to reveal the in-domain performance. For the rest, we use the official train-test splits. We build the model zoo with around 5K GPU hours (on NVIDIA V100
22

---

Corciation (Ta)
0.5
Lee Trening Thaks
Sumplery y Dum (rggee))
Airona
Caltech111
Cars
CIFARIO CIEARIO
DID
Pet
SUNDY
Downstream Task
Figure 6: Ablation studies simpler sb and less training tasks. We observed a slight decrease in performance when employing a weakened fixed feature extractor v for MODEL SPIDER. Reducing the diversity of training tasks may result in performance degradation some datasets.
GPUS). Similarly, when dealing with the expanded model zoo, the utilization of rigorous training methodologies acquire the requisite ground truth for training MODEL SPIDER is eschewed.
Sampling details of training tasks. The sampling process for the multi-source heterogencous model is consistent with the single-source one mentioned above. In this case, datasets as the auxiliary set, Le., Caltech101, Cars, CIFARIO, CIFAR100, Dogs, EuroSAT, Flowers, Food, NABirds, PACS, Resisc45, SUN397, and SVHN. We randomly sample 4352 tasks for training.
the following
The availability of multi-source heterogeneous model zoo introduces a wider array of models with varying structures, effectively covering broader scope of domain knowledge. Consequently, this heightened diversity presents an increased difficulty in accurately ranking PTMs. Particularly, when substantial gap exists between the characteristics of downstream tasks and the major PTMs, the ranking accuracy of some baseline methods undergoes a precipitous decline.
Discussion.
B.3 Large language models zoo
Construction of the model zoo. We considered setting for ranking pre-trained models in natural language processing, wherein we utilized a library of 9 commonly used open-source Large Language Models (LLMs)). These LLMs include Alpaca- [94] Baichuan-7B [109] Baichuan2- 7B [109],
ChatGLM2-6B [116]. InternL (95). LLaMA2-77 (96), Vicuna-7B [119] Qwen- 7B [8 and its
chat fine-tuned version. These open-source LLMs, trained by academic institutions or companies on vast corpora, possess robust zero-shot capabilities. However, compared to the performance on general tasks, LLMS have a domain gap regarding some specific tasks. Some new benchmarks [56 40 23
have been proposed recently to show that while ChatGPT [69] performs well concerning the average performance general tasks, may not consistently outperform other models in certain tasks. Additionally, the deployment complexity and computational costs of LLMs vary significantly. Blindly choosing L.LMS with large model sizes or high running expenses may not achieve optimal accuracy but is more likely waste resources. Therefore, the urgent challenge is accurately and efficiently selecting the most suitable LLM for a given task within the available budget and constraints.
The downstream tasks. We focus unseen tasks, i.e., the examination datasets of AGIEval [20]. as well as language datasets AFQMC [00.. WSC [52] knowledge datasets BoolQ 20), Natural Ques-
tions [50] understanding datasets C3
87 XSum [64], and reasoning datasets RTE 24 AX-b and
AX-g [101] as the target tasks. When constructing tasks, whether for training or testing, we extract
answers from 10 instruction data. During the generation of the final token in sequence generation, extract features from the last layer. We calculate the average ranking score for 3 randomly sampled tasks from the target dataset as the final result.
Sampling details of training tasks. For training tasks, we deliberately choose different data from the test tasks, containing the remaining datasets in the OpenCompass (33 benchmark evaluation, such as
GAOKAO-Bene's [177) TyDiQA (21) CommonSenseQAA [9] LAMBADA 722, COPA (79] and so on. We sample 10 instruction answers for each dataset, forming task on that dataset. For each dataset, we sample maximum of 16 training tasks, with mixed tasks from various datasets used in the training of MODEL SPIDER.
23

---

0.15
1.00
Correlation (ta)
0.50
Correlation (Ta)
0.75
0.50
B-Semz
0.25
H-soo
4025
0.00
0.50
LeghCE
-22
LoghE
10
50
40
50
0.50
Number examples class
Number examples
(a) Aireraft
(b) Caltcch101
Figure Correlation (Twr) given various number of examples per class on (a) Aireraft and (b) Ca1tech101. MODEL. SPIDER shows stable and promising results in the low-shot scenario.
C Additional Experimental Results
C.1 Ablation studies on simpler v and less training tasks
We deploy additional experience with weakened conditions to verify the robustness of MODEL SPIDER. In Ingiirure 6 we first introduce an attenuated simpler tb the additional encoder except for the PTMs in the model zoo. We import the tiny format pre-trained Swin- Triin-Tranffomer from Es (about this, please refer to subsectti A. for more details). It has about half the number of parameters. The results show that although attenuated v has only half of the parameters, it can still assist MODEL SPIDER expressing task representation.
We then halve the training tasks to verify the significance of the training part diversity. We find that except for the performance degradation of the DTD dataset, the others are still flush with performance. MODEL SPIDER learns the characteristics of different PTM ability dimensions well despite the absence of training tasks.
C.2 Ablation studies on the influence of training loss
As stated the main text, the learning process of MODEL. SPIDER incorporates ranking loss. To assess the efficacy of this selection, alternative re- gression or ranking loss functions, such mean square error (MSE) and ListMLE [TOS], are em- ployed as replacements. The outcomes, presented in Table 5] clearly demonstrate that the presented ranking loss function surpasses the other alterna- tives in terms of both effectiveness and robustness. Notably, when alternative loss functions are uti- lized, the overall performance of MODEL. SPIDER experiences substantial decline. These findings underscore the indispensable role of the ranking loss function within the framework of MODEL SPIDER.
Table 5: The weighted To of MODEL SPIDER variants when the training objective is imple- mented by different loss functions. "Mean" de- notes the averaged performance over 8 datasets.
Method
CIFARIO
Mean
w/ MSE
0.558 0.777
0.526
w/ ListMLE 1OS
0.735
w/ Lmok (Ours)
0.845
0.765
C.3 Ablation studies on the different shots of Rank Agg and other baselines
We conduct ablation analysis to compare RankAgg with several baseline methods on Aircraft and 101 datasets with respect to the Tu of the PTM ranking. We examined the variation of these metrics and their corresponding confidence intervals (in 95%) as the number of samples per class (shot) increased. The results, depicted in the provided Figure 7 are based on the average values and confidence intervals obtained from 30 randomly sampled sets for each shot. Due to computational constraints, certain baseline methods were omitted from the analysis. Notably, our findings reveal that the rank aggregation strategy effectively consolidates diverse perspectives on PTM ranking and consistently surpasses the performance of basclines across almost all shots.

---

Table 6: Ablation studies the performance of MODEL SPIDER when the pre-trained model repository grows dynamically.
MODEL SPIDER
Aircraft Caltech101 Cars
CIFARIO CIFARIO0 DTD
Pets
SUN397
Mcan
When the number of PTMs increases
number 3
0.545
1.000
1.000
1.000
0.182
1.000
1.000
1.000
0.841
increase to 6
0.573
0.627
0.818
0.905
0.839
0.445
0.888
0.336
0.679
increase to 10
0.568
0.637
0.576
0.797
0.695
0.796
0.573
0.436
0.635
C.4 Ablation studies on the dynamically incremental model zoo
When encountering new PTMs during the model selection task, the previously trained model repr. in MODEL SPIDER can be dynamically learned and updated. We employ an incremental learning approach 77 to address this challenge. Specifically, we sample 25% target tasks where the PTM ranking is closest to the average of all and insert the approximated accuracy of the new PTMs on them. This newly constructed ranking ground-truths include the correlation between old and new model repr., reducing the influence of imbalanced incremental data.
We performed ablation studies to investigate the behaviour of MODEL SPIDER as the pre-trained model zoo dynamically expanded. Our analysis focused on how can MODEL SPIDER could quickly adapt to newly added PTMs and integrate them into the ranking process. The results in Table demonstrate that as the size of the model zoo increased from 3 to 6 and then to 10, MODEL SPIDER demonstrated the ability to incrementally learn the recommended ranking for the new additions to the model zoo. The incrementally learned ranking for the entire PTM zoo exhibited slightly lower accuracy than the results of direct training on all PTMs. Nonetheless, MODEL SPIDER consistently maintained an excellent level of performance.
C,5 Confidence intervals for few-shot setting in Table 1 of the main text
We include the confidence intervals (in 95%) for the few-shot experiments in the respective section of Table for the main text. These intervals were obtained through 30 repeated trials, providing a robust estimate of the performance variability in few-shot manner.
C.6 Illustration of re-ranking with PTMM-specffic task representation
In Inssubsection 4.4, we diseuss the learnable model repr., which captures the empirical performance of a PTM across various training tasks. This training scheme serves to decouple the task repr. from the forward pass of each PTM. Compared to the task repr. guided solely by general features, the PTM-specific task repr. provides more informative clues. By constructing it with the forwarding pass of PTM, we can incorporate the source PTM's s adaptation information for downstream tasks. Our approach allows for the re-ranking of estimated PTM rankings using PTM-specific task representation. Since more forward passes consume more resources, MODEL SPIDER further improves performance and provides dynamic resource adaptation option with PTM-specific features.
Illustrated in Figure 8 is an example of model re-ranking in the context of heterogeneous multi- source model zoo. The MODEL SPIDER, after extracting PTM--pecificc task repr., accomplished more precise PTM ranking. We re-construct the PTM-specific task repr. the Dogs dataset pre-trained. Our investigation focuses on the Aireraft downstream dataset, and intriguingly, we discover that PTMs trained on multi-scenario multi-target datasets possessed inherent advantages when applied to the aircraft domain. This advantage can be attributed to their generally strong recognition capabilities for diverse targets. Remarkably, even models pre-trained on the Food dataset demonstrated exceptional performance on the Aircraft dataset. Despite the notable dissimilarities between the Food and Aireraft datasets, we conjecture that the cedd-pr--rrined models not only exhibit proficiency in recognizing multiple targets, encompassing various food items but also harbor latent potential for fine-grained recognition within the food domain. Consequently, these PTMs transfer their fine-grained recognition capacity to the aircraft domain. In contrast, the Dogs dataset, characterized by a narrow focus on single biological species, impedes successful transfer to Aircraft task.
25

---

Table 7: The confidence interval (in 95%) for few-shot evaluation (10 examples per class and 30 trials) in Table of the main text. Specific features of Top-3 ranked PTMs are employed.
Downstream Target Dataset CIFARIO
Method
Aircraft
Caltech101
Cars
CIFARI00
DTD
Pets
SUN397
Few-Shot Evaluation (10-example per class)
H-Score 0
-0.01440.14 0.078
0.375
0.018
0.005=a.:4
-00.::44-00022..1-00000000 0.006
0.853=0.02
NCE 97
0.273+0.05 0.53440.07
0.597
0.267=0.08
0.232+0.04 0.362+0.06 0.352+0.09
0.793=0.00
LEEP
0.069 -0.0384..01
0.476g0.03
0.530_0.04 0.47140.o2- -ot.oooo-0.1110...
0.567 0.468_a.00
N-LEEP 55
-0.559
0.476=0.05
0.743
0.515=o.06
0.707
0.027
0.71320.04
0.812=0.02
LogME
0.341 40002
0..5
0.497
0.718
0.69844.0:
0.4077.0.00
0.657.0.00
0.817
PACTran
A7
0.136 0.262 o2
0.48440.05
0.631 #0.02
0.61440.06 -0.227
0.701 +0.03
0.477
OTCE
5X
-0.316+0.01 -0.050
+0.127
0.505 0.515
0.210 0.406
LFC
25
0.226 -0.226=0.or
-0.23520.02
0.330=o.04
0.271
-0.669
0.0559
-0.151_0.00
Ours
0.382 0.711 t0.00
0.727 =0.01
0.870 +0.01
0.686 0.977+t.02
+0.02
0.717
0.933 00.02
#0.02
+0.0s
Downstream Task
Model Zoo
SU
Pre-trained on SUN397
Pre-trained on Dogs
Pre-trained on Food
Pre-trained on Caltech1@1
Aircraft
Ground-Truth:
42 PTMs
PTM Ranking: Food SUN397
Caltech101
Dogs
Baselines:
Update Dogs- pogs-pretr measure w/ PTM--Seciific task representation"
H-Score: Caltech1e1 Dogs
SUN397 Food
7
Bad Result
NCE:
Dogs Caltech101
Food > SUN397
Bad Result
Upditted Tash Rept
MODEL SPIDER (Ours):
-2.363
w/o PTM-spacific feature: Food SUN397 >
Dogs Caltech1e1
X
Upditeed
Update - -27922 < -2.63
Orightl
w/ PTM specific feature: Food SUN397 > Caltech101 >
Dogs
Figure 8: Illustrative re-ranking example with enhanced ranking through PTTM-specific task represen- tation.
The substantial disparities between the datasets pose significant challenge for conventional baseline methods, often failing to prioritize the Food-pre-trained model. However, MODEL SPIDER success- fully learns to rank the Food-pre-traiie trained one and, through meticulous screening process followed by result re-ranking, MODEL SPIDER identifies that the Caltech 101 -reeetrrad model outperforms the bogssprrettainee one due to its superior multi-target recognition capabilities, thereby exhibiting enhanced transfer performance.
D
More Details
D.1
Comparison of the time consumption and memory footprint (details Figure 1(c)
Figure 1(c) shows the average efficiency performance comparison over 5 baseline approaches and MODEL SPIDER. The & = 0, 3, k = 6, = 36, and k = 42 correspond to inference w/o PTM-specific features, w/ 3, 6, 36, and 42 ones. Following 113, we measure the wall-clock time (second) and memory footprint (MB) with code instrumentation.
D.2 Datasets Description
We show the datasets description Table 9 with some examples Figure 9 covered in this paper.
26

---

Table Comparison of the time consumption and memory footprint of fine-tuning, RankAgg. different baseline approaches, and MODEL SPIDER to rank the PTMs.
Approaches
Wall-clock Time (second)
Memory Footprint (MB)
RankAgg
7,318.06
10.405.32
tine-tuning (all parameters)
614,497.22
13,872.81
H-Score
2,358.70 70
9,367.74
NCE
2,196.53
8,121499
LEEP
2,215,06
8,209.33
M-LEEP
4,963.01
9,850.84
LogME
2,571.99
8,217.80
MODEL SPIDER (w/o PTM-Specific Feature)
52.36
608.01
MODEL SPIDER (w/ 3 PTM-Speciee PTM--pccific Feature)
105.19
1,386.43
MODEL SPIDER (w/ 6 PTM-Specific Feature)
175.87
1,760.28
MODEL SPIDER (w/ PTM-Specific Feature)
2,180.23
7,989.35
MODEL SPIDER (w/ (12) PTM-Specific Feature)
2,402.77
9,954.09
Table 9: The number of training images, testing images and classes with the link to download the dataset.
Datasel
Training Images
Testing Images
Chases
URL
Aineral
(59
5,677
3311
100
hSspe: / feve robotes or ac me uyer wwp//att wwt.t//tt/tteeeeeeeeeeeeee Antroraft
II [0
$0,000
10,000
10
CIEAR100 19
50,000
10,000
100
Jaraset
Trape
STL.10 23
5,000
8000
10
B1 129 /fce atant oed adu/ acoalesr at 107
Oxford Flowers 102 M55
2040
6149
102
N1188 leee rohora aeruer ypp PAPTRTTT Lovre/ 1027
CUB 200 100
5991
$793
200
Vir10 carvect cur Surveess VEEL-001/0O 200 2011. htal
Sunforl
12.000
8,580
120
51
2007
SVHN
HEPP atan ard nousenn nontr
Food-100
75,750
25,250
B1126 // bee Ventorr ore CATASAEE CArbiog 1ood101
NABirds 1
23,979
24,633
555
RE12 Kh0ne11 1ir on Drr. Toa
NWPU RESISC44 15
25,200
0,300
45
Oxford DIT
3,680
3,669
AID [06
54
VLCCS
4,000
Office Hlome
11.231
11,231
B1156 7/ere Sen Or CA SEA R0
SUNS97
[00
87.003
21.751
397
B: 7/91 prance 1o 000 20000000110 201661//0011//00
ImageNet IK 81
1,281167
20000
1000
net. org/ooo
E Discussions
There two promising directions of MODEL SPIDER. First, MODEL SPIDER exhibits the unique characteristic not relying on the forward pass of the model zoo, thereby enabling the evaluation of task compatibility with classical machine learning models. Then, MODEL SPIDER could be applied to the case when we use other criteria in addition to fine-tuning performance to measure the fitness between model and task.
27

---

20
Flowers
CUB2211
Dogs
EwrSAAT
SmalINORB
SVHN
Food
NAlirds
Resise15
Pet
PACS
VLCS
SUN391
Figure Examples of datasets.
28


Reviews:
[{"review_id": "rMUJaVboKu", "reviewer": "Reviewer_Gb1N", "summary": "The paper proposes a method called MODEL SPIDER for selecting the most suitable pre-trained models (PTMs) for a given downstream task. The proposed method aims to maintain a balance between efficiency and accuracy in the selection of PTMs. To achieve this, the authors tokenize all PTMs and tasks into vector representations that capture their general properties and their relationship with each other. During the training process, they dynamically select a partial set of PTMs and incorporate the specific tokens into the sampled tasks. During deployment, the authors employ a coarse-grained PTM search to narrow down the candidate PTMs and then fine-tune the selected PTMs for downstream use. \n\nThe proposed approach is evaluated on several benchmark datasets, and the results demonstrate that it outperforms existing PTM selection methods in terms of efficiency and accuracy. As part of the analysis of the effectiveness of the proposed method, the authors conduct ablation studies. They find that incorporating PTM-specific features and prompts improves the performance of the proposed method significantly.", "questions": "Do we need to represent all data instances, or can we use a sampling strategy to select the most representative examples from the data for encoding?", "limitations": "The paper does not include a limitation section that explicitly discusses the drawbacks or challenges of the proposed approach.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper presents a clear and well-motivated problem statement in the Introduction: how to select the best pre-trained model (PTM) for a given task. The approach section introduces the necessary preliminary and explains almost parts of the proposed method in detail. \n\n* The method is novel and interesting, as it constructs tokens for both the PTMs and the target task, and then measures their similarity to find the optimal match. This way, the method can leverage the rich information encoded in the PTMs and adapt it to different tasks.\n\n* The evaluation is comprehensive and detailed, covering 10 PTMs from five architectures and 9 downstream datasets for classification and regression tasks. The paper also compares the method to several strong baselines and shows that it outperforms them in selecting the most suitable PTM for each task", "weaknesses": "* The fitness function is a neural network that maps the PTM and task tokens to a scalar score, but the paper does not specify how this score is calculated or which threshold is compared with.\n\n* The paper lacks details on the design and training of the task encoder, which is a crucial component of the method. In line 42-43, the authors mentioned that they use a Transformer module and refer to the reference [72] \"Attention is all you need\". In my opinion, this is an important detail to explain how the tokenization process is performed. \n\n* The paper does not explain how the authors handle noise and irrelevant data in the task tokenization process.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank Reviewer Gb1N for the insightful review** and for recognizing the strengths of our paper and the Model Spider method for pre-trained model selection. **We're grateful for Reviewer Gb1N's recognition of our novel approach**, which leverages tokens representing both pre-trained models and tasks to measure their similarity.\n\n**We responded to reviewer Gb1N's inquiries as follows:**\n\n\n**Question 1**: How this score $\\hat{\\mathrm{t}}_{\\phi_m \\rightarrow \\mathcal{T}}$ is calculated and dose it need a threshold.\n\n**Answer 1**: Thank Reviewer Gb1N for the insightful feedback. **We calculate the model-task score using Equation 6, as outlined in lines 188-190.** Specifically, we employ **a transformer-based module** to compute the similarity between model and task tokens. This is achieved through a one-layer Transformer, a self-attention mechanism that accommodates various inputs, involving multi-head self-attention, multi-layer perceptron, and layer norm blocks in alternating layers. **The input to the Transformer consists of a union set of model and task tokens** denoted as\n\n$\\boldsymbol{z}=\\left[\\boldsymbol{\\theta}_{\\boldsymbol{m}}, \\boldsymbol{\\mu}(\\mathcal{T})\\right] \\in \\mathbb{R}^{d \\times(1+C)}$,\n\nleading to the similarity score $\\hat{\\mathrm{t}}_{\\phi_m \\rightarrow \\mathcal{T}}$ computed as:\n\n$\\operatorname{sim}\\left(\\boldsymbol{\\theta}_m, \\boldsymbol{\\mu}(\\mathcal{T})\\right)=\\mathrm{FC}(\\operatorname{transformer}(\\boldsymbol{z})[0])$.\n\nFor downstream tasks, we calculate scores for multiple pre-trained models, creating a ranking based on these scores. **Our focus is on the ranking order, where higher scores correspond to better downstream performance.** (lines 118-121). The calculation process **does not involve a threshold** for comparison. **The weighted** $\\tau_w$ metric evaluates the quality of rankings **based on relative order rather than absolute scores**. The $\\tau_w$ considers differences in ranking positions and is not concerned with specific score. **We will provide more details in the final version**.\n\n&nbsp;\n\n**Question 2**: What are the construction details of the task encoder.\n\n**Answer 2**: Thank Reviewer Gb1N for the perceptive feedback. The method's task encoder is indeed a critical component. **The task representation (tokens) is derived using an additional self-supervised training tokenizer, denoted as** $\\psi$, **with relevant explanations in the Task Token section (lines 177-179, 272).** This encoder is an additional frozen unit **with the same parameter magnitudes as the pre-trained models to be ranked**.\n\nWe detail in line 272 (or in Appendix line 35) that $\\psi$ **is realized through a pre-trained Swin-B-based EsViT** [1,2] (accessible at https://github.com/microsoft/esvit), trained on ImageNet-1K using self-supervised learning. In our experiments, this encoder functions as a feature extractor. **Additionally, the Transformer-based module (in lines 42-43) details for assessing model-task similarity in Model Spider** are expounded upon in lines 186-188 and response to the previous question. We will further augment this in the final version.\n\n[1] Efficient self-supervised vision transformers for representation learning.\n[2] Swin transformer: Hierarchical vision transformer using shifted windows.\n\n&nbsp;\n\n**Question 3**: How to handle noise and irrelevant data in the task tokenization process.\n\n**Answer 3**: Thank Reviewer Gb1N for the thoughtful inquiry. **While our task tokenizer might include noise and irrelevant data, Model Spider effectively mitigates noise effects on task representations.** We sample tasks from a mixed dataset during training (lines 273, 293, and Appendix line 149), enhancing diversity. **Model Spider adeptly captures task information,** utilizing numerous training tasks (Appendix line 153) and random sampling to ensure task variability, minimizing noise impact.\n\nMoreover, in the experiments presented **in the latter part of Table 1**, we assess the performance of pre-trained model ranking with employing a few-shot manner (10 examples per class), and each result is repeated 30 times for evaluation. **In few-shot tasks, where the sample size is limited, noise and irrelevant data have a more pronounced impact on task representations (tokens).** Nonetheless, Model Spider demonstrates stable and superior performance even under these conditions. We appreciate Reviewer Gb1N's consideration of these aspects within our methodology.\n\n&nbsp;\n\n**Question 4**: Do we need to represent all data instances, or use a sampling strategy for encoding.\n\n**Answer 4**: Thank Reviewer Gb1N for the precise question. In Model Spider, **we employ random sampling** to create task tokens by selecting 50 instances per class, elaborated in Appendix, line 153. **Our method does not necessitate representing every instance or adopting a sampling strategy to choose the most representative examples.** Model Spider's robustness enables effective learning from randomly sampled task representations (tokens). Our approach of random task sampling is detailed in Appendix lines 153 and 187: single-source experiment (Table 1) uses around 1k tasks, and multi-source (Figure 3) employs about 4k tasks. Ample sampling is employed to mitigate the influence of randomness during training and testing to the fullest extent possible.\n\n&nbsp;\n\n**Question 5**: The paper does not include a limitation section that explicitly discusses the drawbacks or challenges of the proposed approach.\n\n**Answer 5**: Thank Reviewer Gb1N. **Our Discussions and Limitations section can be found on line 252 of the appendix.** We will ensure its placement is appropriately adjusted in the final version.\n\n**We sincerely thank Reviewer Gb1N for their insightful review and valuable questions**. We are committed to addressing these queries thoroughly in the final version."}, {"review_id": "8PfdVglMa1", "reviewer": "Reviewer_PpFQ", "summary": "This paper proposes a method to select the \"best\" pre-trained model for a given task. This problem is important given the large number of available pre-trained models. The key behind Spider relies on tokenizing both the models and the tasks by summarizing their characteristics into vectors. More specifically, the authors use a general encoder and measure the similarity between tokens in a supervised manner: the ranking of models are obtained through some historical tasks. Normally, one would take a list of pre-trained model, (optionally) freeze the feature-extractor part, add a randomly-initialized head on top, fine-tune on the dataset, and then measure their transferability. This process is computationally intensive.\n\nThe proposed model Spider first randomly sample training tasks and assume that we can compute the transferability for M pre-trained models and thus, their ranking. Given this dataset, the model is trained to learn a similarity function to mimic this ranking. The only features that are used are task tokens and tokens from the models. A model token consists of a representation to reflect how good the pre-trained model is in general. A task token is an embedding that represent a class in a dataset. Finally, different re-ranking strategies are proposed with efficiency-accuracy trade-off.\n\nTransferability assessment is not my domain. Nevertheless, the proposed method seems sound to me. The experiment section is really complete and includes many datasets and baselines for single-source and multi-source model zoo. Finally, the ablation study emphasizes the importance of RankAgg.\n\nOverall, the paper is well written, the approach seems novel, and again, given that this topic is not my domain, I don't see any reason to reject this paper.", "questions": "I have no question", "limitations": "There is not a limitation section.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Efficient model selection method for a given task\n- The method is novel\n- Good performance in the experiment section", "weaknesses": "- Some concrete examples of Task & Model tokens would be appreciated ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely value the insightful suggestions** provided by Reviewer PpFQ. **Within our General Response PDF file**, we have provided **concrete examples of model-task tokens**, demonstrating with pre-trained models like Food, SUN397, Caltech101, and Dogs datasets. This presentation effectively underscores the semantic relationships between various models and tasks. We are fully committed to meticulous refining and seamlessly **incorporating these details into the final version**.\n\nOnce again, we extend our heartfelt and profound gratitude to Reviewer PpFQ. **Thank you very much**."}, {"review_id": "3qhaj8Ykmj", "reviewer": "Reviewer_iJup", "summary": "This paper introduces a very interesting approach named \"model spider\", to address the challenging problem of selecting suitable Pre-Trained Models (PTMs) from a large number of options to fit the target tasks. Instead of relying on time-consuming and computationally heavy forward or backward passes over all PTMs, the model spider tokenizes both PTMs and tasks, summarizing their characteristics into vectors for efficient PTM selection. Experiments show that model spider performs well in various configurations of model zoos, providing a balance between efficiency and selection accuracy.", "questions": "See weaknesses.", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem and idea of this paper are quite interesting.\n2. The proposed method is simple yet effective.\n3. The results are quite encouraging.", "weaknesses": "1. The results are tested on vision tasks. It is not clear whether the proposed method can be generalized to tasks in other modalities.\n\n2. The model size used in this work is relatively small (only up to tens of millions of parameters). It is not clear whether the proposed method can handle larger models, such as ViT, BERT, and GPT. It would be more exciting if larger models (especially large language models) can be easily evaluated.\n\n3. The proposed method still requires some samples to train on new tasks. It would be better to consider using some methods (e.g., meta-learning) to generate initial tokens for tasks so that new tasks can be handled without additional training samples.\n\nMinor: \nThe term \"token\" is a little confusing since the meaning of token in this paper is different from the common meaning of tokens in PTMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate Reviewer iJup's perceptive suggestions and valuable feedback**. The suggestions have been instrumental in enhancing our paper. For some questions, our responses are as follows:\n\n**Question 1**: Whether the proposed method can be generalized to tasks in other modalities.\n\n**Answer 1**: Thank Reviewer iJup for the valuable suggestions from Reviewer iJup. To assess the performance of Model Spider in other modalities, such as natural language processing, we followed the approach outlined in LogME. **We introduced cased BERT-D, uncased BERT-D, RoBERTa, and RoBERTa-D as pre-trained models to evaluate the performance of Model Spider**. We conducted our evaluation on the MRPC and SST-2 downstream datasets. **The comprehensive results are presented in Table 2 of the general response, highlighting Model Spider's consistent and robust generalization capabilities**. We will include these additional experimental results in the final version for further clarification.\n\nAdditionally, we have showcased the generalization results on large language models in Table 3 of the general response. These results further highlight that Model Spider maintains strong scalability and exceptional performance across diverse tasks.\n\n&nbsp;\n\n**Question 2**: Whether the proposed method can handle larger models, such as ViT, BERT, and GPT. It would be more exciting if larger models (especially large language models) could be easily evaluated.\n\n**Answer 2**: Thank Reviewer iJup for the forward-looking feedback. We have indeed addressed the suggestions comprehensively. As described in Answer 1, **we incorporated the ViT-B/16 [4] into our existing model zoo**, covering larger models. Moreover, in alignment with the inquiry about NLP pre-trained models, **we evaluated the model ranking capabilities on the BERT series**. These results are presented in Table 1 of the general response. Additionally, in Table 3 of the general response, **we showcased Model Spider's performance on GPT-type large language models**, reaffirming its remarkable extensibility and outstanding performance.\n\n| Method | Aircraft | Caltech101 | Cars | CIFAR10 | CIFAR100 | DTD | Pets | SUN397 | Mean |\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|\n| NCE | 0.523 | **0.681** | **0.790** | 0.701 | 0.659 | 0.305 | 0.681 | 0.762 | 0.638 |\n| LEEP | 0.318 | 0.107 | 0.682 | 0.591 | 0.660 | 0.114 | 0.514 | 0.486 | 0.434 |\n| Model Spider | **0.693** | 0.679 | 0.781 | **0.879** | **0.955**  | **0.699** | **0.812** | **0.869** | **0.796** |\n\n**Table 1**: Comparison of NCE, LEEP, and Model Spider performance after extending the pre-trained model zoo in the original Table 1, including the addition of the ViT-B/16 pre-trained model. The remaining experimental setup is consistent with Table 1 in the paper.\n\n| Method | MRPC | SST-2 |\n|-------|-------|-------|\n| LogME | 0.493 | 1.000 |\n| Model Spider | **0.654** | 1.000 |\n\n**Table 2**: Ranking performance of pre-trained model ranking on NLP tasks.\n\n| Method | Operating System | Computer Architecture | College Physics | College Chemistry | Electrical Engineer | Metrology Engineer | Advanced Mathematics | Probability and Statistics | Modern Chinese History | Legal Professional |\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| Random | 0.083 | -0.237 | -0.185 | 0.021 | -0.532 | 0.181 | -0.010 | 0.013 | -0.005 | -0.605 |\n| ChatGPT-Top1 | 0.136 | 0.052 | 0.024 | -0.162 | 0.232 | -0.061 | -0.198 | 0.172 | 0.105 | 0.305 |\n| Model Spider | **0.720** | **0.682** | **0.311** | **0.686** | **0.308** | **0.682**  | **0.184**  | **0.243** | **0.891** | **0.737** |\n\n**Table 3**: Ranking performance of pre-trained model ranking on GPT-type LLMs, measured by weighted $\\tau_w$. The horizontal axis represents evaluation datasets from C-Eval benchmark.\n\n**For more details, please see the general response.**\n\n&nbsp;\n\n**Question 3**: It would be better to consider using some methods (e.g., meta-learning) to generate initial task tokens so that new tasks can be handled without additional training samples.\n\n**Answer 3**: Thank Reviewer iJup for the sincere reply. Our approach primarily revolves around learning to rank, wherein the model's efficient ranking capability is acquired by learning from its historical performance across a diverse range of tasks. **Our learning process also shares connections with the Few-Shot Learning (FSL) scenario and metric-based approach of Meta-Learning**.\n\nModel Spider starts its journey from the performance achieved through fine-tuning established models. It then advances its capabilities through training on additional tasks, **subsequently enabling the generalization to tasks unseen before**. This is exemplified in **the cross-task setting depicted in line 267, Table 1, and Figure 3 of the paper**. Model Spider's ability to sustain its ranking across tasks of varying types and domains is notable. This capability extends **beyond the generalization power limited to a singular task type and aligns with the notion of FSL within the meta-learning framework**. This involves acquiring recognition abilities from specific subsets of categories in FSL and extending these abilities to categories that have not been encountered previously. We intend to incorporate further relevant descriptions of this aspect in the final version of our work.\n\n&nbsp;\n\n**Question 4**: The term \"token\" is a little confusing since the meaning of token in this paper differs from the common meaning of tokens in PTMs.\n\n**Answer 4**: Thank Reviewer iJup for the valuable feedback. We appreciate the suggestion, and in the final version, **we will use the term \"model representation\" or a more appropriate phrase instead of \"token\" to avoid confusion and ensure clarity**.\n\nOnce again, **we are grateful for Reviewer iJup's time and effort in reviewing our paper**. Thank Reviewer iJup for the continued support and engagement in advancing the field."}, {"review_id": "pXO9tiT83m", "reviewer": "Reviewer_1u8a", "summary": "This paper introduces Model Spider, a unique method to efficiently and accurately rank Pre-Trained Models (PTMs) for a specific task within a model zoo. Model Spider innovatively creates tokens for both PTMs and tasks, encapsulating their characteristics in a manner that facilitates an efficient selection process. It utilizes a separate set of training tasks to learn how to construct these tokens and calculate the fitness score between a model-task pair. The paper also presents a strategy to update the task tokens based on the semantics specific to the top-ranked PTM candidates, improving the final selection. The key contributions of this paper are the innovative method of tokenizing tasks and PTMs for easy ranking, and the ability of the system to incorporate task-specific forward results of certain PTMs within resource limitations. Through rigorous testing across various model zoo configurations, the authors demonstrate the efficacy of Model Spider, showing significant improvements in PTM selection and efficiency. The work represents an innovative solution to the challenge of sifting through the proliferating number of PTMs to find the most suitable model for a given task.", "questions": "1. How good can the method be applied to the pre-trained large language models?", "limitations": "1. Good tokenization of pre-trained models (PTMs) depends highly on the number of high-quality PTMs. How the Model spider perform when the number of PTMs M and the number of tasks varies?\n2. The training of Model Spider depends on the RankAgg. As authors explained (in the appendix), RankAgg introduces a significant computational burden. How could one get a RankAgg model to be used for training Model Spider in the first place?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Proposed methods show quite significant improvements over the strong baselines.\n2. Methodology of tokenizing PTM looks new. PTMs are tokenized by mapping them to unsupervised trained task tokens. The method looks sensible.\n3. Applying learning to rank to rank the model fitness is also new and interesting.\n4. Extensive experiments with ablation studies.", "weaknesses": "1. “Hyperparameter k” in Figure 1 is not explained at all. In the main text, notation k is also not clearly defined, only top-k mentioned here and there. Readers have to guess k means the number of top ranked Pretrained Models.\n2. Some important results and discussions are included in the Appendix, but no reference in the main text. Please refer readers to the appendix for the useful information.\n3. Figure 3 is a bit hard to read. What information should I draw from the figure, besides correlation number?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We deeply appreciate Reviewer 1u8a's insightful queries and constructive input**. The engagement has undoubtedly enhanced the quality and coherence of our paper. Here are our responses:\n\n**Question 1**: Hyperparameter $k$ in Figure 1.\n\n**Answer 1**: The hyperparameter $k$ corresponds to **the number of top-ranked pre-trained models (lines 51, 249, and 319)**. In Section 4.4 \"Re-ranking with Efficiency-Accuracy Trade-off,\" we introduce a two-step process in Model Spider. Initially, a general task token is used for the initial ranking of pre-trained models. Then, the top-$k$ **pre-trained models perform forward passes on the target task**. This generates PTM-specific task tokens, aiming to enhance our ranking outcomes. For instance, with a smaller value of $k$, Model Spider exhibits high efficiency in model ranking, confirmed by the results in Appendix Table 4: \"Comparison of the time consumption and memory footprint.\" Moreover, in lines 319-323 of the main text and Figure 4, we demonstrate that as $k$ increases, the overall ranking accuracy also improves.\n\n&nbsp;\n\n**Question 2**: Refer readers to the Appendix for useful information.\n\n**Answer 2**: Thank Reviewer 1u8a for the valuable reminder. We will include relevant references in the final version, such as comparing **the time consumption and memory footprint of various $k$, other related ablation studies**, and so on.\n\n&nbsp;\n\n**Question 3**: The information in Figure 3, besides the correlation number.\n\n**Answer 3**: In Figure 3, we can observe **the direct linear relationship between the predicted model ranking and the downstream fine-tuned accuracy**. The x-axis represents the fine-tuned accuracy of the corresponding pre-trained models, and the y-axis reflects the scores predicted by the model ranking method. Higher scores imply a higher chance of superior performance. For instance, the better result, like Ours with a score of 0.678, on the Pet dataset compared to LEEP with a score of 0.361 exhibits the fitting line that closely approximates a slope of $1$. **The values of the points along the y-axis offer insights into the distribution of the method's evaluated scores**. Different colors of points correspond to different model structures. **Multiple points of the same color stem from diverse tasks during pre-training**. For example, ResNet-50 models mostly perform well on the Aircraft dataset, whereas this trend is not necessarily consistent on the Pet dataset.\n\n&nbsp;\n\n**Question 4**: Application to the pre-trained large language models.\n\n**Answer 4**: We explore the performance of Model Spider in NLP tasks and large language models. **The results are provided in Tables 2 and 3 of the general response**. We sincerely appreciate the forward-looking suggestions from the reviewers, and **we will certainly incorporate the relevant experiments in the final version of the paper** to achieve even greater advancements in the future.\n\n&nbsp;\n\n**Question 5**: How the Model Spider perform when the number of PTMs $M$ and the number of tasks varies.\n\n**Answer 5**: In Table 2 of the Appendix, we conducted ablation studies on the performance of Model Spider as the size of the pre-trained model zoo dynamically increases. It is observed that changes in the number of models within the pre-trained model zoo can indeed influence the performance of Model Spider, with **greater model diversity posing increased challenges for the model ranking task**.\n\nIn Table 1 of the general response, we further **expanded the model repository by including larger models like ViT-B/16**. This extended coverage enhances the diversity of the model zoo. Model Spider continues to demonstrate consistently high-performance levels in this enriched setting.\n\nAdditionally, concerning the scope of training tasks, we downsized the training task set for training Model Spider. **The results of this reduction are presented in the Appendix at line 194 and Figure 2 (yellow compared to pink)**. Our results show that, apart from a slight drop in performance on the DTD dataset, **Model Spider maintains strong overall performance even with fewer training tasks**. This underscores Model Spider's ability to effectively capture diverse pre-trained model characteristics despite reduced training task diversity. These findings highlight the robust performance of Model Spider across different numbers of pre-trained models and tasks.\n\n&nbsp;\n\n**Question 6**: RankAgg introduces a significant computational burden. How could one get a RankAgg model to be used for training Model Spider?\n\n**Answer 6**: As mentioned in lines 308 of the main text and 48 of the Appendix, RankAgg requires pre-computation and entails some overhead. However, **it is significantly more efficient compared to full parameter fine-tuning.**\n\nIn the experiment detailed in **line 69 of the Appendix and depicted in Appendix Figure 3**, we elaborate on the motivation behind introducing RankAgg. We empirically observe that popular approaches such as NCE, LEEP, and LogME exhibit \"good but diverse\" pre-trained model ranking orders. Consequently, ensembling their ranking outcomes into a stronger single order appears to be an intuitive way to enhance transferability estimation quality.\n\nThe algorithmic process of RankAgg is **outlined in Algorithm 1 (line 16) of the Appendix, as well as in lines 73 and 83**. It involves initially sampling a subset of training tasks containing partial samples, calculating the model rankings derived from NCE, LEEP, LogME, and H-Score methods, and then **aggregating these rankings using RankAgg to obtain improved results**. The RankAgg method is designed to be plug-and-play, demonstrating excellent scalability and ease of use.\n\n**We sincerely thank the Reviewer 1u8a for the valuable insights**. We are fully committed to incorporating these elucidating descriptions into the final version of the paper."}, {"review_id": "xSg3XKZ6TC", "reviewer": "Reviewer_mKeC", "summary": "This paper investigates how to select the most suitable PTM given a target task efficiently and accurately.\nA novel approach called Model Spider has been proposed. It learns to encode both PTMs and tasks into vectors and measures their similarity, which is further used to rank the PTMs. It can also incorporate task-specific forward results of PTMs for more accurate re-ranking when resources budgets allow.\nExtensive experiments have been conducted to verify the effectiveness of the proposed method.", "questions": "In addition to the questions above, I have the following questions:\n1. What's the motivation for using Equation (4) to train the model to capture the ranking order? There are many other methods that can train the model to learn to rank, such as the pairwise BPR and listwise ListMLE. It is recommended to verify the impact of different ranking loss functions.\n2. Does the proposed method require re-training whenever a new PTM comes? If so, the proposed method seems costly when applied in real-world scenarios.", "limitations": "No, the authors have not discussed the limitations of the proposed method. The authors are recommended to explain whether the proposed method is limited to classification tasks and how to adapt it to other tasks.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of encoding PTMs and downsteam tasks into vectors for PTM ranking is well-motivated.\n2. The overall presentation of the proposed method is well-organized and generally easy to follow.", "weaknesses": "1. The generalization ability of the proposed method has not been well-verified. The supervised training method is known to result in models with poor generalization ability. It seems that the proposed method heavily relies on the frozen encoder $\\psi$ to capture the relevance between different tasks. What if the new task is quite different from the tasks used in training? When evaluating the proposed method, all the downstream tasks are about image classification. The authors should evaluate the generalization ability of the proposed method with more diverse downstream tasks.\n2. The proposed method is highly related to the classification task as it uses the class centers as the task token in Equation (5). However,  many tasks do not have such \"classes\", such as regression tasks and generation tasks. How to adopt the proposed method to such tasks has not been elaborated.\n3. Only 10 PTMs are used when evaluating the proposed method on a single-source model zoo, which is not enough. It is recommended to evaluate the proposed method in the NLP domains, where there are numerous pre-training models and diverse downstream tasks, which can be used to better verify the generability of the proposed method to a new task.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank Reviewer mKeC for the valuable insights and thoughtful questions**. The feedback enhances our work's clarity and robustness. Here is our response:\n\n**Question 1**: Generalization ability and dependency on frozen encoder $\\psi$. The new task differs from the training ones.\n\n**Answer 1**: **Our method, Model Spider, showcases broad task generalization for pre-trained model ranking. It minimizes dependency on the potency of $\\psi$**. Model Spider learns task tokens from $\\psi$, reflecting task importance relative to pre-trained models, **not an absolute need for exceptional task representation**. In Appendix Figure 2 and line 194, we introduce an attenuated version of $\\psi$ in a tiny scale and conduct ablation studies(green vs. pink in the figure).\n\nMoreover, when the downstream task significantly differs from the tasks during training, Model Spider maintains robust performance.\n\n+ **New Field Tasks in Natural Language Processing**\n\nTransitioning from visual recognition to NLP, **we employ rule-based task categorization to create NLP sub-zoos**, refining model ranking. We demonstrate the ranking capabilities in NLP and large language model contexts in Tables 2 and 3 of the general response. Model Spider continues to showcase exceptional applicability and generalization.\n\n+ **New Domain Tasks with Out-of-Distribution**\n\nFor downstream tasks beyond training distribution, the general feature extractor $\\psi$ might have limited representation capacity for that particular task. To illustrate this, we conduct experiments that analyze $\\psi$'s performance on certain datasets. We extract features using $\\psi$ and compare the distances to training class centers, akin to Nearest Class Mean (NCM) [1]. In the following, we compare the accuracy (in %) differences between full-parameter fine-tuning and NCM on various datasets. Larger differences indicate more disparities in downstream tasks. Notably, datasets like DTD and Pet, **which deviate notably from the pre-training data distribution, highlight $\\psi$'s limited representation capacity in such cases**.\n\n| Method | CIFAR-100 | Caltech-101 | DTD | Pet | EuroSAT |\n|-------|-------|-------|-------|-------|-------|\n| Fully Fine-tuning | 69.66 | 85.62 | 63.39 | 84.35 | 93.62 |\n| Feature-based NCM | 66.41 | 85.80 | 57.19 | 78.64 | 89.11 |\n\nAs shown in the Table 1 of the paper, we observe that Model Spider still demonstrates exceptional performance on DTD and Pets datasets.\n\nFurthermore, **we extend our evaluation to the EuroSAT remote sensing dataset**, demonstrating Model Spider's effective generalization ability.\n\n| Method | EuroSAT |\n|-------|-------|\n| LEEP | 0.395 |\n| LogME | 0.510 |\n| Model Spider | **0.682** |\n\n+ **New Type Tasks from Regression**\n\nAs shown in Table 2 of the paper (line 277), Model Spider excels in regression tasks, capturing relative model proficiency order.\n\nThe $\\psi$ captures **the relative order of task representation rather than absolute task-specific information**. It's most effective when a semantic connection exists between models and tasks, enhancing model ranking. Furthermore, for the diagram spanning pre-training to fine-tuning, the pre-training knowledge is maximally stimulated when downstream tasks are semantically related to models. Otherwise, the fine-tuned performance of unrelated pre-trained models resembles that of randomly initialized models.\n\n&nbsp;\n\n**Question 2**: Task representation constrained by class centers.\n\n**Answer 2**: In our method, task tokens encapsulate task representations, with various implementation options. The class center is just one specific to classification. **Even without categorical information in the target tasks, meaningful task representation can be established by forming semantically relevant prototype clusters.** For instance, in regression tasks (e.g., dSprites, UTKFace in Table 2), Model Spider constructs semantic clusters representing age and position, yielding favorable results.\n\nWe explore alternative task representation. In DTD dataset of Figure 3, we sample around 235 samples and compute the covariance of these samples. The results are as follows:\n\n| Method | DTD |\n|-------|-------|\n| Prototype Token | **0.549** |\n| Covariance Token | 0.513 |\n\nIn conclusion, Model Spider is capable of learning from diverse task representations.\n\n&nbsp;\n\n**Question 3**: Generability on larger pre-trained model zoo or to new tasks.\n\n**Answer 3**: In Figure 3 of the paper, we extend the model zoo from Table 1 (consisting of 10 pre-trained models, following LogME) **to 42 pre-trained models. This extended zoo covers three similar magnitude architectures**: Inception V3, ResNet 50, and DenseNet 201, each pre-trained on 14 datasets (lines 288-292).\n\nFurthermore, NLP and large language models are evaluated (Tables 2 and 3 of the general response), see Answer 1 for more details. Model Spider consistently demonstrates exceptional performance.\n\n&nbsp;\n\n**Question 4**: The motivation for Equation (4). Comparison to the pairwise BPR and listwise ListMLE.\n\n**Answer 4**: Building on the cross entropy loss which aims to boost the position of the one-hot label, we craft a multi-round optimization where, in round $m$, we elevate the $m^{th}$ largest item above items from $m + 1$ to $M$. We pick the $m^{\\text{th}}$ item using the $\\mathrm{dsc}(\\cdot)$ operator. Also, the denominator summation only includes items from $m$ to $M$ (not all). These nuances distinguish our ranking loss from ListMLE. **Our optimization moves from local to global ranking, capturing richer contextual ordering**. Unlike pairwise BPR which only compares items, our loss considers global ranking, reflecting the entire distribution. We include ListMLE results in Table 1 of the Appendix.\n\n\n**Thank Reviewer mKeC very much for the valuable suggestions. We will incorporate the relevant content into the final version**.\n\n[1] Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost.\n"}]

Author rebuttal:
**Dear Reviewers:**

We would like to express our sincere gratitude to Reviewers Gb1N, PpFQ, iJup, 1u8a, and mKeC for **their insightful reviews of our submission**. We are heartened by the constructive feedback and valuable suggestions each reviewer provides. We acknowledge that **all reviewers** have highlighted the motivating factors behind our proposed method, Model Spider, emphasizing its **novelty and interest**. They have also recognized **our comprehensive and detailed evaluation**, as well as the **clarity and organization of our paper's presentation**.

We are pleased by the positive feedback from Reviewers Gb1N, PpFQ, iJup, and 1u8a regarding Model Spider's **outstanding performance**. We understand the concern raised by Reviewer mKeC about **future task generalization** and the questions posed by Reviewers mKeC, iJup, and 1u8a about the applicability, such as ranking on tasks with new types, modalities, or on larger pre-trained vision or language models. **To address these, we conducted extensive experiments to show Model Spider's robustness across various scenarios**.

Moving forward, we will present our supplementary experiments.

+ **For Pre-trained Larger Vision Models**

For ranking on larger vision pre-trained models, we have incorporated **the ViT-B/16 model into our existing pre-trained model zoo**. This medium-to-large-scale vision model with approximately 100 million parameters was introduced by fine-tuning only the last linear layer (linear probing). We then **compared the performance of NCE, LEEP, and Model Spider** on this extended pre-trained model zoo.

| Method | Aircraft | Caltech101 | Cars | CIFAR10 | CIFAR100 | DTD | Pets | SUN397 | Mean |
|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|
| NCE | 0.523 | **0.681** | **0.790** | 0.701 | 0.659 | 0.305 | 0.681 | 0.762 | 0.638 |
| LEEP | 0.318 | 0.107 | 0.682 | 0.591 | 0.660 | 0.114 | 0.514 | 0.486 | 0.434 |
| Model Spider | **0.693** | 0.679 | 0.781 | **0.879** | **0.955**  | **0.699** | **0.812** | **0.869** | **0.796** |

**Table 1**: Comparison of NCE, LEEP, and Model Spider performance after extending the pre-trained model zoo in the original Table 1, including the addition of the ViT-B/16 pre-trained model. The remaining experimental setup is consistent with Table 1 in the paper.

+ **For Pre-trained Models of NLP**

To assess the adaptability of Model Spider to tasks in other modalities, such as Natural Language Processing (NLP). **We introduced cased BERT-D, uncased BERT-D, RoBERTa, and RoBERTa-D as pre-trained models** to evaluate the performance of Model Spider. Our evaluation included the **MRPC** [1] and **SST-2** [2] downstream datasets. MRPC consists of sentence pairs extracted from online news sources, where the task involves determining the semantic equivalence of sentences within each pair. On the other hand, SST-2 comprises sentences from movie reviews. The task is focused on predicting the sentiment (positive/negative) of a given sentence. **This model ranking task in the field of NLP follows the approach established by LogME** [3]. We got the following results:

| Method | MRPC | SST-2 |
|-------|-------|-------|
| LogME | 0.493 | 1.000 |
| Model Spider | **0.654** | 1.000 |

**Table 2**: Ranking performance of pre-trained model ranking on NLP tasks.

+ **For Pre-trained Larger Language Models**

Furthermore, we perform pre-trained model ranking for Large Language Models (LLM) of GPT-type, which includes ChatGPT, ChatGLM2-6B, Qwen-7B, Baichuan-7B, MOSS, bloomz-mt-176B, and Chinese Alpaca-13B. This involves **introducing the C-Eval Benchmark** [4] and employing Model Spider to rank these models' performance **across 10 sub-evaluation datasets encompassing domains** as science, technology, engineering, mathematics, and humanities. **We utilize embeddings available at** https://huggingface.co/GanymedeNil/text2vec-large-chinese **to encode the downstream tasks**. We leverage historical performance from the other datasets provided by C-Eval as the training set for Model Spider, with the objective of learning model and task representations (tokens). As the output access interface of LLMs is limited (for instance, obtaining features from ChatGPT is challenging), common methods fail in such scenarios. We compare our model ranking approach **against these two setups**: a random ranking, and ranking ChatGPT as the top model with others in random order. The obtained results are as follows:

| Method | Operating System | Computer Architecture | College Physics | College Chemistry | Electrical Engineer | Metrology Engineer | Advanced Mathematics | Probability and Statistics | Modern Chinese History | Legal Professional |
|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
| Random | 0.083 | -0.237 | -0.185 | 0.021 | -0.532 | 0.181 | -0.010 | 0.013 | -0.005 | -0.605 |
| ChatGPT-Top1 | 0.136 | 0.052 | 0.024 | -0.162 | 0.232 | -0.061 | -0.198 | 0.172 | 0.105 | 0.305 |
| Model Spider | **0.720** | **0.682** | **0.311** | **0.686** | **0.308** | **0.682**  | **0.184**  | **0.243** | **0.891** | **0.737** |

**Table 3**: Ranking performance of pre-trained model ranking on GPT-type LLMs, measured by weighted $\tau_w$. The horizontal axis represents evaluation datasets from C-Eval benchmark.

**Thank you for your consideration and support.** We are committed to addressing the reviewers' feedback and further improving our work based on the insightful comments. **We once again extend our appreciation to the reviewers for their invaluable contributions**.

[1] Automatically constructing a corpus of sentential paraphrases.

[2] Recursive deep models for semantic compositionality over a sentiment treebank.

[3] Logme: Practical assessment of pre-trained models for transfer learning.

[4] C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models, https://cevalbenchmark.com/static/leaderboard.html

Meta-review: This is a well-written paper that proposes a very interesting and novel approach for automatically selecting the best pre-trained model for a given task. This is a straightforward but effective approach that has significant practical utility. The experimental evaluation is rigorous and covers a wide range of pre-trained models, downstream tasks, etc.

The reviewers raised a few concerns about this paper, but the most critical aspects of those were adequately addressed during the rebuttal.

Given that the strengths of this paper clearly outweigh its weaknesses, this paper is suitable for publication.

The authors are strongly encouraged to carefully consider all of the reviewer feedback, including the rebuttal-related discussion, and to take meaningful steps to incorporate it into the final version of their paper.
Decision: Accept (spotlight)