SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We claim that RankAgg proposed by us cannot be considered as a direct baseline method. Firstly, RankAgg involves a substantial computational overhead when used as a stand-alone method for ranking PTMs. This is primarily due to the time and memory requirements of computing the base selection methods. Using RankAgg directly as a baseline would introduce a significant computational burden. However, we introduce RankAgg as an approximate ground-truth method for pre-computing in the training part of MODEL SPIDER. It is more efficient compared to full parameter fine-tuning.
Actually, MODEL SPIDER aims to demonstrate its broad generalization capacity by leveraging RankAgg to process an independent set of mixed data that has no overlap with the test data. This independent evaluation showcases the effectiveness of MODEL SPIDER in a real-world scenario and emphasizes its ability to handle diverse data efficiently. RankAgg itself does not play a role during the test execution of MODEL SPIDER.
A.4 How to efficiently approximate the training ground-truth of MODEL SPIDER
This section complements subsection 4.4, wherein the training and ranking of the model zoo across multiple datasets are discussed. However, obtaining the ranking for all historical tasks through brute force is computationally expensive. To mitigate this issue, we introduce a rank aggregation method denoted as RankAgg, which serves as an approximation of the ground truth ranking.
Existing PTM selection methods rely on the PTM-specific features  $\Phi_{\mathcal{T}}^{m}$  to estimate the transferability score. Different methods may have diverse score values — a PTM will be placed in different positions based on the scores provided by various methods. We empirically observe that some popular approaches such as NCE [97], LEEP [66], and LogME [113, 114] show "good but diverse" PTM ranking orders, so an intuitive approach to improving the transferability estimation quality is to ensemble their ranking results to a stronger single order.
As mentioned in subsection 4.3, given  $\{\hat{t}_1,\hat{t}_2,\dots ,\hat{t}_A\}$  as multiple rankings over the same set of  $M$  PTMs for a target task  $\mathcal{T}$ , i.e., the order sorted by the estimations of transferability via various methods, we take advantage of Copeland's aggregation method [7, 82] to ensemble the orders.
$$
\bar {t} = \left\{\bar {t} _ {\phi_ {m} \rightarrow \tau} \right\} _ {m = 1} ^ {M} = \operatorname {R a n k A g g} \left(\left\{\hat {t} _ {1}, \hat {t} _ {2}, \dots , \hat {t} _ {A} \right\}\right). \tag {7}
$$
Copeland's aggregation compares each pair of ranking candidates and considers all preferences to determine which of the two is more preferred as illustrated in Figure 5.
Taking model  $m, m'$  as an example, we define the majority relation to express the one-on-one dominance between these two models. Precisely, assuming that  $A_{m}$  approaches rank model  $m$  above model  $m'$ , i.e.,  $\hat{t}_{i,m} > \hat{t}_{i,m'}$  with  $A_{m} \times$  such  $\hat{t}_{i}$ , while the remaining  $A_{m'}$  ones do the opposite. Note that  $A_{m} + A_{m'} = A$ . The  $m > \mathbb{M}$ $m'$  just in case  $A_{m} > A_{m'}$ , and correspondingly  $m =_{\mathbb{M}} m'$  indicates  $A_{m} = A_{m'}$ . In summary, we define the aggregation score for model  $m$  as:
$$
\bar {t} _ {\phi_ {m} \rightarrow \mathcal {T}} = \# \left\{i \mid m > _ {\mathbb {M}} i \right\} + \frac {1}{2} \# \left\{i \mid m = _ {\mathbb {M}} i \right\}, \tag {8}
$$
where  $\# \{\cdot\}$  is the size of the set. The aggregation score for a model is the number of others over which they have a majority preference plus half the number of models with which they have a preference tie. In our implementation, we aggregate the results of NCE, LEEP, LogME, and H-Score.
Algorithm 1 The Training Part of the MODEL SPIDER
1: Input: fixed  $\psi$ , learnable parameters  $\Theta$ , including model repr.  $\{\pmb{\theta}_m\}_{m=1}^M$ , FC for projection, and parameters of the transformer-based MODEL SPIDER  
2: Sample training tasks  $\{\mathcal{T}_i\}$  from the additional mixed datasets as in subsection A.2  
3: Extract and save all task repr.  $\bigcup_i \{\pmb{\mu}(\mathcal{T}_i)\}$  with  $\psi$ .  
4: for all sampled task  $\mathcal{T}_i$  do  
5: for  $m = 1$  to  $M$  do  
6: if the  $m$ th PTM-specific features is available (randomly holds) then  
7: Derive the PTM-specific task repr. as mentioned in subsection 4.4.  
8:  $\hat{\mathrm{t}}_{\phi_m \rightarrow \mathcal{T}_i} = \mathrm{sim}_{\Theta}(\pmb{\theta}_m, \mathbf{P}_m^\top \pmb{\mu}_m(\mathcal{T}_i))$ .  
9: else  
10: Take model repr.  $\pmb{\theta}_m$  and estimate the similarity of model-representation pairs as Eq. 6.  
11:  $\hat{\mathrm{t}}_{\phi_m \rightarrow \mathcal{T}_i} = \mathrm{sim}_{\Theta}(\pmb{\theta}_m, \pmb{\mu}(\mathcal{T}_i))$ .  
12: end if  
13: end for  
14: From above for, the estimation scores of MODEL SPIDER  $\hat{t}$  is conducted.  
15: Calculate H-Score, NCE, LEEP, and LogME on  $\mathcal{T}_i$ .  
16: Aggregate on the results of existing approaches to obtain ground-truth  $\bar{t}$  as in subsection 4.3.  
17:  $\bar{t} = \{\bar{t}_{\phi_m \rightarrow \mathcal{T}_i}\}_{m=1}^M = \mathrm{RankAgg}(\{\hat{t}_1, \hat{t}_2, \dots\})$ .
18: Optimize the parameters of MODEL SPIDER with ranking loss  $\ell_{\mathrm{rank}}$  w.r.t. the ranking of  $\bar{t}$ .
$$
\ell_ {\text {r a n k}} (\hat {\boldsymbol {t}}, \boldsymbol {t}) = \sum_ {m = 1} ^ {M} - \log \left(\frac {\exp \left(\hat {\boldsymbol {t}} _ {\text {d s c} (m)}\right)}{\sum_ {l = m} ^ {M} \exp \left(\hat {\boldsymbol {t}} _ {\text {d s c} (l)}\right)}\right) .
$$
19: Compute  $\nabla_{\Theta}\ell_{\mathrm{rank}}$  and update corresponding parameters with the gradients
20: end for
21: Output: learned  $\Theta$ , including  $\{\pmb{\theta}_m\}_{m=1}^M$ , FC, and parameters of the MODEL SPIDER
RankAgg can become quite time-consuming when calculating PTM ranking scores for the entire dataset, mainly due to the substantial overhead of computing the base selection methods. In our experimental setup, we integrate the RankAgg method as a module during the training phase, enabling us to pre-compute the rankings for each task. The RankAgg may raise the computational burden if employed directly as a testing baseline. Therefore, we employ RankAgg for the sampled few-shot tasks to balance ranking accuracy with efficiency and only use it in the training part. Note that MODEL SPIDER learns based on the RankAgg results, but is deployed independently of it and other baseline methods. Since RankAgg summarizes the PTM generalization capability on differentiated tasks spanning multiple domains, our model derived from the pre-aggregated rankings can learn the PTM ranking ability on a broader range of unseen tasks.
A.5 How to learn the similarity of model-task representation
This section elaborates on subsection 4.1, i.e., the learning process of MODEL SPIDER, especially the Transformer based estimation. The Transformer-based module of model-task similarity. The model-task repr. is concatenated as a sequence of features. The Transformer based module naturally fits and takes such input. Concretely, in operation, transformer  $(\cdot)$  is formalized as:
$$
\begin{array}{l} \operatorname {t r a n s f o r m e r} (z) = z + \alpha (\mathbb {Q}, \mathrm {K}, \mathrm {V} = z) \\ = z + \operatorname {s o f t m a x} \left(\frac {z \mathrm {W} ^ {\mathrm {Q}} \cdot \left(z \mathrm {W} ^ {\mathrm {K}}\right) ^ {\top}}{\sqrt {d}}\right) z \mathrm {W} ^ {\mathrm {V}}. \tag {9} \\ \end{array}
$$
we apply linear projections on the query, key, and values using  $\mathbf{W}^{\mathbb{Q}}$ ,  $\mathbf{W}^{\mathbb{K}}$ , and  $\mathbf{W}^{\mathbb{V}}$ , respectively. The similarity between prototypes is measured by the inner product in the transformed space, which
Algorithm 2 The Downstream Inference Part of MODEL SPIDER
Input: target task  $\mathcal{T}$  , fixed  $\psi$  , learned  $\Theta$    
Obtain task repr.  $\pmb{\mu}(\mathcal{T})$  with  $\psi$  as Eq. 5.   
Estimate similarity of model-representation pairs as Eq. 6  $\hat{\pmb{t}} = \{\hat{\mathrm{t}}_{\phi_m\rightarrow \mathcal{T}} = \mathrm{sim}_{\Theta}(\pmb {\theta}_m,\pmb {\mu}(\mathcal{T}))\}_{m = 1}^M.$    
Select top-  $k$  PTMs via  $\hat{\pmb{t}} = \{\hat{\mathrm{t}}_{\phi_m\rightarrow \mathcal{T}}\}_{m = 1}^M$  .   
Obtain the indexes in descending order via dsc ().   
for  $m =$  dsc (1) to dsc (k) do Re-construct enriched repr.  $\pmb{\mu}_{m}$ $(\mathcal{T})$  , and update:  $\hat{\mathrm{t}}_{\phi_m\rightarrow \mathcal{T}} = \mathrm{sim}_{\Theta}\left(\pmb {\theta}_m,\mathbf{P}_m^\top \pmb {\mu}_m(\mathcal{T})\right)$    
end for   
Output: Rank PTMs with  $\hat{\pmb{t}} = \{\hat{\mathrm{t}}_{\phi_m\rightarrow \mathcal{T}}\}_{m = 1}^M$
results in larger weights of the attention head  $\alpha$ . Here  $d$  is the size of every attention head. The output of the corresponding position of the model repr. is forwarding passed through a learnable MLP and then obtains the fitness estimated score of PTM selection.
The learnable parameters in MODEL SPIDER. To learn a PTM ranker, we optimize  $M$  model repr.  $\{\theta_{m}\}_{m=1}^{M}$ , the fully connected layer projection heads of the PTM-specific task repr.  $\Phi_{T_i}^m$  (mentioned in subsection 4.4) and the transformer-based model-task similarity evaluator  $\mathrm{sim}(\cdot, \cdot)$ , which is the main mapping and estimation module (mentioned in subsection 4.2).
A.6 How to re-rank with PTM-specific task representation
As described in subsection 4.4 of the main text, we initially extract generic features using a fixed  $\psi$  and conduct with the invariant task repr. across all PTMs. These features are used to generate a coarse-grained ranking by comparing the similarity between each task and the model representation. However, this ranking is solely based on a standardized task representation and does not account for the specific task-related information for each individual PTM.
Hence, we propose the re-ranking strategy specifically targeted at the top-k PTMs. During the testing phase, we leverage the coarse-grained ranking and perform inference on the downstream task with these top-k PTMs. Such PTM-specific task repr. are worked to update their similarity with the downstream task, as outlined in Algorithm 2. Notably, in the third line of the algorithm, we conduct a re-ranking based on the revised similarity scores obtained through this process.
A.7 How to deploy MODEL SPIDER for testing
For a novel downstream task, we employ the generic feature extractor  $\psi$  to extract the task representation. We then evaluate the similarity between each PTM in the model zoo and the given downstream task using the learned model repr. and a transformer-based MODEL SPIDER. If computational resources are available, we can leverage the results from the previous round to enhance the ranking process. Specifically, we can select the top-k PTMs from the previous ranking, extract their features, and apply the re-ranking approach as described in subsection A.6.
B Experimental Setups and Implementation Details
In this section, we introduce the experiment setups and implementation details, including constructing the pre-trained model zoo and training as well as deploying MODEL SPIDER.
B.1 Single-source heterogeneous model zoo
Construction of the model zoo. We follow [113] and construct a model zoo with 10 PTMs pretrained on ImageNet [81] across 5 families of architectures available from PyTorch. Concretely, they are Inception V1 [88], Inception V3 [88], ResNet 50 [35], ResNet 101 [35], ResNet 152 [35], DenseNet 121 [38], DenseNet 169 [38], DenseNet 201 [38], MobileNet V2 [83], and NASNet-A Mobile [90]. The model zoo spans PTMs of multiple parameter quantities. These pre-training models cover most of the supervised pre-training models the researchers employ.
The downstream tasks. There are 9 downstream tasks from various fields, including Aircraft [59], Caltech101 [32], Cars [47], CIFAR10 [49], CIFAR100 [49], DTD [19], Pets [73], and SUN397 [107] for classification, UTKFace [118] and dSprites [61] for regression. We use official train-test splits on each dataset and calculate the estimation scores for the baseline approaches on the training part.
Transferred accuracy ranking of PTMs (ground-truth) after fine-tuning downstream tasks. We follow You et al. [113] to obtain the ground-truth transferability score as well as the rankings  $t = \{\mathrm{t}_{\phi_m \to T}\}_{m=1}^M$  ( $M = 10$ ) with careful grid-search of hyper-parameters. Specifically, we grid search the learning rates (7 learning rates from  $10^{-1}$  to  $10^{-4}$ , logarithmically spaced) and weight decays (7 weight decays from  $10^{-6}$  to  $10^{-3}$ , logarithmically spaced) to select the best hyper-parameter on the validation set and compute the accuracy on the downstream test set. The training and computation of such a ground truth necessitates a substantial investment of over 1K GPU hours, imposing significant financial and computational burdens. Consequently, the feasibility of accomplishing this task within the constraints of training MODEL SPIDER is rendered unattainable.
Sampling details of training tasks. We sample the training tasks from a diverse pool of datasets. The datasets considered for sampling include EuroSAT, OfficeHome, PACS, SmallNORB, STL10, and VLCS. To ensure a representative training set, we randomly sample 832 tasks from all datasets. Each task is distributed across 2 to 4 mixed datasets and consists of 100 categories, and for each category, we randomly select 50 examples. In cases where the number of categories or examples to be sampled exceeds the specified limits, we select the maximum allowable value.
**Discussions.** This model zoo covers several classical structures commonly used in deep learning. The number of model parameters ranges widely, with large application potential. Still, there is also a situation where PTMs with larger scales tend to perform better in classification tasks and regression ones, making certain rankings always better on some datasets.
B.2 Multi-source heterogeneous model zoo
Construction of the Model Zoo. As mentioned in the main text, we construct a large model zoo where 42 heterogeneous PTMs are pre-trained from multiple datasets in different domains, including animals [37, 46], general and 3D objects [32, 51, 49, 47, 14], plants [68], scene-based [107], remote sensing [106, 18, 36] and multi-domain recognition [54]. The concrete datasets are Caltech101 [32], Cars [47], CIFAR10 [49], CIFAR100 [49], SUN397 [107], Dogs [46], EuroSAT [36], Flowers [68], Food [14], NABirds [37], PACS [54], Resisc45 [18], SmallNORB [51] and SVHN [65]. The models' structures are 3 similar parameter-magnitude architectures, i.e., Inception V3 [88], ResNet 50 [35] and DenseNet 201 [38]. The setting of the multi-source heterogeneous model zoo includes significantly more pre-training data than the single-source heterogeneous one described above. We pre-train the models with 3 structures on 14 datasets mentioned above ( $3 \times 14 = 42$ , initialized from the weights of the corresponding ImageNet pre-trained models).
The downstream tasks. We select 3 representative datasets as the downstream test tasks and conduct the PTM selection methods on them. Concretely, they are Aircraft [59], DTD [19] and Pets [73]. As outlined in the following description, we obtain the transferred fine-tuning accuracy (ground-truth) with an equivalent level of hyper-parameters search strategies.
Transferred accuracy ranking (ground-truth). Similarly, we adopt downstream supervised learning with optimizing by cross-entropy loss. We meticulously conduct a grid-search of hyper-parameters, such as optimizers, learning rates, and weight decays (2 optimizers as SGD or Adam, 6 learning rates from  $5 \times 10^{-2}$  to  $10^{-4}$ , and 3 weight decay values from  $5 \times 10^{-4}$  to  $10^{-5}$ , batch size of 128, and the maximum epoch of 100). For the multi-domain dataset, like PACS [54], we set the test set to the same domain as the training set to reveal the in-domain performance. For the rest, we use the official train-test splits. We build the model zoo with around 5K GPU hours (on NVIDIA V100
Figure 6: Ablation studies on simpler  $\psi$  and less training tasks. We observed a slight decrease in performance when employing a weakened fixed feature extractor  $\psi$  for MODEL SPIDER. Reducing the diversity of training tasks may result in performance degradation on some datasets.
GPUs). Similarly, when dealing with the expanded model zoo, the utilization of rigorous training methodologies to acquire the requisite ground truth for training MODEL SPIDER is eschewed.
Sampling details of training tasks. The sampling process for the multi-source heterogeneous model zoo is consistent with the single-source one mentioned above. In this case, we use the following datasets as the auxiliary set, i.e., Caltech101, Cars, CIFAR10, CIFAR100, Dogs, EuroSAT, Flowers, Food, NABirds, PACS, Resisc45, SUN397, and SVHN. We randomly sample 4352 tasks for training.
Discussion. The availability of a multi-source heterogeneous model zoo introduces a wider array of models with varying structures, effectively covering a broader scope of domain knowledge. Consequently, this heightened diversity presents an increased difficulty in accurately ranking PTMs. Particularly, when a substantial gap exists between the characteristics of downstream tasks and the major PTMs, the ranking accuracy of some baseline methods undergoes a precipitous decline.
B.3 Large language models zoo
Construction of the model zoo. We considered a setting for ranking pre-trained models in natural language processing, wherein we utilized a library of 9 commonly used open-source Large Language Models (LLMs). These LLMs include Alpaca-7B [94], Baichuan-7B [109], Baichuan2-7B [109], ChatGLM2-6B [116], InternLM-7B [95], LLaMA2-7B [96], Vicuna-7B [119], Qwen-7B [8] and its chat fine-tuned version. These open-source LLMs, trained by academic institutions or companies on vast corpora, possess robust zero-shot capabilities. However, compared to the performance on general tasks, LLMs have a domain gap regarding some specific tasks. Some new benchmarks [56, 40, 23] have been proposed recently to show that while ChatGPT [69] performs well concerning the average performance of general tasks, it may not consistently outperform other models in certain tasks. Additionally, the deployment complexity and computational costs of LLMs can vary significantly. Blindly choosing LLMs with large model sizes or high running expenses may not achieve optimal accuracy but is more likely to waste resources. Therefore, the urgent challenge is accurately and efficiently selecting the most suitable LLM for a given task within the available budget and constraints.
The downstream tasks. We focus on unseen tasks, i.e., the examination datasets of AGIEval [120], as well as language datasets AFQMC [108], WSC [52], knowledge datasets BoolQ [20], NaturalQuestions [50], understanding datasets C3 [87], XSum [64], and reasoning datasets RTE [24], AX-b and AX-g [101] as the target tasks. When constructing tasks, whether for training or testing, we extract answers from 10 instruction data. During the generation of the final token in sequence generation, we extract features from the last layer. We calculate the average ranking score for 3 randomly sampled tasks from the target dataset as the final result.
Sampling details of training tasks. For training tasks, we deliberately choose different data from the test tasks, containing the remaining datasets in the OpenCompass [23] benchmark evaluation, such as GAOKAO-Bench [117], TyDiQA [21], CommonSenseQA [89], LAMBADA [72], COPA [79], and so on. We sample 10 instruction answers for each dataset, forming a task on that dataset. For each dataset, we sample a maximum of 16 training tasks, with mixed tasks from various datasets used in the training of MODEL SPIDER.
(a) on Aircraft
(b) on Caltech101
Figure 7: Correlation  $(\tau_w)$  given various number of examples per class on (a) Aircraft and (b) Caltech101. MODEL SPIDER shows stable and promising results in the low-shot scenario.
C.1 Ablation studies on simpler  $\psi$  and less training tasks
We deploy additional experience with weakened conditions to verify the robustness of MODEL SPIDER. In Figure 6, we first introduce an attenuated simpler  $\psi$ , the additional encoder except for the PTMs in the model zoo. We import the tiny format pre-trained Swin-Transformer from EsViT (about this, please refer to subsection A.1 for more details). It has about half the number of parameters. The results show that although attenuated  $\psi$  has only half of the parameters, it can still assist MODEL SPIDER in expressing task representation.
We then halve the training tasks to verify the significance of the training part diversity. We find that except for the performance degradation of the DTD dataset, the others are still flush with performance. MODEL SPIDER learns the characteristics of different PTM ability dimensions well despite the absence of training tasks.
C.2 Ablation studies on the influence of training loss
As stated in the main text, the learning process of MODEL SPIDER incorporates a ranking loss. To assess the efficacy of this selection, alternative regression or ranking loss functions, such as mean square error (MSE) and ListMLE [105], are employed as replacements. The outcomes, presented in Table 5, clearly demonstrate that the presented ranking loss function surpasses the other alternatives in terms of both effectiveness and robustness. Notably, when alternative loss functions are utilized, the overall performance of MODEL SPIDER
Table 5: The weighted  $\tau_w$  of MODEL SPIDER variants when the training objective is implemented by different loss functions. "Mean" denotes the averaged performance over 8 datasets.
Method	CIFAR10	Mean
w/ MSE	0.558	0.526
w/ ListMLE [105]	0.777	0.735
w/ ℓrank (Ours)	0.845	0.765
experiences a substantial decline. These findings underscore the indispensable role of the ranking loss function within the framework of MODEL SPIDER.
C.3 Ablation studies on the different shots of RankAgg and other baselines
We conduct an ablation analysis to compare RankAgg with several baseline methods on Aircraft and Caltech101 datasets with respect to the  $\tau_w$  of the PTM ranking. We examined the variation of these metrics and their corresponding confidence intervals (in  $95\%$ ) as the number of samples per class (shot) increased. The results, depicted in the provided Figure 7, are based on the average values and confidence intervals obtained from 30 randomly sampled sets for each shot. Due to computational constraints, certain baseline methods were omitted from the analysis. Notably, our findings reveal that the rank aggregation strategy effectively consolidates diverse perspectives on PTM ranking and consistently surpasses the performance of baselines across almost all shots.
Table 6: Ablation studies on the performance of MODEL SPIDER when the pre-trained model repository grows dynamically.
MODEL SPIDER	Aircraft	Caltech101	Cars	CIFAR10	CIFAR100	DTD	Pets	SUN397	Mean
When the number of PTMs increases
w/ number of 3	0.545	1.000	1.000	1.000	0.182	1.000	1.000	1.000	0.841
increase to 6	0.573	0.627	0.818	0.905	0.839	0.445	0.888	0.336	0.679
increase to 10	0.568	0.637	0.576	0.797	0.695	0.796	0.573	0.436	0.635
C.4 Ablation studies on the dynamically incremental model zoo
When encountering new PTMs during the model selection task, the previously trained model repr. in MODEL SPIDER can be dynamically learned and updated. We employ an incremental learning approach [77] to address this challenge. Specifically, we sample  $25\%$  target tasks where the PTM ranking is closest to the average of all and insert the approximated accuracy of the new PTMs on them. This newly constructed ranking ground-truths include the correlation between old and new model repr., reducing the influence of imbalanced incremental data.