SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given  $\{\hat{t}_{\mathcal{T}_i}^1, \hat{t}_{\mathcal{T}_i}^2, \ldots\}$  as multiple predicted rankings over  $M$  PTMs for a sampled task  $\mathcal{T}_i$ , i.e., the order sorted by the estimations of transferability via various methods, we take advantage of Copeland's aggregation method [7, 82] to ensemble the orders:  $\bar{t}_{\mathcal{T}_i} = \{\bar{t}_{\phi_m \to \mathcal{T}_i}\}_{m=1}^M = \mathrm{RankAgg}(\{\hat{t}_{\mathcal{T}_i}^1, \hat{t}_{\mathcal{T}_i}^2, \ldots\})$ . Copeland's aggregation compares each pair of ranking candidates and considers all preferences to determine which of the two is more preferred. The output  $\bar{t}_{\mathcal{T}_i}$  acts as a good estimation of the ground-truth supervision  $t_{\mathcal{T}_i}$ . The aggregated  $\bar{t}_{\mathcal{T}_i}$  is more accurate than a particular transferability assessment method, which improves the quality of the supervision in ranking loss in Equation 4.
Sampling Tasks for Training. We assume that the training data  $\mathcal{D}$  contains a large number of classes with sufficient data. To sample tasks for training, we randomly select a set of classes from  $\mathcal{D}$  and choose a subset of their corresponding examples. Benefiting from the supervision estimation approach  $\mathrm{RankAgg}$ , we are able to obtain the aggregated ranking  $\bar{t}$  for any sampled task.
Training Complexity. The training phase in MODEL SPIDER is efficient. First, we pre-extract features  $\{\Phi_{\mathcal{D}}^{m}\}_{m = 1}^{M}$  for  $\mathcal{D}$  with all PTMs in advance. Then only the computational burden of base transferability assessment methods, rank aggregation methods, and the optimization of top-layer parameters are involved. Furthermore, training tasks with the same set of classes share the same  $\bar{t}_{\mathcal{T}_i}$ .
4.4 Re-ranking with Efficiency-Accuracy Trade-off
The learnable model representation captures the PTM's empirical performance on various fields of training tasks, which decouples the task repr. from the PTM. Each model repr. implicitly expresses the field in which the PTM excels, so the PTM selection only requires a task repr. to express the field in which the task is. In contrast to the general task repr.  $\mu(\mathcal{T}_i)$ , PTM-specific features  $\Phi_{\mathcal{T}_i}^m$  for a subset of PTMs provide rich clues about how those PTMs fit the target examples, which are also used in related transferability assessment approaches [25, 71]. We claim that given specific features with  $a$  subset of PTMs when the budget is available, our MODEL SPIDER can re-rank the estimated PTM order and further improve performance.
Specifically, we extract the PTM-specific task repr.  $\pmb{\mu}_{m}(\mathcal{T})\in \mathbb{R}^{d_{m}\times C_{\mathcal{T}}}$  with the specific features  $\Phi_T^m$  of the  $m$ th PTM as Equation 5. To take account of different values of  $d_{m}$  due to the heterogeneity of PTMs, we learn a projection  $\mathbf{P}\in \mathbb{R}^{d_m\times d}$  for the  $m$ th PTM to align the dimensionality of  $\pmb{\mu}_{m}(\mathcal{T})$  with the model representation. We then replace the general task repr.  $\pmb{\mu}(\mathcal{T})$  via the specific one  $\mathbf{P}_m^\top \pmb {\mu}_m(\mathcal{T})$  when calculating the similarity with the repr.  $\theta_{m}$  of the  $m$ th PTM. The specific task repr. may facilitate obtaining more accurate estimations. During the training process, we dynamically select a partial set of PTMs and incorporate the specific repr. into the sampled tasks. Thus, the same Transformer module in Equation 6 can deal with the new type of representation. To differentiate the general and specific representation, we learn two additional  $d$  -dimensional embeddings as prompts.
Table 2: Performance comparisons of 10 baseline approaches and MODEL SPIDER on a model zoo with 10 PTMs [113]. We measure the performance with Kendall's [45] weighted  $\tau_w$ . The downstream tasks from diverse fields (8 datasets) are evaluated in a standard manner (all training examples) and a few-shot manner (10 examples per class and 30 trials). Specific features of top-3 ranked PTMs are used in MODEL SPIDER. We denote the best-performing results in bold.
Method	Downstream Target Dataset
Aircraft	Caltech101	Cars	CIFAR10	CIFAR100	DTD	Pets	SUN397	Mean
Standard Evaluation
H-Score [9]	0.328	0.738	0.616	0.797	0.784	0.395	0.610	0.918	0.648
NCE [97]	0.501	0.752	0.771	0.694	0.617	0.403	0.696	0.892	0.666
LEEP [66]	0.244	0.014	0.704	0.601	0.620	-0.111	0.680	0.509	0.408
N-LEEP [55]	-0.725	0.599	0.622	0.768	0.776	0.074	0.787	0.730	0.454
LOGME [113]	0.540	0.666	0.677	0.802	0.798	0.429	0.628	0.870	0.676
PACTran [27]	0.031	0.200	0.665	0.717	0.620	-0.236	0.616	0.565	0.397
OTCE [93]	-0.241	-0.011	-0.157	0.569	0.573	-0.165	0.402	0.218	0.149
LFC [25]	0.279	-0.165	0.243	0.346	0.418	-0.722	0.215	-0.344	0.034
GBC [71]	-0.744	-0.055	-0.265	0.758	0.544	-0.102	0.163	0.457	0.095
MODEL SPIDER	0.506	0.761	0.785	0.909	1.000	0.695	0.788	0.954	0.800
Few-Shot Evaluation (10-example per class)
H-Score [9]	-0.014	0.078	0.375	0.018	0.005	-0.028	-0.006	0.853	0.160
NCE [97]	0.273	0.534	0.597	0.267	0.232	0.362	0.352	0.793	0.426
LEEP [66]	0.069	-0.038	0.476	0.530	0.471	-0.111	0.567	0.468	0.304
N-LEEP [55]	-0.559	0.476	0.743	0.515	0.707	0.027	0.713	0.812	0.429
LOGME [113]	0.341	0.453	0.497	0.718	0.698	0.407	0.657	0.817	0.574
PACTran [27]	0.136	0.262	0.484	0.631	0.614	-0.227	0.701	0.477	0.385
OTCE [93]	-0.316	-0.050	-0.127	0.515	0.505	-0.168	0.406	0.210	0.123
LFC [25]	0.226	-0.226	-0.235	0.330	0.271	-0.669	-0.059	-0.151	-0.064
MODEL SPIDER	0.382	0.711	0.727	0.870	0.977	0.686	0.717	0.933	0.750
The prompts are added to the input repr., allowing the transformer to utilize represented-type context for a better ranking process. Notably,  $\mu_{m}(\mathcal{T})$  depends on  $\Phi_{\mathcal{T}}^{m}$ , and the pre-extracted PTM-specific features for all training tasks make the construction of these specific representation efficient.
4.5 A Brief Summary of MODEL SPIDER
MODEL SPIDER learns to rank PTMs based on the model-task pair, balancing efficiency and accuracy. During the training, we sample tasks where PTM representation and transformer-based similarity are learned. In particular, to enable the model-task similarity to incorporate PTM-specific features, we replace some of the inputs to the transformer with enriched representations. We pre-extract PTM-specific features for all training tasks, and then the estimated ground-truth and the specific repr. could be constructed efficiently. During deployment, we first employ a coarse-grained PTM search with a general representation. Then we carry out forward passes over the target task only for top-k ranked PTMs, where the obtained PTM-specific task repr. will re-rank the PTMs by taking the distributed examples with PTM's features into account.
We evaluate MODEL SPIDER on three benchmarks: the PTM zoo comprising heterogeneous models from the single-source, multi-source datasets, or composed of large language models. We analyze the influence of key components in MODEL SPIDER and visualize the ability of a PTM using spider charts based on the learned representation.
5.1 Evaluation on a Single-Source Model Zoo
Table 1: Performance comparison of regression-conducted approaches with the same model zoo and weighted  $\tau_w$  measurement as in Table 2. The downstream task is dSprites and UTKFace.
Dataset	Methods for Regression Tasks
H-Score	LogME	GBC	Ours
dSprites	0.106	0.612	-0.283	0.679
UTKFace	0.075	-0.156	0.052	0.364
**Setup.** We follow [113] and construct a model zoo with 10 PTMs pre-trained on ImageNet [81] across five architecture families, i.e. Inception [88], ResNet [35], DenseNet [38], MobileNet [83],
Figure 3: Visualizations when selecting PTMs from a multi-source heterogeneous model zoo (w/ 42 PTMs) on three downstream datasets. Rows represent approaches, and columns represent datasets. Correlations  $(\tau_w)$  are shown above each subfigure. The horizontal axis denotes transferred accuracy (w/ fine-tuning), while the vertical axis is the output ranking score. The PTM architectures are drawn in red, yellow, and green. The bold line and the gray area show the fitted straight line and the confidence interval for all PTMs. The strong linear correlation suggests superior performance.
and MNASNet [90]. We evaluate various methods on 9 downstream datasets, i.e. Aircraft [59], Caltech101 [32], Cars [47], CIFAR10 [49], CIFAR100 [49], DTD [19], Pet [73], and SUN397 [107] for classification, UTKFace [118] and dSprites [61] for regression.
Baselines. There are three groups of comparison methods. First are creating a proxy between PTM-specific features and downstream labels, such as H-Score [9], NCE [97], LEEP [66],  $\mathcal{N}$ -LEEP [55], LogME [113], and PACTran [27]. The second are based on the downstream inter-categories features like OTCE [93], Label-Feature Correlation (LFC) [25], and GBC [71]. Following [66] and [113], we equivalently modify NCE and H-Score to the general model selection application.
**Evaluations.** For the standard evaluation, we follow the official train-test split of each downstream dataset and utilize all the training samples. In few-shot evaluation, we consider if MODEL SPIDER can select useful models with limited labeled examples under privacy and resource constraints. We sample 10 examples per class from the training set as a "probe set" and report the average results over 30 trials. The full results, along with  $95\%$  confidence intervals, are presented in the appendix.
Training Details of MODEL SPIDER. We implement the  $\psi$  with the pre-trained Swin-B [57, 53] to extract the task representation. MODEL SPIDER is trained on 832 sampled tasks from the mix of 6 datasets, i.e., EuroSAT [36], OfficeHome [99], PACS [54], SmallNORB [51], STL10 [22] and VLCS [31]. MODEL SPIDER utilizes specific features from the top-3 ranked PTMs (out of 10) for downstream tasks, resulting in a 3-4 times speedup.
Results of Standard and Few-Shot Evaluation. For the standard evaluation shown in Table 2 and Table 1, MODEL SPIDER outperforms other baselines across datasets, except for Aircraft, which ranks top-2. It also demonstrates superior stability and outperforms all the existing approaches in few-shot scenarios, as displayed in the lower part of Table 2. Consistently ranking and selecting the correct PTMs, MODEL SPIDER achieves the highest mean performance among all methods.
5.2 Evaluation on a Multi-Source Model Zoo
We construct a large model zoo where 42 heterogeneous PTMs are pre-trained from multiple datasets.
Setup. PTMs with 3 similar magnitude architectures, i.e., Inception V3, ResNet 50, and DenseNet 201, are pre-trained on 14 datasets, including animals [37, 46], general and 3D objects [32, 51, 49, 47, 14], plants [68], scene-based [107], remote sensing [106, 18, 36] and multi-domain recognition [54]. We evaluate the ability of PTM selection on Aircraft [59], DTD [19], and Pet [73] datasets.
Training Details. We use the same task representation extractor as in subsection 5.1 with 4352 training tasks sampled from the mix of the above datasets for pre-training the model zoo.
Analysis of Multi-Source Model Zoo. With many PTMs in the model zoo, we first set  $k = 0$  and select PTMs based on general representation. We visualize the results in Figure 3, with each
Table 3: Top-1 ranked Large Language Model (LLM) performance comparisons against LLM evaluation results [94, 116, 96, 119, 69], which includes 2 directly baselines and our MODEL SPIDER, ranking on a pre-trained model zoo of 9 LLMs. The 10 downstream tasks are construct based on the OpenCompass [23] benchmark from 5 diverse fields as examination, language, knowledge, understanding, reasoning. We denote the best-performing results in bold.
Method	Downstream Target Dataset	Mean
Exam.	Language	Knowledge	Understand.	Reason.
LLM Evaluations
Alpaca-7B [94]	24.30	67.20	41.95	33.30	51.70	43.69
ChatGLM2-6B [116]	39.00	67.30	44.35	40.25	68.67	51.91
LLaMA2-7B [96]	31.30	67.40	55.90	40.30	52.93	49.57
Vicuna-7B [119]	29.10	66.70	49.45	34.70	52.67	46.52
ChatGPT [69]	39.90	60.90	57.10	55.40	69.90	56.64
Top-1 Results of LLM Ranking Methods, Selected by
Self-assessed Confidence	34.60	67.40	45.10	37.45	62.60	49.43
Perf. on Similar Tasks	29.10	67.20	44.35	53.45	63.03	51.43
MODEL SPIDER	41.30	67.65	55.90	56.80	70.07	58.34
subfigure showing the transferred accuracy using the selected PTM with fine-tuning and the predicted ranking score. A better-performing method will show a more obvious linear correlation. The results demonstrate that MODEL SPIDER achieves the optimum in all three datasets. Furthermore, a visualization of efficiency, the averaged performance over all datasets, and model size on this benchmark with standard evaluation is shown in Figure 1. The different configurations of  $k$  balance the efficiency and performance in PTM selection, which "envelope" the results of other methods. These results confirm that MODEL SPIDER performs well in complex scenarios, highlighting its ability to select heterogeneous PTMs in a large model zoo.
5.3 Evaluation on a Zoo of Large Language Models
We introduce 9 open-source Large Language Models (LLMs) to construct our LLM zoo and deploy the MODEL SPIDER framework. We conduct a comparative analysis of the performance of the selected top-1 model against ChatGPT [69].
Setup. The LLM zoo involves Alpaca-7B [94], Baichuan-7B [109], Baichuan2-7B [109], ChatGLM2-6B [116], InternLM-7B [95], LLaMA2-7B [96], Vicuna-7B [119], Qwen-7B [8] and its chat fine-tuned version. We assess their zero-shot performance on diverse target tasks using the OpenCompass [23] LLM evaluation benchmark. We then focus on unseen tasks from the examination to language, knowledge, understanding, and reasoning datasets as the target tasks. For more details, please see appendix subsection B.3. We report the performance of the top-1 model recommended by each LLM ranking method and compare it with existing LLM evaluation results.
Training Details. For task representation, we employ a general Sentence-T5 [67] to obtain task representation. We extract answers from 10 instruction samples as a representative task for a dataset. We initialize the corresponding model repr. to encode the capabilities of LLMs on instruction data.
Analysis of Ranking on the Zoo of LLMs. Given that LLMs are computationally intensive in PTMs, we rank LLMs based on their general task representation. Intuitive methods for LLM ranking, like proxy measures relying on self-assessed confidence scores from generated answers or few-shot tasks in related domains, often fall short in assessing target task performance. The results indicate that while ChatGPT-3.5 demonstrates impressive performance in terms of universal performance across all diverse target tasks, as shown in Table 3 being 56.64, the top-1 ranked of MODEL SPIDER can surpass ChatGPT when efficiently choosing the appropriate LLM for each specific task. Our method achieves the average best and excels in the 4 out of 5 major fields of target tasks.
We analyze the properties of MODEL SPIDER on some downstream datasets, following the evaluation of a single-source model zoo in subsection 5.1.
Will RankAgg provide more accurate ground-truth during training? As discussed in subsection 4.3, MODEL SPIDER is trained on historical tasks and we utilize RankAgg to approximate
(a) Number of Used Features vs. Correlation.
Figure 4: (a): The ablation analysis of how the ranking correlation changes (Y-axis) with more PTM-specific features (X-axis). (b): Visualization of the PTM's ability on 6 major semantic clusters of datasets with spider chart. The score on the vertex of the spider chart is the averaged similarities between a PTM and the task representation in the cluster. The higher the vertex value, the better a PTM would perform on that kind of task.
(b) Spider chart of which semantic aspects the PTM excels in.
accuracy ranking. We investigate if this approximation offers better supervision and if using previous model selection methods like H-Score or LogME without aggregation is sufficient. The results in Table 4 include CIFAR10 and averaged results over eight classification datasets. It is evident that RankAgg provides stronger supervision during MODEL SPIDER's training.
Will more PTM-specific features help? As mentioned in subsection 4.4, MODEL SPIDER is able to incorporate PTM-specific features — the forward pass of a PTM over the downstream task – to improve the ranking scores. When no specific features ( $k = 0$ ) exist, we use the general representation to rank PTMs (most efficient). In Figure 4 (a), we show that  $\tau_w$  increases when MODEL SPIDER receives more PTM-specific features. It balances the efficiency and accuracy trade-off.
5.5 Interpreting MODEL SPIDER by Spider Chart
An interesting by-product of MODEL SPIDER is that we can visualize the ability of a PTM with a spider chart, which demonstrates which fields the PTM is good at. We cluster the datasets in our multi-source model zoo into six major groups. Then, we approximate a PTM's ability on the six types of tasks with the averaged similarity between a PTM to the tasks in the cluster. The larger the similarity, the better the PTM performs on that task. In Figure 4 (b), we find a PTM pre-trained on AID [106] dataset works well on medical and remote
sensing tasks, and a PTM pre-trained on NABirds [37] dataset shows strong ability on birds and animal recognition. The spider charts provide valuable insights into PTM capabilities and assist in PTM recommendations for specific application scenarios.
Table 4: The weighted  $\tau_w$  of MODEL SPI- DER variants when the training supervision is approximated by different methods. "Mean" denotes the averaged performance over 8 downstream datasets in Table 2.
Method	CIFAR10	Mean
w/ H-Score [9]	0.386	0.642
w/ LogME [113]	0.695	0.689
w/ RankAgg (Ours)	0.845	0.765
The proposed MODEL SPIDER learns to rank PTMs for existing tasks and can generalize the model selection ability to unseen tasks, even with few-shot examples, and is applicable to both visual and large language models (LLMs). The two-stage pipeline in MODEL SPIDER enables it to fit the resources adaptively. A task is matched with PTMs efficiently based on their task-agnostic representation if resource is limited. While there is a sufficient resource budget, limited forward passes are carried out over the candidates of top-ranked PTMs, which re-ranks candidates via incorporating the detailed fitness between the task and the selected PTMs. The learned representations help construct a spider chart for each task, illustrating its relevance with all PTMs. The representation for models and tasks acts as a kind of specification that matches the main design in Learnware [121, 122, 91, 34, 92].
Acknowledgments. This work is partially supported by the National Key R&D Program of China (2022ZD0114805), NSFC (62250069, 62376118, 62006112, 62206245), Young Elite Scientists Sponsorship Program of Jiangsu Association for Science and Technology 2021-020, Collaborative Innovation Center of Novel Software Technology and Industrialization.
[119] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[120] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023.
[121] Zhi-Hua Zhou. Learnware: on the future of machine learning. Frontiers Computer Science, 10(4), 2016.
[122] Zhi-Hua Zhou and Zhi-Hao Tan. Learnware: Small models do big. CoRR, abs/2210.03647, 2022.
- Appendix A: Workflow of MODEL SPIDER, encompassing the construction of model-task repr., training, and testing, with the "how to" and "answer" format.
- Appendix B: Experimental setups and implementation details of MODEL SPIDER, especially the two types of pre-training model zoos utilized in the experimental section.
- Appendix C: Additional experimental results conducted along different dimensions of robustness analysis.
- Appendix D: Additional datasets descriptions and other details mentioned in the main text.
- Appendix E: Discussions and future exploration of MODEL SPIDER.
A Details and Discussions of MODEL SPIDER
In the method section of the main text, we elucidate the comprehensive workflow for training and testing the deployment of MODEL SPIDER. This process encompasses three main steps, including (1) the extraction of task repr., (2) the extraction of model repr., and (3) the construction of a training scheme that assesses the ranking of matching between model-task repr., thereby establishing the ground-truth rank of the model zoo for a given task. Once these three steps have been accomplished, the subsequent phase entails training the MODEL SPIDER by leveraging the extracted repr. in conjunction with the ranked ground-truth information.
In essence, the testing and deployment strategy employed by the MODEL SPIDER framework epitomizes a balance between flexibility and efficiency. By employing a fixed feature extractor  $\psi$  to acquire repr. pertaining to downstream target tasks, the trained MODEL SPIDER undergoes a singular inference pass, generating an output quantifying the similarity between each model and the downstream task representation. It then accomplishes the task of ranking the PTMs.
In the forthcoming sections, we elaborate on the details in the form of "how to do it" questions. The training process of MODEL SPIDER is illustrated in Algorithm 1, while the sampling procedure for training tasks is elaborated in detail in subsection A.2. Additionally, in subsection A.6, we expound upon the training strategy of PTM-Specific task representation. Analogously, the testing process of MODEL SPIDER is presented in Algorithm 2, and in subsection A.7, we provide a comprehensive exposition of the entire deployment workflow for ranking pre-trained models.
A.1 How to construct model representations and task ones
This section supplements the details of subsection 4.2 and subsection 4.4, i.e., the construction of the model-task repr., including the enriched PTM-specific ones.
PTM representation. The dimension of PTM repr., i.e., the  $d$  of  $\pmb{\theta} \in \mathbb{R}^d$  is implemented as 1024. It is a learnable parameter that is optimized with the training process.
Task representation. The  $\psi$  is implemented by a pre-trained Swin-B-based EsViT [57, 53] (linked at https://github.com/microsoft/esvit), self-supervised learning on the ImageNet-1K [81] with batch size 512. In our experiments, this encoder acts as a wide-field feature extractor and is fixed without updating. The shape of task repr.  $\mu (\mathcal{T})\in \mathbb{R}^{d\times C_{\mathcal{T}}}$  varies with the number of categories of downstream tasks. As mentioned in subsection 4.4, task reprs. enriched by the PTM-specific features are obtained through the forward pass of a PTM. We use another fully connected layer to project the PTM-specific feature to align with the model representation.
A.2 How to sample the training tasks of MODEL SPIDER
We sample tasks for training MODEL SPIDER from additional datasets that are disjoint from the downstream tasks. These additional datasets possess notable differences and encompass diverse domains. Notably, MODEL SPIDER does not require substantial additional data for training. We sample the training tasks from a diverse pool of datasets. The number and size of the mixed datasets are controlled within a certain range. For more details, please see Appendix B.
Figure 5: An illustration of the rank aggregation approach to ensemble the ranking of PTMs relying on diverse transferability assessment methods (three methods depicted in the figure). The PTMs that outperform more other PTMs should be placed ahead.
A.3 How to see the relationship between RankAgg and MODEL SPIDER