SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We performed ablation studies to investigate the behaviour of MODEL SPIDER as the pre-trained model zoo dynamically expanded. Our analysis focused on how can MODEL SPIDER could quickly adapt to newly added PTMs and integrate them into the ranking process. The results in Table 6 demonstrate that as the size of the model zoo increased from 3 to 6 and then to 10, MODEL SPIDER demonstrated the ability to incrementally learn the recommended ranking for the new additions to the model zoo. The incrementally learned ranking for the entire PTM zoo exhibited slightly lower accuracy than the results of direct training on all PTMs. Nonetheless, MODEL SPIDER consistently maintained an excellent level of performance.
C.5 Confidence intervals for few-shot setting in Table 1 of the main text
We include the confidence intervals (in  $95\%$ ) for the few-shot experiments in the respective section of Table 1 for the main text. These intervals were obtained through 30 repeated trials, providing a robust estimate of the performance variability in a few-shot manner.
C.6 Illustration of re-ranking with PTM-specific task representation
In subsection 4.4, we discuss the learnable model repr., which captures the empirical performance of a PTM across various training tasks. This training scheme serves to decouple the task repr. from the forward pass of each PTM. Compared to the task repr. guided solely by general features, the PTM-specific task repr. provides more informative clues. By constructing it with the forwarding pass of PTM, we can incorporate the source PTM's adaptation information for downstream tasks. Our approach allows for the re-ranking of estimated PTM rankings using PTM-specific task representation. Since more forward passes consume more resources, MODEL SPIDER further improves performance and provides a dynamic resource adaptation option with PTM-specific features.
Illustrated in Figure 8 is an example of model re-ranking in the context of a heterogeneous multi-source model zoo. The MODEL SPIDER, after extracting PTM-specific task repr., accomplished a more precise PTM ranking. We re-construct the PTM-specific task repr. on the Dogs dataset pre-trained. Our investigation focuses on the Aircraft downstream dataset, and intriguingly, we discover that PTMs trained on multi-scenario multi-target datasets possessed inherent advantages when applied to the aircraft domain. This advantage can be attributed to their generally strong recognition capabilities for diverse targets. Remarkably, even models pre-trained on the Food dataset demonstrated exceptional performance on the Aircraft dataset. Despite the notable dissimilarities between the Food and Aircraft datasets, we conjecture that the Food-pre-trained models not only exhibit proficiency in recognizing multiple targets, encompassing various food items but also harbor latent potential for fine-grained recognition within the food domain. Consequently, these PTMs transfer their fine-grained recognition capacity to the aircraft domain. In contrast, the Dogs dataset, characterized by a narrow focus on a single biological species, impedes successful transfer to the Aircraft task.
Table 7: The confidence interval (in  $95\%$  ) for few-shot evaluation (10 examples per class and 30 trials) in Table 1 of the main text. Specific features of Top-3 ranked PTMs are employed.
Method	Aircraft	Caltech101	Cars	Downstream Target Dataset
CIFAR10	CIFAR100	DTD	Pets	SUN397
Few-Shot Evaluation (10-example per class)
H-Score [9]	-0.014±0.14	0.078±0.13	0.375±0.09	0.018±0.12	0.005±0.14	-0.028±0.12	-0.006±0.15	0.853±0.02
NCE [97]	0.273±0.05	0.534±0.07	0.597±0.02	0.267±0.08	0.232±0.04	0.362±0.06	0.352±0.09	0.793±0.03
LEEP [66]	0.069±0.04	-0.038±0.01	0.476±0.03	0.530±0.04	0.471±0.02	-0.111±0.02	0.567±0.02	0.468±0.01
N-LEEP [55]	-0.559±0.06	0.476±0.05	0.743±0.04	0.515±0.06	0.707±0.03	0.027±0.07	0.713±0.04	0.812±0.02
LogME [113]	0.341±0.02	0.453±0.01	0.497±0.01	0.718±0.02	0.698±0.03	0.407±0.01	0.657±0.02	0.817±0.00
PACTran [27]	0.136±0.05	0.262±0.02	0.484±0.05	0.631±0.02	0.614±0.03	-0.227±0.03	0.701±0.03	0.477±0.03
OTCE [93]	-0.316±0.01	-0.050±0.00	-0.127±0.00	0.515±0.00	0.505±0.00	-0.168±0.01	0.406±0.00	0.210±0.00
LFC [25]	0.226±0.01	-0.226±0.01	-0.235±0.02	0.330±0.04	0.271±0.01	-0.669±0.03	-0.059±0.04	-0.151±0.02
Ours	0.382±0.04	0.711±0.00	0.727±0.01	0.870±0.01	0.977±0.02	0.686±0.02	0.717±0.02	0.933±0.03
Figure 8: Illustrative re-ranking example with enhanced ranking through PTM-specific task representation.
The substantial disparities between the datasets pose a significant challenge for conventional baseline methods, often failing to prioritize the Food-pre-trained model. However, MODEL SPIDER successfully learns to rank the Food-pre-trained one and, through a meticulous screening process followed by result re-ranking, MODEL SPIDER identifies that the Caltech101-pre-trained model outperforms the Dogs-pre-trained one due to its superior multi-target recognition capabilities, thereby exhibiting enhanced transfer performance.
D More Details
D.1 Comparison of the time consumption and memory footprint (details in Figure 1(c))
Figure 1(c) shows the average efficiency vs performance comparison over 5 baseline approaches and MODEL SPIDER. The  $k = 0$ ,  $k = 3$ ,  $k = 6$ ,  $k = 36$ , and  $k = 42$  correspond to inference w/o PTM-specific features, w/ 3, 6, 36, and 42 ones. Following [113], we measure the wall-clock time (second) and memory footprint (MB) with code instrumentation.
D.2 Datasets Description
We show the datasets description Table 9 with some examples Figure 9 covered in this paper.
Table 8: Comparison of the time consumption and memory footprint of fine-tuning, RankAgg, different baseline approaches, and MODEL SPIDER to rank the PTMs.
Approaches	Wall-clock Time (second)	Memory Footprint (MB)
RankAgg	7,318.06	10,405.32
Fine-tuning (all parameters)	614,497.22	13,872.81
H-Score	2,358.70	9,367.74
NCE	2,196.53	8,121.49
LEEP	2,215.06	8,209.33
N-LEEP	4,963.01	9,850.84
LogME	2,571.99	8,217.80
MODEL SPIDER (w/o PTM-Specific Feature)	52.36	608.01
MODEL SPIDER (w/ 3 PTM-Specific Feature)	105.19	1,386.43
MODEL SPIDER (w/ 6 PTM-Specific Feature)	175.87	1,760.28
MODEL SPIDER (w/ 36 PTM-Specific Feature)	2,180.23	7,989.35
MODEL SPIDER (w/ all (42) PTM-Specific Feature)	2,402.77	9,954.09
Table 9: The number of training images, testing images and classes with the link to download the dataset.
Dataset	Training Images	Testing Images	# Classes	URL
Aircraft [59]	6,667	3,333	100	https://www.robots.ox.ac.uk/~vgg/data/fgvc-aircraft/#aircraft
CIFAR10 [49]	50,000	10,000	10	https://www.cs.toronto.edu/~kriz/cifar.html
CIFAR100 [49]	50,000	10,000	100	https://www.cs.toronto.edu/~kriz/cifar.html
DTD [19]	3,760	1,880	47	https://www.robots.ox.ac.uk/~vgg/data/dtd/
Stanford Cars [47]	8,144	8,041	196	https://ai.stanford.edu/~jkrause/cars/car_dataset.html
Caltech101 [32]	3,060	6,084	101	http://www.vision.caltech.edu/Image_Datasets/Caltech101/
STL10 [22]	5,000	8,000	10	https://cs.stanford.edu/~acoates/stl10/
Oxford Flowers 102 [68]	2040	6149	102	https://www.robots.ox.ac.uk/~vgg/dataflowers/102/
CUB-200 [100]	5994	5793	200	http://www.vision.caltech.edu/visipedia/CUB-200-2011.html
Stanford Dogs [46]	12,000	8,580	120	http://vision.stanford.edu/aditya86/ImageNetDogs/
EuroSAT [36]	21,600	5,400	10	https://github.com/phelber/eurosat
SmallNORB [51]	24,300	24,300	5	https://cs.nyu.edu/~ylclab/data/norb-v1.0-small/
SVHN [65]	73,257	26,032	10	http://ufldl.stanford.edu/housenumbers/
Food-101 [14]	75,750	25,250	101	https://www.tensorflow.org/datasets/catalog/food101
NABirds [37]	23,929	24,633	555	https://dl.allaboutbirds.org/nabirds
NWPU-RESISC45 [18]	25,200	6,300	45	https://www.tensorflow.org/datasets/catalog/resisc45
Oxford-IIIT Pets [73]	3,680	3,669	37	https://www.robots.ox.ac.uk/~vgg/data/pets/
AID [106]	8,000	2,000	30	https://captain-whu.github.io/AID/
PACS [54]	5,446	616	7	https://domaingeneralization.github.io/#data
VLCS [31]	4,690	2,234	5	https://github.com/belaalb/G2DM#download-vlcs
Office-Home [99]	11,231	11,231	65	https://www.hemanthv.org/officeHomeDataset.html
SUN397 [107]	87,003	21,751	397	https://vision.princeton.edu/projects/2010/SUN/
ImageNet-1K [81]	1,281,167	50,000	1000	http://image-net.org/download
There are two promising directions of MODEL SPIDER. First, MODEL SPIDER exhibits the unique characteristic of not relying on the forward pass of the model zoo, thereby enabling the evaluation of task compatibility with classical machine learning models. Then, MODEL SPIDER could be applied to the case when we use other criteria in addition to fine-tuning performance to measure the fitness between a model and a task.
Caltech101
CIFAR100
STL10
CUB2011
EuroSAT
SmallNORB
NABirds
Resisc45
VLCS
SUN397
Figure 9: Examples of datasets.