SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fine-tuning Details. The ground truth of the problem of pre-trained model ranking is to fine-tune all pre-trained models with a hyper-parameters sweep on target datasets. Given the model and the target dataset, two of the most important parameters would be learning rate and weight decay in optimizing the model [90]. Therefore, we carefully fine-tune pre-trained models with a grid search of learning rate in  $\{1e - 1,1e - 2,1e - 3,1e - 4\}$  and weight decay in  $\{1e - 3,1e - 4,1e - 5,1e - 6,0\}$ . And using SGD optimizer. After determining the best hyper-parameters candidate, we fine-tune the pre-trained model on the target dataset with the candidate and then obtain the test accuracy as the ground truth. We use a Tesla V100 with a batch size of 128 to perform finetuning. All input images are resized to  $224\times 224$ . To avoid random error, we repeat the above fine-tuning procedure three times and take an average to obtain the final fine-tuning accuracy. For reference, we list the fine-tuning accuracy of supervised CNN models in Table.10, and vision transformer models in Table 11, respectively.
B.4.2 Image Captioning and Visual Question Answering
Fine-tuning Details. The setting of finetune here is approximately the same as in image classification. We carefully fine-tune pre-trained models with a grid search of learning rate in  $\{1e - 4,1e - 5,1e - 6\}$  and weight decay in  $\{1e - 4,1e - 5,1e - 6\}$ . And using AdamW optimizer. After determining the best hyper-parameters candidate, we fine-tune the pre-trained model on the target dataset with the candidate and then obtain the test BLEU-4 and accuracy as the ground truth. However, since Flickr10k-H and Flickr10k-R do not provide a test set, we use a 6:1 ratio to divide the original training set of 7000 images into a training set and a test set. For visual question answering, Due to the lack of a test set for CLEVR dataset, we also assign its training set as training set and test set in the ratio of 6:1. We use an Nvidia A100 with a batch size of 64 to perform finetuning. All input images are resized to  $224\times 224$ . To avoid random error, we repeat the above fine-tuning procedure three times and take an average to obtain the final fine-tuning accuracy. For inference, We use BLEU-4 as the score for the model with image captioning and accuracy as the score for the model with VQA. We list result of image captioning models in Table.12, and visual question answering models in Table 13, respectively.
Table 12: The fine-tuning BLEU-4 of image captioning models on 5 target tasks.
F8k	F30k	RSD	F10k-H	F10k-R
Vit-Bert	18.51	26.65	31.39	5.31	5.18
Vit-Roberta	20.53	23.70	29.92	5.88	5.48
Vit-Bart	21.90	25.13	31.35	5.75	5.53
Swinvit-Bert	22.91	26.61	33.54	6.24	5.67
Swinvit-Roberta	23.99	28.84	33.07	7.11	5.49
Swinvit-Bart	24.68	28.03	32.99	6.10	5.95
Swin2vit-Bert	25.69	31.33	35.45	5.86	5.49
Swin2vit-Roberta	23.40	28.81	36.22	6.80	7.13
Swin2vit-Bart	26.24	30.35	34.72	7.90	5.96
Table 13: The fine-tuning accuracy of visual question answering models on 3 target tasks.
DAQUAR	COCO-QA	CLEVR
Vit-Bert	25.01	55.11	59.29
Vit-Roberta	26.38	57.30	62.80
Vit-Bart	26.30	59.60	64.98
Swinvit-Bert	28.05	61.72	68.25
Swinvit-Roberta	27.75	62.81	66.09
Swinvit-Bart	27.06	60.62	67.17
Swin2vit-Bert	26.45	63.1	67.4
Swin2vit-Roberta	26.33	66.54	65.91
Swin2vit-Bart	26.25	64.4	70.34
B.4.3 Text Question Answering
Fine-tuning Details. The accuracy of most models in TQA is provided by DeBERTa [85, 86], except for DeBERTa-V3 [86](Base, Small, XSmall). Following the setting of Bert [43], we finetune these models with a batch size of 24 for 2 epochs. We use AdamW optimizer with an initial learning rate of  $3e - 5$ , polynomial decay. The Dev F1 score is used for pre-trained model ranking. All experiments are implemented on an NVIDIA Tesla A100 GPU. The finetune accuracy is shown in Table 14.
Table 14: The standard metric the Dev F1 score of text question answering models on 2 target tasks.
SQu1.1	SQu2.0
BERT-Large	90.9	81.8
RoBERTa-Large	94.6	89.4
XLNet-Large	95.1	90.6
DeBERTa-Large	95.5	90.7
DeBERTa-V2-XLarge	95.8	91.4
DeBERTa-V2-XXLarge	96.1	92.2
DeBERTa-V3-Base	93.9	88.4
DeBERTa-V3-Small	89.8	82.9
DeBERTa-V3-XSmall	91.5	84.8
Table 15: The standard metric Acc@0.5 of referring expression comprehension models on 3 target tasks.
RefCOCO	RefCOCO+	RefCOCOg
Blip	88.67	84.68	85.08
ALBEF	87.98	82.20	82.89
CLIP-ViT-B-32	83.20	74.56	76.98
CLIP-ViT-B-16	87.35	80.12	81.69
CLIP-ViT-L-14	90.17	86.09	87.13
CLIP-ViT-L-14-336	91.67	87.60	87.89
CLIP-RN50	84.69	76.72	79.39
OFA-Base	88.48	81.39	82.29
OFA-Large	90.05	85.80	85.89
OFA-Huge	92.04	87.86	88.07
B.4.4 Referring Expression Comprehension
Fine-tuning Details. For referring expression comprehension, the standard metric Acc@0.5 on the validation set is used as the ground truth. For finetuning, we use a batch size of 128 with a resolution of  $512 \times 512$  for each image. We finetune the models on each dataset for 12 epochs with a learning rate of  $\{3e - 5, 5e - 5\}$  and weight decay in  $\{1e - 3, 1e - 5\}$  using Adam optimizer. The best performance on the validation set for each task is reported among these hyper-parameters. Table 15 shows the performance of referring expression comprehension models.
Table 16: The fine-tuning MSE on test set of models used in regression on 2 target tasks.
CUB	Pets
ResNet-34	4.114e-4	4.245e-5
ResNet-50	3.521e-4	4.489e-5
ResNet-101	2.746e-4	3.224e-5
ResNet-152	2.539e-4	2.775e-5
DenseNet-121	5.354e-4	1.096e-4
DenseNet-169	4.787e-4	9.469e-5
DenseNet-201	4.651e-4	1.058e-4
MNet-A1	1.1475e-3	1.878e-4
MobileNetV2	6.253e-4	9.510e-5
Googlenet	7.192e-4	1.197e-4
InceptionV3	6.174e-4	9.633e-5
Table 17: EMMS under different measurements of transferability assessment. The results are obtained on Flickr8k and RSICD datasets with image captioning task and Aircraft and DTD datasets with image classification task with ViT-based models. EMMS outperforms LogME and other baselines under various measures.
Data	Method	Rel@1	Rel@3	r	rw	τ	τw	Data	Method	Rel@1	Rel@3	r	rw	τ	τw
2*F8k	LogME	0.928	1.0	0.735	0.799	0.537	0.483	2*RSD	LogME	0.957	1.0	0.727	0.708	0.518	0.501
EMMS	1.0	1.0	0.741	0.823	0.667	0.660	EMMS	1.0	1.0	0.783	0.765	0.611	0.705
3*Aircraft	LogME	0.852	0.993	0.407	0.060	0.378	0.299	3*DTD	LogME	0.992	1.0	0.641	0.694	0.556	0.569
TransRate	0.926	0.967	0.457	0.499	0.289	0.244	TransRate	0.992	1.0	0.607	0.676	0.422	0.533
EMMS	0.926	0.967	0.622	0.608	0.511	0.481	EMMS	0.992	1.0	0.704	0.785	0.644	0.621
B.4.5 Regression
Fine-tuning Details. For regression, mean square error (MSE) on the test data is the ground truth. For finetuning, we use a batch size of 64 with resolution of  $224 \times 224$  for each image. We carefully fine-tune pre-trained models with a grid search of learning rate in  $\{1e - 1, 1e - 2, 1e - 3, 1e - 4\}$  and weight decay in  $\{1e - 3, 1e - 4, 1e - 5, 1e - 6, 0\}$  with SGD optimizer. The fine-tuning MSE on test set of models used in regression is in Table 16
Table 18: The effect of Label Embedding in EMMS. Three variants of EMMS are considered: (1) EMMS with one-hot label; (2) EMMS with single F-Label; (3) EMMS with multiple F-Labels which is the original. We see that label embedding brings some performance improvement to EMMS.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC	Avg.
Weighted Kendall's tau τw
(1)	0.481	0.546	0.304	0.963	0.804	0.701	0.498	0.588	0.574	0.638	0.707	0.618
(2)	0.531	0.562	0.426	0.952	0.804	0.720	0.481	0.602	0.535	0.667	0.726	0.636
(3)	0.556	0.562	0.565	0.963	0.840	0.720	0.498	0.608	0.604	0.667	0.735	0.664
C More Ablation Analysis
The Efftiveness of EMMS under Various Measurements. In addition to weighted Kendall's tau, we employ various other measures to evaluate our EMMS. These include Kendall's tau  $(\tau)$ , Pearson's correlation  $(r)$ , weighted Pearson's correlation  $(r_w)$ , and top- $k$  relative accuracy, denoted as  $\mathrm{Rel}@\bar{k}$ , which represents the ratio between the best fine-tuning accuracy achieved on the downstream task using the top- $k$  ranked models and the best fine-tuning precision achieved with all models. We test the robustness of our transferability metrics to different measurements on the Flickr8k and RSICD datasets for image captioning tasks, as shown in Table 17. Our EMMS consistently outperforms the previous transferability metric, including LogME and TransRate. Under the aforementioned measurements, demonstrating the superiority of our EMMS.
The Effect of Label Embedding In some multimodal tasks or text tasks, including image captioning or text question answering. Label emebdding directly affects the applicability of existing model selection metric to these tasks. In addition, even in classification tasks, the use of F-Label can also bring improvements in results. Here we focus on the comparison between label embedding and direct one-hot vectors for image classification tasks in CNN-based models. As shown in Table 18, the use of F-Label can bring performance improvement compared to One-Hot vector, the average  $\tau_w$  increase from 0.618 to 0.636; furthermore, the use of multiple F-Label also brings some improvement compared to the average of single F-Label with  $\tau_w$  increasing from 0.636 to 0.664.
The Effect of Computational Speedup. Here we experimentally demonstrate the effect of our accelerated algorithm. As shown in Table 21, the algorithm is similar to the in-accelerated version in terms of results, but much shorter in terms of the wall-clock time.
Comparison with different number of iterations The number of iterations affects the EMMS time, here we conduct experiments on the VQA task for the effect of the number of iterations on the results. As shown in Table .19, we find that the number of iterations does not have a large impact on the performance of our method, and even a small number of iterations can guarantee the final result(e.g. the number of iterations is 1). We believe that firstly our method converges very fast. And secondly, for the ranking problem of model ranking, even if the convergence is not sufficient, the original order can still be maintained to a large extent in EMMS, thus ensuring the effect.
Table 19: The effect of the number of iterations  $r$  on VQA models in rank correlation  ${\tau }_{w}$  . We find that even a small number of iterations allows the method to maintain its effect.
DAQUAR	COCO	CLEVR	DAQUAR	COCO	CLEVR
Weighted	Kendall's tau τw	Wall-Clock Time (s)
r = 3	0.743	0.812	0.804	111.05	735.21	745.11
r = 2	0.712	0.812	0.804	78.01	536.45	573.22
r = 1	0.712	0.812	0.804	50.54	263.72	274.56
We have several observations. (1) Different downstream tasks prefer F-Labels obtained from different foundation models. No single foundation model is dominant in all target tasks. In particular, CLIP is not the best model for extracting F-Labels. (2) For image classification, both language and multimodal foundation models are competent for acquiring F-Labels. (3) Our EMMS can achieve the best results by combining F-Labels obtained from multiple foundation models.
Table 20: The effect of the single foundation model on EMMS. The results are obtained on image classification regarding  $\tau_w$ .
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC	Avg.	SOTA/All
Weighted Kendall's tau τw
Previous SOTA	0.299	0.412	0.693	0.741	0.736	0.621	0.655	0.580	0.707	0.619	0.651	0.610	4/11
(1) Gpt2	0.481	0.463	0.448	0.652	0.745	0.621	0.562	0.652	0.740	0.616	0.730	0.610	6/11
(2) Bert	0.481	0.444	0.458	0.718	0.745	0.621	0.562	0.592	0.740	0.616	0.730	0.609	5/11
(3) RoBerta	0.448	0.444	0.507	0.701	0.745	0.608	0.562	0.580	0.740	0.574	0.730	0.604	3/11
(4) CLIP	0.481	0.444	0.496	0.608	0.720	0.621	0.562	0.558	0.740	0.616	0.706	0.595	3/11
(5) FLAVA	0.481	0.444	0.508	0.741	0.745	0.621	0.562	0.652	0.740	0.574	0.706	0.615	5/11
(6) AltCLIP	0.481	0.444	0.437	0.741	0.745	0.621	0.562	0.580	0.740	0.595	0.730	0.607	6/11
EMMS	0.481	0.444	0.706	0.718	0.745	0.621	0.562	0.673	0.740	0.619	0.730	0.639	8/11
The Wall-clock Time of Label Embedding. For classification tasks, since the maximum number of categories is often only a few hundred, Label Embedding is very fast. Here we focus on documenting the time required for multimodal tasks, e.g. image captioning, text question answering, and referring expression comprehension, where label embedding is more time-consuming. For each task, we use 8 Nvidia A100 GPUs for label embedding, with a batch size of 512 for each GPU. The running time of label embedding for image captioning, text question answering, and referring expression comprehension is shown in Table 22. We measure the time for each dataset on the same CPU device (AMD EPYC 7H12 with 64-Core Processor) for three times and take the average as the final result.
Table 21: The effect of computational speedup in image classification with ViT models. We can see that the accelerated version of the algorithm achieves a significant reduction in time while guaranteeing results. Two variants of EMMS are considered: (1) EMMS with normal algorithm; (2) EMMS with fast algorithm.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC	Avg.
Weighted Kendall's tau τw
(1)	0.564	0.463	0.706	0.718	0.745	0.589	0.592	0.531	0.755	0.532	0.730	0.629
(2)	0.481	0.444	0.706	0.718	0.745	0.621	0.562	0.673	0.740	0.619	0.730	0.639
Wall-Clock Time (s)
(1)	102.06	114.72	177.25	718.34	724.5	50.24	87.28	944.57	83.37	336.92	104.9	313.10
(2)	21.31	17.23	28.06	154.61	182.11	13.87	15.95	265.99	17.93	63.86	16.63	72.55
Table 22: The wall-clock time (s) of label embedding in image captioning on 5 target tasks, text question answering on 2 target tasks, and referring expression comprehension on 3 target tasks, respectively.
Task	Image Captioning	Text QA	Referring EC
Dataset	F8k	F30k	RSD	F10k-H	F10k-R	SQuAD1.1	SQuAD2.0	RefCOCO	RefCOCO+	RefCOCOg
Time	14.56	89.31	18.92	3.37	3.13	35.67	53.87	49.19	48.88	31.63
The computational complexity of EMMS. We compare the computational complexity between LogME and EMMS in Table 23. We see that EMMS has lower computation complexity than  $\mathrm{LogME}(\mathrm{F})$  because  $\mathrm{LogME}(\mathrm{F})$  needs several iterations (T=3 on average) to converge. Moreover, EMMS allows for full vector computation and can be efficiently solved by existing scientific computation packages such as np.linalg.lstsq. Nevertheless,  $\mathrm{LogME}(\mathrm{F})$  cannot be written in fully vectorized form because the model parameters in  $\mathrm{LogME}(\mathrm{F})$  are highly coupled. Hence,  $\mathrm{LogME}(\mathrm{F})$  can only be executed in a while loop.
In addition, in the classification task, we compare EMMS and LogME. EMMS usually has higher computation complexity because  $D_{2} \gg C$ . In some cases, when the number of categories  $C$  and the iteration number  $T$  are large, EMMS could be faster than LogME with vector computation. For example, we find that  $C = 397$  and  $T = 4.46$  on average over all models when LogME is convergent on the Sun397 dataset. It results in higher time complexity than LogME, as indicated in Table 23. We further verify this by implementing LogME with  $T = 1$ . As shown in Table 24, EMMS spends more time in calculating the transferability than LogME (T=1) on all datasets. However, LogME performs much worse than EMMS because it does not converge when  $T = 1$ .
Table 23: The comparison of computational complexity between LogME, EMMS(one), and EMMS in image classification. We denote model feature  $X \in R^{N \times D_1}$  and F-labels  $Z \in R^{N \times D_2 \times K}$  with  $N \approx 10^4$ ,  $D_1 \approx 10^3$ ,  $D_2 = 1024$ ,  $K = 3$ , and  $C \approx 10^2$ . Moreover,  $T \approx 3$  denotes the iteration number of LogME. Moreover, LogME(F) denotes LogME with F-Label.
Complexity	Simplified Complexity	Vector Compuration
LogME	3TCD12+(2T+1)NCD1+D13+ND12	3TCD12+ND12+ND1C(2T+1)
LogME(F)	3TD2D12+(2T+1)NCD1+D13+ND12	3TD2D12+ND12+ND1D2(2T+1)
EMMS(One)	CD12+NCD1+D13+ND12	ND12
EMMS	ND12+2ND1D2+D13+D12+ (K2+K)(ND2)+K3+K2+K log K	ND12+2ND1D2
Table 24: The comparison between LogME and EMMS. The results are obtained on image classification regarding  $\tau_w$ . LogME ( $T = 1$ ) indicates that the inner loop of LogME only performs once.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC
Weighted Kendall's tau τw
LogME (T=1)	0.378	0.341	-0.408	0.645	0.727	0.112	-0.074	0.561	0.528	0.259	-0.04
LogME	0.299	0.382	0.633	0.741	0.727	0.569	0.512	0.580	0.528	0.619	0.591
EMMS(One)	0.412	0.444	0.565	0.740	0.736	0.621	0.562	0.579	0.740	0.592	0.730
EMMS	0.481	0.444	0.706	0.718	0.745	0.621	0.562	0.673	0.740	0.619	0.730
Wall-Clock Time (s)
LogME (T=1)	4.45	4.72	8.18	34.81	40.15	3.65	5.13	53.7	4.59	31.66	6.03
LogME	8.93	10.89	30.28	53.07	62.13	4.78	9.27	104.92	6.28	425.43	7.42
EMMS(One)	4.12	4.45	8.07	19.45	26.18	2.65	4.03	39.72	3.50	24.84	4.07
EMMS	21.31	17.23	28.06	154.61	182.11	13.87	15.95	265.99	19.73	63.86	16.63
Comparison with variants of existing methods. To further validate the efficacy of EMMS, we compare it with TransRate using F-Labels on image classification. To this end, we estimate the mutual information of the model feature and F-label following TransRate. Specifically, denote model feature  $X \in R^{N \times D_1}$ , and the F-Label  $Z_k \in R^{N \times D_2}$ , we estimate the mutual information of  $X$  and  $Z_k$  after the discretization operation for each dimension of  $D_2$  separately and then take average to obtain the final score.
Moreover, we implement two baselines based on TransRate. When  $K = 1$ , we instantiate the F-Label as the CLIP embedding. When  $K = 3$ , we instantiate the F-Labels as the embedding collection extracted from the CLIP, BERT, and GPT-2. In this case, the final score is averaged over three F-Labels. The results are shown in Table 25, where we can see that our EMMS consistently outperforms TransRate with F-Labels (both  $\mathrm{K} = 1$  and  $\mathrm{K} = 3$ ).
Table 25: The comparison between TransRate and EMMS. The results are obtained on image classification regarding  $\tau_w$ . TransRate ( $K$ ) indicates that the number of foundation models used.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC	Avg.
Weighted Kendall's tau τw
TransRate(K=1)	0.297	0.440	0.682	0.655	0.501	0.533	0.548	0.537	0.736	0.533	0.666	0.557
TransRate(K=3)	0.295	0.441	0.682	0.523	0.501	0.542	0.548	0.539	0.730	0.533	0.679	0.546
EMMS	0.481	0.444	0.706	0.718	0.745	0.621	0.562	0.673	0.740	0.619	0.730	0.639