SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{R}(\gamma_k) = \mathcal{L}_1 + \sum_{k=1}^K \gamma_k (z_k^n)^2 - (1 + \frac{\gamma_0}{\Gamma(\Gamma + \gamma_0)} (\sum_{k=1}^K \gamma_k z_k^n)^2$ . The computational intractability of Eqn.(16) comes from the regularization term  $\mathcal{R}(\gamma_k)$ . Note that the coefficient  $\frac{\Gamma \gamma_0}{\Gamma + \gamma_0} > 0$  and  $\sum_{k=1}^K \frac{\gamma_k}{\Gamma} = 1$ . By removing regularizer  $\mathcal{R}(\gamma_k)$  and positive scale parameter  $\frac{\Gamma \gamma_0}{\Gamma + \gamma_0}$ , the minimization of negative log-likelihood can be approximately treated as a weighted linear square regression, as given by
$$
\min  _ {w \in \mathbb {R} ^ {D \times 1}, t \in \triangle^ {K - 1}} s (w, t) = \frac {1}{2} \| X w - Z t \| _ {2} ^ {2} \tag {17}
$$
In Eqn.(17),  $X \in \mathbb{R}^{N \times D}$  is the data matrix whose  $n$ -th row is model feature  $(\hat{x}^n)^T$ ,  $w \in \mathbb{R}^{D \times 1}$  are weight parameters,  $Z \in \mathbb{R}^{N \times K}$  is F-Label matrix whose  $k$ -th column is the label embedding  $z_k$ , and  $t \in \mathbb{R}^{K \times 1}$  satisfies that  $1_K^T t = 1$ ,  $t \geq 0$  which is a  $(K - 1)$ -D simplex denoted as  $\triangle^{K - 1}$ .
A.2 Convergence Analysis and Proof Outline
We will prove the convergence property of the function value. Indeed, we demonstrate a stronger condition that the function value decreases after each round of iterations on  $w$  and  $t$ . From the monotone convergence theorem, the convergence can thus be derived. For other convergence properties of alternating minimization, readers can refer to the literature [72], which can be of independent interest.
In the proof, we exploit the smoothness of the function and design a projection gradient descent method with sufficient decrease for the constraint optimization problem. The sufficient decrease in the unconstrained problem is a direct corollary.
Definition 1. A function  $f(x): \mathbb{R}^d \to \mathbb{R}$  is said to be  $\beta$ -smooth with constant  $\beta$  if
$$
| \nabla f (x) - \nabla f (y) | \leq \beta \| x - y \|, \forall x, y \in \mathbb {R} ^ {d}.
$$
Lemma 1. Suppose  $X$  is the simplex constraint, and  $y \in \mathbb{R}^d$ ,  $\Pi$  denotes the projection operator. Then the inequality holds
$$
\left(\Pi_ {X} (y) - x\right) ^ {T} \left(\Pi_ {X} (y) - y\right) \leq 0.
$$
Proof. For the projection  $\Pi_X(y)$ , it is a convex optimization problem and can be formulated as
$$
\min  _ {x} f (x) = \| x - y \| _ {2} ^ {2},
$$
where  $x^T 1 = 1$  and  $x > 0$ . We denote  $x^{\star}$  as the optimal solution to the problem. For the convex optimization problem, it holds for all  $x \in \mathbb{R}^d$  that
$$
\nabla f (x ^ {\star}) ^ {T} (x ^ {\star} - x) \leq 0.
$$
Therefore we can derive
$$
2 (x ^ {\star} - y) ^ {T} (x ^ {\star} - x) \leq 0.
$$
The lemma is proved.
Lemma 2. Let  $f$  be a  $\beta$ -smooth function. For any  $x, y \in \mathrm{dom}(f)$
$$
\left| f (x) - f (y) - \nabla f (y) ^ {T} (x - y) \right| \leq \| x - y \| ^ {2}.
$$
$$
\begin{array}{l} \left| f (x) - f (y) - \nabla f (y) ^ {T} (x - y) \right| = \left| \int_ {0} ^ {1} \nabla f (y + t (x - y)) ^ {T} (x - y) d t - \nabla f (y) ^ {T} (x - y) \right| \\ \leq \int_ {0} ^ {1} \| \nabla f (y + t (x - y)) - \nabla f (y) \| \| x - y \| d t \\ \leq \int_ {0} ^ {1} \beta t \| x - y \| ^ {2} d t = \frac {\beta}{2} \| x - y \| ^ {2}. \\ \end{array}
$$
The last inequality holds because  $f$  is a  $\beta$ -smooth function.
Lemma 3. Suppose the function  $f$  is the  $\beta$ -smooth function, and  $X$  is the simplex constraint. For any  $x, y \in X$ , let  $x^{+} = \Pi_{X}(x - \frac{1}{\beta}\nabla f(x))$  and  $g_{X}(x) = \beta (x - x^{+})$ . Then the inequality holds
$$
f (x ^ {+}) - f (y) \leq g _ {X} (x) ^ {T} (x - y) - \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2}.
$$
Proof. Using Lemma 1, we have
$$
\left(x ^ {+} - \left(x - \frac {1}{\beta} \nabla f (x)\right)\right) ^ {T} \left(x ^ {+} - y\right) \leq 0.
$$
$$
\nabla f (x) ^ {T} (x ^ {+} - y) \leq g _ {X} (x) ^ {T} (x ^ {+} - y).
$$
By using Lemma 2 and the fact  $f(x^{+}) - f(y) = f(x^{+}) - f(x) + f(x) - f(y)$ , we have
$$
\begin{array}{l} f (x ^ {+}) - f (y) \leq \nabla f (x) ^ {T} (x ^ {+} - x) + \frac {\beta}{2} \| x ^ {+} - x \| ^ {2} + \nabla f (x) ^ {T} (x - y) \\ = \nabla f (x) ^ {T} \left(x ^ {+} - y\right) + \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2} \\ \leq g _ {X} (x) ^ {T} \left(x ^ {+} - y\right) + \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2} \\ = g _ {X} (x) ^ {T} \left(x ^ {+} - x + x - y\right) + \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2} \\ = g _ {X} (x) ^ {T} \left(x ^ {+} - x\right) + g _ {X} (x) ^ {T} (x - y) + \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2} \\ = g _ {X} (x) ^ {T} (x - y) - \frac {1}{\beta} \| g _ {X} (x) \| ^ {2} + \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2} \\ = g _ {X} (x) ^ {T} (x - y) - \frac {1}{2 \beta} \| g _ {X} (x) \| ^ {2}. \\ \end{array}
$$
Theorem 3. Suppose  $s(w,t) = \frac{1}{2}\| Xw - Zt\| _F^2$  where  $X\in \mathbb{R}^{N\times D}$ ,  $Z\in \mathbb{R}^{N\times K}$ ,  $w\in \mathbb{R}^{D\times 1}$  and  $t\in \triangle^{K - 1}$ , the inner loop of  $t$  in Algorithm lines 7 - 10 decreases after each iteration. Specifically, denote  $\beta = 1 / \| 2Z^T Z\|$  and  $t^+ = \Pi_{\triangle^K -1}(t - \beta \nabla s(w,t))$ . For any  $t\in \triangle^{K - 1}$ ,  $s(w,t^{+}) - s(w,t)\leq -\frac{1}{2\beta}\| t - t^{+}\|^{2}\leq 0$ .
Proof. Since we fix parameter  $w$  and consider the optimization problem of variable  $t$ , we denote  $s(t) = s(w, t)$ , which leads to  $\nabla s(t) = -2Z^T(Xw^\star - Zt)$ . For any  $t_1, t_2 \in \mathrm{dom}(s)$
$$
\left\| \nabla s \left(t _ {1}\right) - \nabla s \left(t _ {2}\right) \right\| = \left\| 2 Z ^ {T} Z t _ {1} - 2 Z ^ {T} Z t _ {2} \right\| \leq \left\| 2 Z ^ {T} Z \right\| \left\| t _ {1} - t _ {2} \right\|.
$$
According to the definition 1, it shows that the  $f(t)$  is  $\beta$ -smooth, where  $\beta = \|2Z^T Z\|$ . We denote  $t \in \triangle^{K-1}$  to be the initial point and  $t^+$  to be the result of one iteration of  $t$ , where  $t^+ = \Pi_{\triangle^{K-1}}(t - \frac{1}{\beta} \nabla f(t))$ . From Lemma 3, we can replace  $x^+, y$  and  $x$  with  $t^+, t$ , and  $t$ , respectively. In this way, the inequality holds
$$
0 \leq s (t ^ {+}) \leq s (t) - \frac {1}{2 \beta} \| \beta (t - t ^ {+}) \| ^ {2} \leq s (t)
$$
Therefore, according to Monotone convergence theorem, the function value of the iterative algorithm will converge.
Theorem 4. Suppose  $s(w, t) = \frac{1}{2} \| Xw - Zt\|_2^2$  where  $X \in \mathbb{R}^{N \times D}$ ,  $Z \in \mathbb{R}^{N \times K}$ ,  $w \in \mathbb{R}^{D \times 1}$  and  $t \in \triangle^{K-1}$ , the function value in Algorithm will be convergent. Specifically, denote  $w^\star, t^\star$  as the result after one iteration of  $w, t$  respectively, we have  $0 \leq s(w^\star, t^\star) \leq s(w^\star, t) \leq s(w, t)$ .
Proof. In the first step, we denote  $t \in \triangle^{K-1}$  is the initial point, then use gradient descent algorithm to calculate  $w^{\star}$ . Since the optimization problem for  $w$  is a convex optimization problem and use lemma 2, the decreasing property for the gradient part can be derived. That is, for each  $w \in \mathbb{R}^{D \times 1}$ , we have  $s(w^{\star}, t) \leq s(w, t)$ . In the second step, we fix  $w$  as  $w^{\star}$ , from Theorem 3, we have  $s(w^{\star}, t^{\star}) \leq s(w^{\star}, t)$ . Therefore, the value of  $s(w, t)$  satisfies:  $0 \leq s(w^{\star}, t^{\star}) \leq s(w^{\star}, t) \leq s(w, t)$  from Monotone convergence theorem,  $s(w, t)$  converges to the limiting point. As shown above, the overall convergence of our algorithm is guaranteed.
B Experiment
In this section, we present detailed descriptions of datasets in Sec. B.2, pre-trained models and baselines in Sec. B.3, and ground-truth scores in Sec. B.4 in various target tasks. More ablation studies can be found in Sec. C.
Foundation Models. On image classification, image captioning, referring expression comprehension, and visual question answering, we use foundation models CLIP [26], BERT [43] and GPT-2 [27]. On text question answering, we use foundation models GPT-2 [27], BART [73], and ELECTRA [74]. CLIP was trained on a large dataset of images and their corresponding captions, which can understand the relationship between images and text. BERT is a pre-trained language model that can understand and generate natural language. GPT-2 was trained on a large corpus of text and can be fine-tuned for specific tasks such as text completion and text summarization. Bart is a sequence-to-sequence model, which is both auto-regressive and bidirectional. Electra is a different type of language model that key idea is to pre-train a generator model to produce fake data and shows promising results in various NLP tasks.
Interpretation of weighted Kendall's tau. The Kendall's  $\tau$  represents the ratio of concordant pairs minus discordant pairs when enumerating all pairs of  $\{T_m\}_{m=1}^M$  and  $\{G_m\}_{m=1}^M$  as given by
$$
\tau = \frac {2}{M (M - 1)} \sum_ {1 \leq i <   j \leq M} \operatorname {s g n} \left(G _ {i} - G _ {j}\right) \operatorname {s g n} \left(T _ {i} - T _ {j}\right) \tag {18}
$$
where  $\operatorname{sgn}(x)$  returns  $-1$  if  $x < 0$  and 1 otherwise. In this work, a weighted version of Kendall's  $\tau$ , denoted as  $\tau_w$ , is employed to assess transferability metrics considering that a top-performing model is always preferred for target tasks in transfer learning. In principle, a larger  $\tau_w$  implies the transferability metric can rank pre-trained models better. And if a metric can rank top-performing models better,  $\tau_w$  would be also larger. We also use other measurements to assess the performance of transferability metrics in Table 17 of Sec. C.
Table 7: Comparison of different transferability metrics on CNN models regarding  $\tau_w$  and the wall-clock time where EMMS(One) denotes EMMS with the one-hot label. Our proposed EMMS achieves the best transfer-ability assessment over 11 target tasks and exhibits higher efficiency than NLEEP.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC	Avg.
Weighted Kendall's tau τw
LEEP	-0.234	0.605	0.367	0.824	0.677	0.486	-0.243	0.491	0.389	0.722	0.371	0.409
LogME	0.506	0.435	0.576	0.852	0.677	0.647	0.111	0.385	0.411	0.487	0.669	0.509
NLEEP	-0.41	0.614	0.265	0.818	0.805	0.796	0.122	0.214	0.753	0.925	0.687	0.611
TransRate	0.172	0.269	0.172	0.513	0.197	0.336	-0.176	-0.071	0.173	0.612	0.651	0.236
EMMS(One)	0.481	0.546	0.304	0.963	0.804	0.701	0.498	0.588	0.574	0.638	0.707	0.618
EMMS	0.556	0.562	0.565	0.963	0.840	0.720	0.498	0.608	0.604	0.667	0.735	0.664
Wall-Clock Time (s)
LEEP	5.1	4.9	8.3	22.3	23.8	3.5	3.8	37.1	3.9	21.1	4.8	10.4
LogME	30.36	31.24	56.26	90.34	188.3	15.16	22.27	334.53	17.55	180.01	20.05	289.64
NLEEP	253.8	488.7	973.8	1.1e4	1.7e4	146.0	294.0	2.0e4	580.8	8.6e3	678.8	5455.9
TransRate	147.90	163.41	300.29	65.25	193.64	75.48	166.24	195.92	60.53	430.33	18.72	165.24
EMMS(One)	17.43	20.53	35.22	70.01	78.24	12.75	18.04	116.23	15.04	70.98	18.42	42.99
EMMS	65.85	63.49	79.79	245.49	295.37	46.38	63.52	417.80	59.64	173.59	64.60	143.2
Table 8: Comparison of different transferability metrics on VQA models in rank correlation  ${\tau }_{w}$  with the ground truth and the wall-clock time. The LogME denotes using LogME with F-Label. Our proposed EMMS performs better than PACTran head over 3 target tasks with much less time.
DAQUAR	COCO-QA	CLEVR	DAQUAR	COCO-QA	CLEVR
Weighted Kendall's tau τw	Wall-Clock Time (s)
LogME	0.586	0.591	0.281	116.72	716.35	4665.06
PACTran(Dir)	0.671	0.296	0.347	633.16	1169.91	428.03
PACTran(Gam)	0.595	0.419	0.319	614.23	1061.72	428.49
PACTran(Gau)	0.478	0.378	0.415	637.39	1075.88	418.34
EMMS	0.712	0.812	0.804	50.54	263.72	274.56
B.1 More experimental results
B.1.1 Performance on Image Classification with CNN Models
Performance and wall-clock time comparison. We compare EMMS with previous LEEP, NLEEP, LogME, and TransRate. As shown in Table.7, our EMMS achieve the best average  $\tau_w$  on 11 target datasets and the best  $\tau_w$  on 6 target datasets. Compared to NLEEP, which is the most effective other than EMMS, we have almost 1/40 of the time of NLEEP.
B.1.2 Performance on Visual Question Answering
To further demonstrate the generality of EMMS in multi-model tasks, we show how EMMS can work for VQA. We follow previous practice ([24]) which treats VQA as a classification task (vocab-based VQA). That is, we construct a vocabulary based on the top answers in the training sets and classify them into some of those labels. The models to be selected and the architecture is the same as in the image captioning.
Performance and wall-clock time comparison. As shown in Table.8, EMMS is clearly ahead of PACTran in terms of results and time, proving that EMMS has the ability to handle multi-modal tasks very well. We can find that EMMS outperforms PACTran on all datasets. In particular, EMMS achieves  $93.8\%$  and  $93.7\%$  gain over PACTran on the COCO-QA and CLEVR datasets with rank correlation  $\tau_w$  while reducing time consumption by  $75.1\%$  and  $34.3\%$  respectively compared to Pactran. This indicates that EMMS performs well on both ordinary VQA datasets(DAQUAR, COCO-QA) as well as VQA datasets(CLEVR) that focus on inference capabilities.
B.1.3 Regression
In addition to image classification and a variety of multi-modal tasks, here we show that EMMS can also be used for regression tasks. The daatets for regression task we use is CUB200 [75] and IIIT
Pets [54]. The input is an image containing various birds and pets, respectively. We need to predict the coordinates of the bird's or pet's bounding box in the image and mean square error (MSE) on the test data is the ground-truth. The pre-trained models used are the same as the image classification task with CNN models and the only baseline is LogME. We extract F-Labels using Bert and RoBerta.
As shown in Table 9, EMMS significantly outperforms LogME, with  $29.5\%$  and  $13.9\%$  performance improvement on CUB and Pets respectively.
Table 9: Comparison of different transferability metrics on regression models in rank correlation  ${\tau }_{w}$  with ground truth.
CUB	Pets
Weighted Kendall's tau τw
LogME	0.464	0.437
EMMS	0.601	0.498
B.2 Descriptions of Datasets
B.2.1 Image Classification
For image classification, we adopt 11 classification benchmarks, including FGVC Aircraft [47], Caltech-101 [48], Stanford Cars [49], CIFAR-10 [50], CIFAR-100 [50], DTD [51], Oxford 102 Flowers [52], Food-101 [53], Oxford-IIIT Pets [54], SUN397 [55], and VOC2007 [56]. These datasets cover a broad range of classification tasks, which include scene, texture, and coarse/ fine-grained image classification, which are widely used in transfer learning. In particular, CF10 and VOC2007 are typical coarse-grained classification datasets, Aircraft, and Cars are typical fine-grained classification datasets, and CF100 contains both coarse- and fine-grained classifications.
B.2.2 Image Captioning
For image captioning, We use Flickr8k [57], Flickr30k [58], FlickrStyle10K-Humor [59], FlickrStyle10K-Romantic [59] and RSICD [60]. Among them, Flickr8k and Flickr30k have commonly used image captioning datasets for natural images and have no emotional color; RSICD is a commonly used image captioning dataset in remote sensing; Flickr10k-H and Flickr10k-R are also image captioning datasets for natural images, but their images are depicted with humorous and romantic emotional colors, respectively.
B.2.3 Visual Question Answering
For visual question answering, we apply COCOQA [61], DAQUAR [62] and CLEVR [63]. Among them, DAQUAR is an early VQA dataset on real images; CLEVR is a synthetic dataset, which is a visual scene composed of some simple geometric shapes, focusing on evaluating the inference ability of VQA models; the questions and answers of COCO-QA are generated by NLP algorithms, and the images are from the COCO dataset, which is also a commonly used VQA dataset.
B.2.4 Text Question Answering
For text question answering, we separately use SQuAD1.1 [64], SQuAD2.0 [65], which are collections of question-answer pairs derived from Wikipedia articles and are widely used in text question answer.
B.2.5 Referring Expression Comprehension
For referring expression comprehension, we separately use RefCOCO [66], RefCOCO+ [66] and RefCOCOg [67]. Specifically, RefCOCO includes instances where there is only one object of its kind in the image, while RefCOCO+ includes instances where multiple objects of the same kind exist in the image.
B.3 Pre-trained Models and Baselines
B.3.1 Image Classification
Pre-trained Models. For CNN-based models, We select 11 widely-used CNN models including ResNet-34 [6], ResNet-50 [6], ResNet-101 [6], ResNet-152 [6], DenseNet-121 [7], DenseNet-169 [7], DenseNet-201 [7], MNet-A1 [76], MobileNetV2 [77], GoogleNet [8], and InceptionV3 [78]. All these models are trained on ImageNet dataset [1], which are widely used within the field of migration learning. For ViT-based models, we collect 10 ViT models including ViT-T [9], ViT-S [9], ViT-B [9], DINO-S [79], MoCov3-S [80], PVTv2-B2 [11], PVT-T [11], PVT-S [11], PVT-M [11], and Swin-T [10], which are widely used in various vision tasks. Besides, we append EMMS with one-hot label, which degenerates to a linear regression whose label is the one-hot vector. We fine-tune these models on the 11 target datasets to obtain the ground truth.
Comparison Baselines. Here we use some of the latest methods as baselines, including LEEP [12], NLEEP [31], LogME [13], and TransRate [23], which have been experimented with model selection on image classification tasks.
B.3.2 Image Captioning
Pre-trained Models. We use a classic and effective image captioning model architecture, which contains an image encoder and a language encoder to extract the features of the image and the corresponding caption, then fuses the image feature and the text feature and input it to the classifier. We aim to choose the best combination of image encoder and language encoder. Besides, We finetune each model in COCO Caption [81] and use these as the pre-trained models.
Specifically, We separately use ViT-B [9], Swin-B [10], Swinv2-B [82] as image encoder and Bert [43], Roberta [68], Bart [73] as language encoder, and use VisionEncoderDecoderModel from HuggingFace as the model architecture. Following the setting in PACTran [24], We finetune the model in COCO Caption [81] and use these as the pre-trained models. Following common practice ( [83]), we treat image captioning as a vocab-based classification task. That is we use a vocabulary and classify the caption into the index of some words in the vocabulary. Afterward, training is done according to the classification task criteria.
Comparison Baselines. In this common setup, each caption is converted to a matrix  $Y \in R^{L \times N}$ , where  $L$  denotes the length of the caption after padding or truncation and  $N$  denotes the size of the vocabulary, and each row in the matrix is a one-hot vector. Since  $N$  is generally very large, Existing model selection metrics do not scale to this case due to the huge amount of time spent. The only baseline we use is to model the fused feature with F-label using LogME since only LogME can handle the regression task. Here we calculate the average  $\tau_w$  and time of it with  $K$  single F-label from  $K$  foundation models we use respectively.
B.3.3 Visual Question Answering
Pre-trained Models. The model architecture and the model selection settings are the same as in the image captioning, Following the setting in PACTran [24], here we use the model after finetune on VQA-v2 [83] as the pre-trained model waiting for selection and treat VQA as a vocab-based classification task.
Comparison Baselines. Here we calculate the average  $\tau_w$  and time of it with  $K$  single F-label from  $K$  foundation models we use respectively. And in addition to that, the three methods proposed in PACTran [24] are added here, which are the only methods currently applied to VQA tasks.
B.3.4 Text Question Answering
Pre-trained Models. The selected models include BERT-Large [43], RoBERTa-Large [68], XLNet-Large [84], DeBERTa [85] (XLarge), DeBERTa-V2 [85] (XLarge and XXLarge), DeBERTa-V3 [86] (Base, Small, XSmall). More specifically, we simultaneously input the question and passage into the aforementioned models, utilizing the distinctive symbol [SEP] to demarcate them. By stacking the predicted head onto each model, we could further fine-tune the model such that it can predict the start and end positions of the answer within the passage. This is achieved by using two binary classifiers, where one is dedicated to identifying the start position and the other to pinpointing the end.
Table 10: The fine-tuning accuracy of supervised CNN models on 11 target tasks.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC
ResNet-34	84.06	91.15	88.63	96.12	81.94	72.96	95.2	81.99	93.5	61.02	84.6
ResNet-50	84.64	91.98	89.09	96.28	82.8	74.72	96.26	84.45	93.88	63.54	85.8
ResNet-101	85.53	92.38	89.47	97.39	84.88	74.8	96.53	85.58	93.92	63.76	85.68
ResNet-152	86.29	93.1	89.88	97.53	85.66	76.44	96.86	86.28	94.42	64.82	86.32
DenseNet-121	84.66	91.5	89.34	96.45	82.75	74.18	97.02	84.99	93.07	63.26	85.28
DenseNet-169	84.19	92.51	89.02	96.77	84.26	74.72	97.32	85.84	93.62	64.1	85.77
DenseNet-201	85.38	93.14	89.44	97.02	84.88	76.04	97.1	86.71	94.03	64.57	85.67
MNet-A1	66.48	89.34	72.58	92.59	72.04	70.12	95.39	71.35	91.08	56.56	81.06
MobileNetV2	79.68	88.64	86.44	94.74	78.11	71.72	96.2	81.12	91.28	60.29	82.8
Googlenet	80.32	90.85	87.76	95.54	79.84	72.53	95.76	79.3	91.38	59.89	82.58
InceptionV3	80.15	92.75	87.74	96.18	81.49	72.85	95.73	81.76	92.14	59.98	83.84
Table 11: The fine-tuning accuracy of vision transformer models on 11 target tasks.
Aircraft	Caltech	Cars	CF-10	CF-100	DTD	Flowers	Food	Pets	SUN	VOC
ViT-T	71.26	89.39	82.09	96.52	81.58	71.86	95.5	81.96	91.44	58.4	83.1
ViT-S	73.12	92.7	86.72	97.69	86.62	75.08	96.79	86.26	94.02	64.76	86.62
ViT-B	78.39	93.47	89.26	98.56	89.96	77.66	97.98	88.96	94.61	68.62	87.88
PVTv2-B2	84.14	93.13	90.6	97.96	88.24	77.16	97.89	88.67	93.86	66.44	86.44
PVT-T	69.76	90.04	84.1	94.87	75.26	72.92	95.8	83.78	91.48	61.86	84.6
PVT-S	75.2	93.02	87.61	97.34	86.2	75.77	97.32	86.98	94.13	65.78	86.62
PVT-M	76.7	93.75	87.66	97.93	87.36	77.1	97.36	85.56	94.48	67.22	87.36
Swin-T	81.9	91.9	88.93	97.34	85.97	77.04	97.4	86.67	94.5	65.51	87.54
MoCov3-S	76.04	89.84	82.18	97.92	85.84	71.88	93.89	82.84	90.44	60.6	81.84
DINO-S	72.18	86.76	79.81	97.96	85.66	75.96	95.96	85.69	92.59	64.14	84.8
Comparison Baselines. Here we calculate the average  $\tau_w$  and time of it with F-labels from  $K$  foundation models respectively.
B.3.5 Referring Expression Comprehension
Pre-trained Models. The candidate multi-modal architectures considered for REC task incorporate Blip [87], ALBEF [88], CLIP [26] (ViT-B-32, ViT-B-16, ViT-L-14, ViT-L-14-336, RN50), OFA [89] (Base, Large, Huge). In practice, we respectively extract the visual and textual representations from each of these models and feed them into a multi-modal interaction module followed by a stacked detection head, and further fine-tune the model to generate the ground truth of model selection.
Comparison Baselines. Here we calculate the average  $\tau_w$  and time of LogME with  $K$  single F-label from  $K$  foundation models we use respectively.
B.4 Fine-tuning Score on Various Target Tasks
B.4.1 Image Classification