SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fanqing Meng
OpenGVLab, Shanghai AI Laboratory
mengfanqing33@gmail.com
Zhanglin Peng
Kaipeng Zhang
Wenqi Shao†
shaowenqi@pjlab.org.cn
Chonghe Jiang
Yu Qiao
Ping Luo
pluo@cs.hku.edu
This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering. A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metrics to measure models' transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable in a multi-modal multi-task scenario. To tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models to transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into a unified noisy label embedding. EMMS can estimate a model's transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability of pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the state-of-the-art method LogME enhanced by our label embeddings, EMMS achieves  $9.0\%$ ,  $26.3\%$ ,  $20.1\%$ ,  $54.8\%$ ,  $12.2\%$  performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing  $5.13 \times$ ,  $6.29 \times$ ,  $3.59 \times$ ,  $6.19 \times$ , and  $5.66 \times$  speedup in wall-clock time, respectively. The code is available at https://github.com/OpenGVLab/Multitask-Model-Selector.
† Corresponding Authors: shaowenqi@pjlab.org.cn; pluo@cs.hku.edu
(a)
(b)
Figure 1: Comparison between prior pre-trained model selectors and our multi-task model selector. (a) denotes that a model selector measures transferability by modeling the compatibility between the model feature and task label. Previous model selectors can only receive labels with one-hot or real-valued vectors. Our multi-task model selector can be employed in various tasks with diverse label formats. (b) denotes that our proposed EMMS is applicable and effective in various downstream tasks while previous transferability metrics can be only used in classification or regression tasks.
Pre-trained models (such as neural network backbones) are crucial and are capable of being fine-tuned to solve many downstream tasks such as image classification [1], image captioning [2], question answering [3], and referring segmentation [4]. This "pre-training  $\rightarrow$  fine-tuning" paradigm shows that the models pre-trained on various datasets (e.g., ImageNet [1] and YFCC100M [5]) by many objectives (e.g., supervised and self-supervised) can provide generic-purpose representation, which is transferable to different tasks. A large number of pre-trained models have been produced with the rapid development of network architecture research, such as convolutional neural networks (CNNs) [6, 7, 8] and transformers [9, 10, 11]. When given a large collection of pre-trained models to solve multiple multi-modal tasks, an open question arises: how to efficiently predict these models' performance on multiple tasks without fine-tuning them?
Existing works [12, 13, 14, 15, 16, 17, 18, 19] answered the above question using model selection approaches, which are of great benefit to transfer learning. For example, when a neural network is properly initialized with a better pre-trained checkpoint, it will achieve faster convergence and better performance on a target task [20, 21]. However, it is challenging to quickly identify an optimal one from a large collection of pre-trained models when solving each multi-modal task. This is because of two reasons. Firstly, the ground truth of model ranking can only be obtained by brute-force fine-tuning and hyper-parameter grid search, which are computationally expensive [22]. Secondly, the recent methods [12, 13, 14, 23] that can estimate the transferability of pre-trained models are not generic enough for a variety of multi-modal tasks. For instance, an approach [14, 12] that relies on the prior knowledge of a single specific task would be ineffective in others.
To address the above challenges, we need a unified representation to represent diverse label formats in each multi-modal task e.g., categories, texts and bounding boxes. Existing methods cannot be employed in multi-task scenarios because they only receive labels with one-hot or real-valued vectors, as shown in Fig.1. For example, LEEP [12] and PACTran [24] are carefully designed for classification tasks. GBC [25] and TransRate [23] measure transferability using class separability, which relies on prior knowledge in classification task. Although LogME [13] can be used in both classification and regression tasks, it relies on real-valued labels, making it inapplicable in other label formats such as text descriptions.
In contrast to the above works, we propose an Efficient Multi-task Model Selector with an acronym, EMMS, which can select the most appropriate pre-trained model for solving each multi-modal task.
This is achieved by employing foundation models, such as CLIP [26] and GPT-2 [27], to transform diverse label formats into a unified label embedding space. The estimated label embedding contains more rich information than the conventional one-hot and real-valued label encoding.
In this way, EMMS can measure the compatibility between the models' features and corresponding label embeddings on various tasks, as shown in Fig. 1 and Fig. 2. This results in a more generic assessment of the models' transferability than previous methods. Specifically, EMMS treats the estimated label embeddings as noisy oracles of the ground-truth labels, and it turns a log-likelihood maximization problem into a simple weighted linear square regression (WLSR). We propose an alternating minimization algorithm to solve WLSR, which can be solved with a theoretical convergence guarantee efficiently. Extensive experiments validate the effectiveness of EMMS on multiple tasks, including image classification [1], image captioning [2], question answering on both image [28] and text [29], referring comprehension [4], and landmark detection [30].
The contributions of this work are summarized as follows. (1) We propose a generic transferability estimation technique, namely Efficient Multi-task Model Selector (EMMS). Equipped with a unified label embedding provided by foundation models and a simple weighted linear square regression (WLSR), EMMS can be fast, effective, and generic enough to assess the transferability of pre-trained models in various tasks. (2) We propose a novel alternating minimization algorithm to solve WLSR efficiently with theoretical analysis. (3) Extensive experiments on 5 downstream tasks with 24 datasets demonstrate the effectiveness of EMMS. Specifically, EMMS achieves  $9.0\%$ ,  $26.3\%$ ,  $20.1\%$ ,  $54.8\%$ ,  $12.2\%$ , performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing  $5.13\times$ ,  $6.29\times$ ,  $3.59\times$ ,  $6.19\times$ , and  $5.66\times$  speedup in wall-clock time compared with the state-of-the-art method LogME enhanced by our label embeddings, respectively.
Transferability Estimation. Model selection is an important task in transfer learning. To perform model selection efficiently, methods based on designing transferability metrics have been extensively investigated. LEEP [12] pioneers to evaluate the transferability of source models by empirically estimating the joint distribution of pseudo-source labels and the target labels. But it can only handle classification tasks with supervised pre-trained models because the modeling of LEEP relies on the classifier of source models. Recent works propose several improvements over LEEP to overcome the limitation. For example, NLEEP [31] replaces pseudo-source labels with clustering indexes. Moreover, LogME [13], TransRate [23], and PACTran [24] directly measure the compatibility between model features and task labels. Although fast, these metrics can only be used on limited tasks such as classification and regression. This work deals with model selection in multi-task scenarios. We propose EMMS to evaluate the transferability of pre-trained models on various tasks.
Label Embedding. Label embedding represents a feature vector of task labels, which can be generated in various ways. The classical approach is to use one-hot encoding to represent the labels as sparse vectors, which is widely used in image classification. Another way is to transform labels into vectors by embedding layers. For example, an RNN module is employed to generate label representation in [32], which is encouraged to be compatible with input data vectors in text classification tasks. In addition, it is also common to treat the labels as words and use techniques such as word2vec [33] or GloVe [34] to learn vector representations of the labels. The main obstacle in the multi-task scenario is how to deal with diverse label formats. In this work, we follow the idea of word embedding and treat task labels as texts, which are then transformed into embeddings by publicly available foundation models [26, 27].
Foundation Models. CLIP [26] is the first known foundation model which learns good semantic matching between image and text. The text encoder of CLIP can perform zero-shot label prediction because it encodes rich text concepts of various image objects. By tokenizing multi-modal inputs into homogeneous tokens, recent work on foundation models such as OFA [35] and Uni-Perceiver [36] use a single encoder to learn multi-modal representations. In this work, we utilize the great capacity of foundation models in representing image-text concepts to generate label embedding. It is noteworthy that although foundation models can achieve good performance in various downstream tasks, they may not achieve good zero-shot performance on many tasks[37, 38, 39] and it is still computationally expensive to transfer a large model to the target task [40, 41]. On the contrary, a multi-task model
selector can quickly select an optimal moderate-size pre-trained model that can generalize well in target tasks. In this sense, a multi-task model selector is complementary to foundation models.
3 Preliminary of Model Selection
Problem Setup. A target dataset with  $N$  labeled samples denoted as  $\mathcal{T} = \{(x^n, y^n)\}_{n=1}^N$  and  $M$  pre-trained models  $\{\phi_m = (\theta_m, h_m)\}_{m=1}^M$  are given. Each model  $\phi_m$  consists of a feature extractor  $\theta_m$  producing a  $D$ -dimension feature (i.e.  $\hat{x} = \theta_m(x) \in \mathbb{R}^D$ ) and a task head  $h_m$  outputting predicted label given input  $x$  [6, 9]. In multi-task scenarios, the ground-truth label comes in various forms, such as category, caption, and bounding box, as shown in 1. The task of pre-trained model selection is to generate a score for each pre-trained model thereby the best model can be identified to achieve good performance for various downstream tasks.
Ground Truth. The ground truth is obtained by fine-tuning all pre-trained models with hyperparameters sweep on the target training dataset and recording the highest scores of evaluation metrics [31, 13] (e.g. test accuracy and BLEU4 [42]). We denote in-tuning scores of different models as  $\{G_m\}_{m=1}^M$ . Since fine-tuning all models on all target tasks requires massive computation cost, research approaches design lightweight transferability metrics which offer an accurate estimate of how well a pre-trained model will transfer to the target tasks.
Transferability Metric. For each pre-trained model  $\phi_{m}$ , a transferability metric outputs a scalar score  $T_{m}$  based on the log-likelihood, as written by
$$
T _ {m} = \sum_ {n = 1} ^ {N} \log p \left(y _ {n} \mid x _ {n}; \theta_ {m}, h _ {m}\right) \tag {1}
$$
where  $(x_{n},y_{n})$  denotes the  $n$ -th data point in target dataset  $\mathcal{T}$ . A higher log-likelihood value for  $T_{m}$  indicates that the model  $\phi_{m}$  is likely to achieve better performance on the intended task. Numerous transferability metrics have been proposed by modeling prediction probability  $p(y_{n}|x_{n};\theta_{m},h_{m})$  in various ways. Although being efficient, they can hardly be used in multi-task scenarios.
Challenges in Multi-task Scenarios. Existing transferability metrics fail to generalize to various tasks for two reasons. Firstly, existing methods such as LEEP and LogME can only deal with real-value label formats. But  $y_{n}$  can be a sentence of words in the task of image caption. Secondly, a large number of the previous metrics estimate transferability through the target task's prior information such as maximizing inter-class separability, which is inapplicable in multi-task scenarios except for the classification. To overcome these difficulties, we introduce a simple regression framework with unified label embeddings provided by several foundation models in Sec.4.
In this section, we introduce our Efficient Multi-task Model Selector (EMMS). To overcome the difficulty of diverse label formats, EMMS employs foundation models to transform various labels into unified label embeddings in Sec.4.1. By treating label embeddings provided by multiple foundation models as noisy oracles of ground truth labels, EMMS can calculate transferability metric under a simple weighted linear square regression (WLSR) framework in Sec.4.2. We design an alternating minimization algorithm to solve WLSR efficiently in Sec. 4.3. The illustration of our EMMS is provided in Fig. 2.
4.1 Foundation Models Unify Label Embedding
In general, label embeddings or label representations should encode the semantic information such that two labels with low semantic similarity have a low chance to be grouped. A common scheme is to represent label embedding as a one-hot vector. However, one-hot representation can not embed labels with text formats such as captions in the image caption task. Following the design in multi-modality foundation models [42], we treat labels with diverse formats as a text sequence, which can be encoded by pre-trained foundation models, as shown in Fig. 2.
Label Embedding via Foundation Models (F-Label). Thanks to the great representational capacity, the foundation model can construct label embedding (termed F-label) while preserving its rich
Figure 2: Overview of our EMMS. (a) shows that labels in various tasks can be expressed by texts. (b) presents the graph model of regression with multiple noisy labels. We use several foundation models to encode text labels as label embeddings which are deemed as noisy oracles of true label embedding  $z$ . Moreover,  $z$  is a linear mapping of model feature  $\hat{x}$  with Gaussian noise  $\epsilon \sim N(0, \sigma_0^2)$ .
Figure 3: Label embedding has richer semantic information than one-hot labels. (a) indicates that in the classification task, F-Label can capture the correlation of labels with different granularity than one-hot encoding. (b) shows that in the image caption task, F-label can model the semantic relevance of two captions corresponding to the same image better than the one-hot label.
semantic information of labels. Given a label  $y$  in the target task, the label embedding  $z \in \mathbb{R}^L$  is obtained by  $z = F(y) / \|F(y)\|_2$  where  $F$  can be instantiated by various foundation models to process diverse label formats.  $\ell_2$  normalization is utilized to normalize the representations extracted from different foundation models. Moreover  $F$  can be implemented as CLIP [26], BERT [43] and GPT-2 [27] when task label  $y$  is text. Note that label embedding extraction can be fast enough with GPU parallel computation. We provide the runtime analysis in Appendix Sec.C.
Benefits of F-Label. F-Label has several advantages over one-hot label representations. Firstly, it embeds richer semantic information than one-hot label, leading to accurate modeling of the semantic relationships between different labels. As shown in Fig.3, F-Label leads to a higher correlation between fine-grained classes than one-hot encoding. Secondly, compared with one-hot labels, F-label can be obtained in a variety of tasks as long as the task label can be transformed into a text sequence. With the assistance of F-Labels, model selection can be established in multi-task scenarios.
4.2 Regression with Unified Noisy Label Embeddings
To estimate the transferability of pre-trained models, the relationship between model features  $\hat{x} \in \mathbb{R}^D$  and F-Label  $z \in \mathbb{R}^L$  should be modeled in order to calculate the transferability score  $T_m$  in Eqn. (1). On the other hand, since a semantic label  $y$  can be embedded by several foundation models, the label embedding set can be constructed as  $\mathcal{Z} = \{z_k = F_k(y) / \|F_k(y)\|_2, k \in [K]\}$  where  $\{F_k\}_{k=1}^K$  denotes  $K$  foundation models. Now, we utilize data points  $\{(\hat{x}_k^n, z_1^n, \dots, z_K^n)\}_{n=1}^N$  to model the relationship between model features and F-Labels.
Setup. As shown in Fig.2, we assume that true label embedding  $z$  is a linear mapping of the model feature with additive Gaussian noise with a variance of  $\sigma_0^2$ , as given by  $z = z_{0} + \epsilon = w^{T}\hat{x} +\epsilon$  and  $\epsilon \sim N(0,\sigma_0^2 I_L)$  where  $z_{0} = w^{T}\hat{x}$  is the regression prediction,  $w\in \mathbb{R}^{D\times L}$  and  $\epsilon$  are regression weights and regression error, respectively, and  $I_{L}$  is a L-by-L identity matrix.
We assume that F-labels  $\{z_k\}_{k=1}^K$  obtained from different foundation models are oracles that independently provide noisy estimates of the true label embedding  $z$ . Formally, we have  $P(z_k|z) = N(z, \sigma_k^2 I_L)$ . By the above setup, EMMS would be performed with noisy labels. Hence, EMMS tends to select pre-trained models robust to the label noise.
Reasonableness of the Linear Assumption. Specifically, EMMS assumes that the true label embedding  $z$  is a linear mapping of the model feature with Gaussian noise. The linear assumption is reasonable in image and text classification tasks because a linear classifier is usually used when the pre-trained model is transferred to a target task, which is commonly used in recent methods. For example, LogME [13] assumes that:  $z \gets N(w^{T} \hat{x}, \beta^{-1})$ , which implies that there is a linear mapping from the model feature space to the label space. PACTran [24] also has a similar setting. The difference is that LogME takes a one-hot label as the true label embedding, which limits its applicability. But our EMMS treat the true label embedding  $z$  as an implicit variable. And F-Labels  $\{z_k\}_{k=1}^K$  obtained from different foundation models are assumed to be noisy oracles of true label embedding  $z$ . Since labels in many tasks can be easily encoded into F-Labels, our EMMS can be used as a multitask model selector. We verify effectiveness the linear assumption in various multi-model tasks with extensive experiments in Sec.5.
Computation of Log-Likelihood. To model the relationship between model features and F-Labels, we need to estimate regression weights  $w$ , strengths of label noises  $\{\sigma_k\}_{k=0}^K$ . For simplicity of notation, we consider the case  $L = 1$ , i.e. F-labels are scalars. Given  $N$  data points, the log-likelihood is given by
$$
\mathcal {L} = N \log A _ {1} - \frac {N}{2} \log A _ {2} + \sum_ {n = 1} ^ {N} \left(\frac {\left(A _ {3} ^ {n}\right) ^ {2}}{4 A _ {2}} - A _ {4} ^ {n}\right) + \text {c o n s t} \tag {2}
$$
where  $A_{1} = \prod_{k = 0}^{K}1 / \sigma_{k}$ ,  $A_{2} = \sum_{k = 0}^{K}1 / 2\sigma_{k}^{2}$ ,  $A_{3}^{n} = \sum_{k = 0}^{K}z_{k}^{n} / \sigma_{k}^{2}$ , and  $A_4^n = \sum_{k = 0}^K (z_k^n)^2 /\sigma_k^2$ . The detailed derivation of Eqn.(2) is provided in the Appendix Sec.A.
Maximizing Log-likelihood as Weighted Linear Square Regression (WLSR). The remaining issue is to determine parameters  $w$  and  $\{\sigma_k\}_{k=0}^K$  by maximizing the log-likelihood in Eqn. (2). But it can be intractable because  $w$  and  $\{\sigma_k\}_{k=0}^K$  are heavily coupled. To mitigate this issue, we turn the log-likelihood maximization into a weighted linear square regression by rearranging Eqn. (2) as  $-\mathcal{L} = \frac{1}{2}\|Xw-Zt\|_2^2 + R(\{\sigma_k\}_{k=0}^K)$ , where  $X \in \mathbb{R}^{N\times D}$  is the data matrix whose  $n$ -th row is model feature  $(\hat{x}^n)^T$ ,  $w \in \mathbb{R}^{D\times 1}$  are weight parameters,  $Z \in \mathbb{R}^{N\times K}$  is F-Label matrix whose  $k$ -th column is the label embedding  $z_k$ , and  $t \in \mathbb{R}^{K\times 1}$  satisfies that  $1_K^T t = 1, t \geq 0$  which is a  $(K-1)$ -D simplex denoted as  $\triangle^{K-1} R(\cdot)$  is a regularization term parameterized with  $\{\sigma_k\}_{k=0}^K$ . We provide the derivations in Appendix Sec.A.
We note that the computational intractability comes from the data-dependent regularizer  $R(\cdot)$ . For efficient computation, we drop  $R(\cdot)$ , turning the log-likelihood maximization into a problem of WLSR, as given by
$$
\min  _ {w \in \mathbb {R} ^ {D \times 1}, t \in \triangle^ {K - 1}} s (w, t) = \frac {1}{2} \| X w - Z t \| _ {2} ^ {2} \tag {3}
$$
When considering the case  $L > 1$ , Eqn. (3) becomes  $\min_{w\in \mathbb{R}^{D\times L},t\in \triangle^{K - 1}}\frac{1}{2}\| Xw - Zt\| _F^2$  where  $Z\in \mathbb{R}^{N\times L\times K}$  and  $\| \cdot \| _F$  is Frobenius norm. From Eqn. (2) and Eqn. (3),  $s(w,t)$  is an approximation
Algorithm 1 Alternating Minimization
1: Input: Model feature  $X \in R^{N \times D}$ ; F-Label matrix  $Z \in R^{N \times K}$ ; Learning step-sizes  $\eta$  and  $\beta$  for  $w$  and  $t$ , respectively;
2: Output: Score of WLSR;
3: Initialize  $t = \frac{1}{K} 1_K$  and  $w = \frac{1}{D} 1_D$ ;
4: while  $s$  not converge do
5:  $s = \frac{1}{2}\| Xw - Zt\| _2^2;$
6:  $w\gets w - \eta X^T (Xw - Zt)$
7: while  $t$  not converge do
8:  $t\gets t - \beta Z^T (\bar{Z} t - Xw)$
9:  $t = \Pi_{\triangle K^{-1}}(t)$  // Projection
10: end while
11: end while
12: Return:  $s$
Algorithm 2 Fast Alternating Minimization
1: Input: Model feature  $X \in R^{N \times D}$ , F-Label matrix  $Z \in R^{N \times K}$ ;
2: Output: Score of WLSR;
3: Initialize  $t = \frac{1}{K} 1_K$  and  $w = \frac{1}{D} 1_D$ ;
4: while  $s$  not converge do
5:  $s = \frac{1}{2}\| Xw - Zt\| _2^2;$
6:  $w = (X^T X)^{-1} X^T Z t$ ; // LSR for  $w$
7:  $t = (Z^T Z)^{-1}Z^T Xw;$  // LSR for  $t$
8:  $t = \operatorname{Sparsemax}(t)$ ; // Projection
9: end while
10: Return:  $s$
of negative log-likelihood. Hence, a smaller  $s(w,t)$  indicate the larger  $T_{m}$  in Eqn. (1) and better transferability. We design an efficient algorithm to solve WLSR.
4.3 Fast Computation by Alternating Minimization
Algorithm. The optimization problem in Eqn. (3) can be formulated to a classical second-order conic program[42][44](simply called SOCP). However, the excessive data in our problem leads to a large dimension of the variable, making it inefficient for standard solvers. Therefore, we are motivated to find the smooth structure of the problem and design an alternating minimization algorithm to achieve fast computation. As shown in Algorithm 1, we separately fix  $w$  and  $t$  to optimize the other one until the function value in Eqn. (3) converges. Specifically, when we fix  $t$ , the whole problem degenerates to a least square problem with respect to  $w$ . When we fix  $w$ , we also need to solve a least square problem concerning  $t$  under the simplex constraint.