How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey
Authors: Jun Bai
Xiaofeng Zhang
Chen Li
Hanhua Hong
Xi Xu
Chenghua Lin
Wenge Rong
Venue: EMNLP 2023 Findings (2023)
Keywords: Pre-trained Language Models
Transfer Learning
Transferability Estimation
Model Selection

Abstract: Transferability estimation has been attached to great attention in the computer vision fields. Researchers try to estimate with low computational cost the performance of a model when transferred from a source task to a given target task. 
Considering the effectiveness of such estimations, the communities of natural language processing also began to study similar problems for the selection of pre-trained language models. 
However, there is a lack of a comprehensive comparison between these estimation methods yet.
Also, the differences between vision and language scenarios make it doubtful whether previous conclusions can be established across fields. 
In this paper, we first conduct a thorough survey of existing transferability estimation methods being able to find the most suitable model, 
then we conduct a detailed empirical study for the surveyed methods based on the GLUE benchmark.
From qualitative and quantitative analyses, we demonstrate the strengths and weaknesses of existing methods and show that H-Score generally performs well with superiorities in effectiveness and efficiency.
We also outline the difficulties of consideration of training details, applicability to text generation, and consistency to certain metrics which shed light on future directions.

How to Determine the Most Powerful Pre-trained Language Model without
Brute Force Fine-tuning? An Empirical Survey
Jun Bai
Xiaofeng Zhang'
Chen Li
Hanhua Hong
Xi Xu²
Chenghua Lin Wenge Rong'
School of Computer Science and Engineering, Beihang University, China
2 Faculty of Information Technology, Beijing University of Technology. China
3 Department of Computer Science, University of Manchester, United Kingdom xiaofeng Z, chen.li, hanhua Soona, w.rong) @ wwwa..ddu.co xuxi @ bjut.edu.cn
chenghua. lin @ manchester.ac.uk
Abstract
Treen
Transferability estimation has been attached to great attention in the computer vision fields. Researchers try to estimate with low computa- tional cost the performance of model when transferred from a source task to a given tar- get task. Considering the effectiveness of such estimations, the communities of natural lan- guage processing also began to study similar problems for the selection of pre-trained lan- guage models. However, there is lack of comprehensive comparison between these es- timation methods yet. Also, the differences between vision and language scenarios make doubtful whether previous conclusions can be established across fields. In this paper, we first conduct a thorough survey of existing transfer- ability estimation methods being able find the most suitable model, then we conduct de- tailed empirical study for surveyed methods based on the GLUE benchmark. From qual- itative and quantitative analyses, we demon- strate the strengths and weaknesses of existing methods and show that H-Score generally per- forms well with superiorities in effectiveness and efficiency. We also outline the difficulties of consideration of training details, applicabil- ity to text generation, and consistency to certain metrics which shed light future directions.
Terget Featune
Model Similarity-based Methods
genual al
Target Labels
Target Dataset
Pre traireend
Features
0.26
Candidate PLMs
Training-free Methods
Figure I: Diagram of transferability estimation meth- ods. Based on the pre-trained features of target samples output from candidate PLM, model similarity-based methods and training-free methods estimate the transfer- ability by inter- anter-model similarity and the compatibility between pre-trained features and target labels.
Basically, the key to such model selection is to figure out the transferability between the model and the target task. Pioneering works conducted fine- tuning on every candidate model in brute-force manner (Phang et al., 2018; Zamir et al., 2019). Though the true fine-tuning performance can be obtained in this way, expensive parameter optimiza- tion is practically prohibitive (Wolf et al., 2020). Thus, there an urgent need to quantify the trans- ferability at a low cost of computation. To this end, Transferability Estimation (TE), as an essen- tial task of Transfer Learning (TL), has emerged as key challenge with several solutions proposed in Computer Vision (CV) fields initially (Agostinelli et al., 2022). Recently, some of these remarkable approuches have also been applied NLP tasks which show promising results on PLMM selection (Bassignana et al., 2022; Vu et al., 2022).
1 Introduction
Recent advances in the community of Natural Language Processing (NLP) are heavily built on the effectiveness Pre-trained Language Models (PLMs), especially on large ones (LLMs) (Zeng et al., 2023; OpenAI, 2022; Touvron et al., 2023; Wang et al., 2023). As the number of available PLMs continually grows, a critical question arises: "Which PLM can make the performance of a down- stream task best?". The fine-tuning result task usually varies across different PLMS, and this vari- ation becomes more pronounced low-resource scenarios (Bassignana et al., 2022).
Despite a great number of surveys established for TL and PLMs (Niu et al., 2020; Plested and Gedeon, 2022; Guo et al., 2022), there is no com- prehensive survey TE yet, especially with the purpose of PLM selection. Therefore, this paper aims to fill this gap by providing comprehensive and vell-structured summary of recent progress. To ensure comprehensive coverage, a multi-stage approach is employed to identify and select the

---

studies included in this review. Firstly, an exten- sive literature search was carried out using online databases, such as Google Scholar and DBLP. The search terms used were carefully chosen to cap- ture the key concepts and themes related to TE and PLMs. After retrieving an initial pool of nearly 100 articles, thorough screening of titles, abstracts, and keywords was conducted to exclude irrelevant studies, leading to a final selection of 20 studies that met the predetermined criteria for inclusion.
et al., 2019; Niu et al., 2020; Alyafeai et al., 2020; Iman et al., 2022), these works mainly focus on "what to transfer?" and "how to transfer?" that describe specific transfer approaches. To the best of our knowledge, there is no comprehensive survey on TE yet, which seeks to answer the question of "when to transfer?". This work is expected to fill this gap by shedding light on how to appropriately choose TE methods for PLMs practitioners.
Transferability Estimation. To avoid exhaus- tive attempts on all pairs of source tasks and tar- get tasks, TE provides efficient heuristics to ex- hibit the best-performing source task at a minor cost (Agostinelli et al., 2022). Originated in the field of CV, a great number of approaches, in- cluding model-similarity -itis-iiiitittbasseee methods (Dwivedi and Roig. 2019), label-comparison- based methods (Tran et al., 2019) and souree features- based meth- ods (Ding et al., 2022), etc., have been proposed in the past few years. To adapt such techniques to PLM selection for NLP tasks, Bassignana et al. (2022) found the predictions of LogME can posi- tively correlate with the true performances of can- didate PLMs, and Vu et al. (2022) exhibited the model similarity computed by soft prompts reflects the transfer performance across different models, Built on these remarkable researches, we further re- view the TE methods and provide a comprehensive empirical study of them for PLM selection.
Based on these research, we present a method taxonomy. As shown in Fig. 1, according to the need for training on target task, we divide TE meth- ods into: (1) Model Similarity-bssed Methods that assume the inter-model similarity reflects the trans- ferability which require the model trained target task (Dwivedi and Roig, 2019). (2) Training-free Methods that accelerate the estimation process by computing metrics free of target model training to examine the compatibility of the PLM's fea- ture space on the target dataset (Ding et al., 2022). Then we conduct qualitative analysis for the appli- cability and provide empirical results on the GLUE benchmark (Wang et al., 2019) to manifest specific strengths and weaknesses in existing methods. We show that model similarity-basee methods have the superiority of applicability to different target tasks, and training-free methods have the advantage over fast estimation. And for the methods simu- lating the dynamics of fine-tuning, they generally perform better. Besides, we analyze some factors that can affect the estimation effectiveness and ef- ficiency including task type, sample size, feature dimension, target model well sample affinity function. The empirical observations demonstrate that H-Score (Bao et al., 2019) generally shows desired usability. Based on these investigations, we further exhibit some under-explored aspects to shed light on the future directions!
Pre-trained Language Models. From BERT (De- vlin et al., 2019) to LLaMA (Touvron et al., 2023), significant efforts have been put into scaling PLMs into LLMs and some abilities such performing arithmetic, answering questions are emerging si- multaneously (Schaeffer et al., 2023). Neverthe- less, training and fine-tuning LLMs or even small ones require substantial computational resources which can limit accessibility to these models for researchers and developers with limited resources even with the help of parameter--fffiicen tuning (Hu et al., 2022). Based these considerations, the efficient utilization of PLMs is still a problem worth studying. Thus we focus on the selection of PLMs this work which aims at releasing the com- puting resources needed for exhaustive fine-tuning.
2 Related Work
Transfer Learning. Training robust supervised models from scratch non-trivial, especially in low-rcsource scenarios (Jin et al., 2023). Aiming at transferring knowledge from a source task to a target task, TL can achieve superior performances on the target dataset by spending far less time and using far fewer data (Niu et al., 2020). Despite good number of surveys available for TL (Ruder
3 Transferability Estimation Taxonomy
3.1 Problem Formalization
Formally, given a pool of candidate PLMs { and a target dataset D = {(1, vi)le, e X, yj E y)N containing N samples where each
'The code is available at https: thgisbb../uub....... http://thth..../aaaaaaaaaa oodel-selection

---

@i can encode the sample pre-trained feature (xzz (usually the [CLS] embedding), the true per- formance (D) can be measured by certain eval- uation metrics after fine-tuning Φ; on D with care- ful tuning of syyper-parameters The TE approach should produce a score (D) for each o; to ap- proximate the true fine-tuning performance (D). Intuitively, well-designed method should return (5 that correlates well with (T) under an acceptable burden, such that the top- performing PLM can be determined rapidly.
sample features as vertices, e.g., V4 {= and the inter-sample affinities as edges, c.g., &g {A(ds, k5)l ΦJ E Vo). For instance, Repre- sentation Similarity Analysis (RSA) (Dwivedi and Roig, 2019), Graph-Based Similarity (GBS) (Chen et al., 2021), Kernel Alignment (KA) (Huang et al., 2021) and Centered Kernel Alignment (CKA) (Ko- rbblthh et al., 2019) which are only slightly different in the ways to compute inter-sample affinities and inter-graph similarities.
3.3 Training-free Methods
3.2 Model Similarity-based Methods
Although model similarity- -bsedd methods only need to fine-tune on the target task once, they still require a large load of computational cost. There- fore, the training-free methods try to directly com- pare the pre- vre-trained features { with the true target labels { by cheap metrics to fur- ther save the estimation time. According whether directly measure the fine tineitunig loss, the metrics can be divided into class separability metrics and loss approximation metrics.
To avoid brute force fine-tuning, the model similarity- -bated methods are designed based on the assumption that a high similarity between two models correlates with a high degree of transfer- ability between the tasks bonded to the models. To this end, one model z fine-tuned on the target task, i.e., the target model, is required compute its sim- ilarity to each candidate PLM. Therefore, the time consumption of fine-tuning can be significantly re- duced to 1 I/L. of brute force approach extra with a minor cost of model similarity computation.
Class Separability Metries These metrics intu- itively examine whether pre-trained features are easy to separate according to their target labels, and assume that well-separated pre-trained fea- tures results in desired Some of these metries directly measure the sepa- rability of static pre-trained features. For exam- ple, MSC (Meiseles and Rokach, 2020) uses the mean intra-cluster distance and the mean nearest- cluster distance to quantify the clustering quality of pre-trained features over target classes. Simi- larly, Puigcerver et al. (2021) rank the candidate PLMs by the test accuracy of kNN on pre-trained features via the leave-one-out cross-validation. PARC (Bolya et al.. 2021) first computes the pair- wise affinities between the pre-trained features of each pair of target samples, which is then compared with the pairwise label affimities of each pair of tar- get samples to quantify the source feature space's 's fitness on target dataset. And GBC (Pándy et al., 2022) uses the Bhattacharyya coefficient to mea- sure the inter-class overlap of pre-trained features, where higher overlap means poorer separability. To further consider the fine-tuning dynamics by suming the pre-trained features can be adjusted by an extra linear transformation, Kumari al. (2022) train cheap Logistic Regression (LR) model on pre- pre-trained features to estimate how fitting the lin- early transformed pre-trained features are for their
Currently, the sample features output from mod- els are mainly used to measure the inter-model sim- ilarity. Therefore, the target is to design similarity function to maximize the correlation between fine- tuning performances and similarities between the pre-trained features (+(x))) E and target features { (x In terms of the similarity computation mechanism, existing functions can fall into sample- wise similarity functions and graph-wise similarity functions:
performance.
Sample-wise Similarity Functions The main idea is to directly compute the similarity between features across models. Under the Direct Simi- larity Estimation (DSE) (Luo et al., 2022) frame- work, Vu al. (2020) compute the affinity be- tween the mean features as the model similarity, i.e., A(E, ((x;)/N, E, (*:)/N) where the sample affinity function such as Euclidean and co- sine distances, while Luo et al. (2022) utilize aver- aged sample affinities E; A(b(z;), (xx,))/N.
Graph-wise Similarity Functions In the form of Duality Diagram Similarity (DDS) (Dwivedi al., 2020) framework, the graph-wise functions first measure the affinities for every sample pair in each model feature space separately, then com- pute the similarity between the resulting affinity graphs Gop = (Vo: Ex) and Gop = (Vo,, Ex) with the

---

Task
Dynamic Consideration
Free of Training
Methods
Input
Agnostic
Model Similarity-baset Methods
DSE (Vu et al., 2020)
d(x), (x)
X
X
DDS (Dwivedi et al., 2020)
<(x), (x)
X
x
Training-fie free Methods
MSC (Meiseles and Rokach, 2020)
6(2),3
X
X
KNN (Puigcerver et al., 2021)
6(x),1
X
X
PARC (Bolya al., 2021)
6(x),y
X
X
GBC (Pándy et al., 2022)
6(x),1
X
Logistic (Kumari et al., 2022)
6(x),y
X
H-score (Bao et al., 2019)
((r),V
x
X
Reg. H-score (lbrahim et al., 2022)
6(z),y
x
x
NLEEP (Li et al., 2021)
6(x),1
x
x
TransRate (Huang al., 2022)
6(x),
X
X
LogME (You et al., 2021)
6(x),y
X
SFDA (Shao et al., 2022)
6(x),
X
PACTran (Ding et al., 2022)
6(x),y
Table 1:
The comparison of the surveyed approaches, where Φ(z), (xx) and y denote the feature from candidate PLM. feature from target model and target label; v/ and x represent whether the method fulfill the property not.
target classes by LR's test accuracy.
classification over projected features and also adds a self-challenging module to further measure the ability of the pre-trained models on hard samples. To avoid over-fitting problem of maximum likeli- hood estimation, LogME (You et al., 2021) turns to approximate the marginalized likelihood of label given pre- pre-trained features over all possible linear transformation. More recently, motivated by learn- ing theory, PACTran (Ding et al., 2022) minimizes the PAC-Bayesian upper bound over the log-loss.
Loss Approximation Metrics Based on solid theoretical proof, these metrics try to directly ap- proximate the fine-tuning loss that correlates with the fine-tuning performance well. Inspired by Eu- clidean information geometry, H-Score (Bao et al.. 2019) approximates the optimal log-loss by inter- class variance and feature redundancy that char- acterize the asymptotic error probability of us- ing pre-trained features estimate target labels. Ibrahim et al. (2022) then propose regularized H- score which further shrinks the error that occurred when inverting the high- nigh-dimensional features using a pseudo-inverse. NLLEPP (Li et al., 2021) first uses Gaussian mixture model to attach a poste- rior distribution Gaussian components to each pre-trained feature, then computes the likelihood from posterior distribution target label to approx- imate that from pre-trained feature to target label. TransRate (Huang et al., 2022) approximates the correlation between pre-trained features and tar- get labels by Mutual Information (MI) which has been proven upper bound and lower bound to the log- ligeliiel There are also some metrics that involve fine-tuning dynamics. SFDA (Shao al., 2022) simulates the dynamics by projecting the pre-trained features using Fisher Discriminant Analysis (FDA) to increase the class separability. Then, approximates the log- likelihodood by Bayes
Qualitative Analysis
To examine the applicability of each method, we qualitatively compare them shown in Table from the following perspectives: (1) Task Agnostic: the method does not require certain target task type; (2) Dynamic Consideration: the fine-tuning dynam- ics pre-trained features are considered; (3) Free of Training: the method does not need fine-tuning on target task.
Task Agnostic A widely applicable method should be able to deal with multiple task types such as classification, regression, and generation. Currently, only model similarity-based methods satisfy this property. However, the inter-model sim- ilarity is only aware of sample features and does not consider the output space the task.

---

Dataset
ITriil IDevl
Task
Metric
et al., 2019) whose fine-tuning performances are shown in Appendix A.
Sentence Classification Tasks
CoLA
8.5k
lk
Acceptability
Mec.
Methods Setups Each candidate PLM be fine- tuned is followed by classification layer which takes the output embedding of BOS (Beginning of Sequence, c.g., [CLS] for BERT and 
 for RoBERTa) token as input, and all the TE methods utilize the same BOS embeddings as pre- pre-trained features. Before estimating by certain methods, also conduct dimensionality reduction on the sample features by Principal Component Analysis (PCA) (Roweis, 1997) since some methods' perfor- mances heavily depend on the number of feature dimensions. For model similarity-basee methods, we also try different PLMs to train the target model (i.c., ALBERT: albert-base-v2, DeBERTa: deberta- base and ELECTRA: bassera-base-eee (aaass-dscciminttrr)) Moreover, since some methods need to compute affinity graph which can be very time-consuming when there are too many samples, i.e., DDS, ANN, MSC, PARC, LFC, we limit the maximum number of samples that can be used by them to 10k. We run each method times with different random seeds and report the mean results. For the implementa- tion details, please refer to Appendix B.
SST-2
67k
872
Sentiment
Acc.
Paraphrase Tasks
MRPC
3.7k
408
Paraphrase
Acc.
QQP
364k
40.4k
Paraphrase
Acc.
Inference Tasks
MNLI
393k
9.8k
NLI
Acc.
QNLI
105k
5.5k
QA/LLI Acc.
RTE
2.5k
277
NLI
Acc.
WNLI
635
71
Coref./NLI
Acc.
Table 2: Statistic, task type and metric of GLUE tasks.
Dynamic Consideration Since fine-tuning ap- propriately adjust the representation of pre-trained features to adapt to the target task, the training dynamics are also a key factor. To date, only Lo- gistic, LogME, SFDA, and PACTran assume that the pre-trained features can be adjusted by linear transformation, while the fine-tuning process can be more diverse, e.g., adapter tuning (Houlsby et al.. 2019) and prompt tuning (Lester et al., 2021), how to simulate different dynamics under-explored.
Free of Training The trained target model is mainly needed by model similarity-busce meth- ods. Although this is not serious problem since the target model can be reused when estimating for different PLMs, it still takes a training time and whether model similarity- based methods perform stably on different target models is also doubtful.
To measure the deviation of TE meth- ods' predictions to true fine-tuning performances, we use MRR and mean Spearman's 's rank correla- tion (u₂) on all GLUE tasks. Among them, MRR reveals the average ranking of the best-performing PLM, and UP evaluates the overall correlation be- tween predicted score list (D) and true per- formance list {T)) Besides, the time con- sumption is measured by mean training time (Hu) that records the time of target model training and mean estimating time (Hex) that tells the wall clock time of estimation value computing. which are both averaged over all GLUE tasks. Note that we omit the time of sample features encoding since this is the same across all methods.
Evaluation
5 Experimental Setup
Datasets Following You et al. (2021), we vali- date TE methods on the GLUE benchmark (Wang et al., 2019) that is a collection of diverse Natural Language Understanding (NLU) tasks. Note, we remove the STS-B task since it is regression task that is not suitable to most TE methods, the details of others are reported in Table 2.
6 Quantitative Analysis
Since the HuggingFace Model Hub (Wolf et al., 2020) has already provided care- fully tuned results on GLUE tasks for some mod- cls, we select 6 PLMs that have all the GLUE tasks' results following You al. (2021): namely bert-base-uncasee bert-base-cased (Devlin et al.. 2019), roberta-base (Liu al., 2019) and their dis- tilled versions which are distilbert basttrbaee-uue uncasedd distilbert seserr-aae--ssses and distilroberta base (Sanh
Candidate PLMs
Effectiveness and Efficiency The overall metric scores of all methods are reported in Table 3. For estimation effeetiveness, although model similarity- based methods excel at adapting to different target tasks, they generally perform worse than training- free methods. Notably, DSE and PARC achieve the best MRR and μº respectively, while they need to carefully select sample affinity function to produce

---

MRR(()
1p(1)
Method
(a(l)
pot(+)
Sen.
Para.
Infer.
Overall
Sen.
Para.
Infer.
Overall
Model Similarit--based Methods
ALBETT
1.00
1.00
0.88
0.94
0.49
0.54
0.40
0.45
2.4h
10.1s
DEEEE
1.00
1.00
1.00
1.00
0.46
0.43
0.48
0,46
1.8h
7.2s
DEEELECTAA
1.00
1.00
1.00
1.00
0.40
0.43
0.49
0,45
1.2h
9.58
DDS ALBERT
0.20
0.50
0.58
0.47
-0.34
0.09
0.42
0.14
2.4h
131.7s
DDS DeBDRRA
0.42
0.50
0.56
0.51
0.20
0.40
0.31
0.32
1.8h
132.8s
DDSELECTRA
0.42
0.75
0.83
0.71
0.20
0.20
0.29
0.25
1.2h
133.5s
Training free Methods
MSC
0.58
0.33
0.55
0.50
0.03
0.03
0.42
0.22
4.58
ANN
0.33
0.67
0.79
0.65
0.26
0.40
0.49
0.41
21.0s
PARC
0.60
1.00
1.00
0.90
0.69
0.69
0.66
0.67
87.5s
GBC
0.33
0.67
0.38
0.44
-0.14
0.54
0.24
0.22
0.6s
Logistic
0.25
1.00
1.00
0.81
0.31
0.46
0.67
0.53
5.1s
H-Score
0.60
1.00
0.79
0.80
0.46
0.57
0.64
0.57
37.5s
Reg. H-Score
0.60
1.00
1.00
0.90
0.49
0.71
0.61
0.60
33.4s
NLEEP
0.33
0.67
1.00
0.75
-0.11
0.46
0.61
0.39
17.0s
TransRate
0.17
0.75
0.25
0.35
-0.69
0.57
0.12
0.03
0.7s
LogME
0.58
1.00
1.00
0.90
0.34
0.66
0.74
0.62
10.9s
SFDA
0.50
0.75
0.81
0.72
0.71
0.54
0.54
0.59
116.5s
PACTran 0.58
1.00
0.75
0.77
0.29
0.63
0.56
0.51
7.2s
Table 3: The performance of TE methods on GLUE benchmark, where MRR, mean Spearman coefficient μ6 each type of task (Sen. single sentence classification, Para. for paraphrase, and Infer. inference) and tasks (Overall) method efficiency where "_" means not applicable. For model similarity- based methods, the subscript indicates the type its target model, e.g., DSEALBERT means DSE implemented with ALBERT as target model.
reported. Moreover, mean training time jar and mean estimating time plat are listed to show the
such desired results. Another obvious observation is that Logistic, LogME, SFDA, and PACTran all result in remarkable performance, which empir- ically validates the importance of fine-tuning dy- namics. For estimation efficiency, model similarity- based methods all need considerable time consump- tion on target model training. Unless the target model training time negligible compared to the whole PLM selection time, this kind of method has no advantage over the training-free method in terms of speed. Generally, the training-free meth- ods pretty fast, only the methods need to com- pute affinity graph will consume lot of time (kNN, MSC, LFC, PARC), which have limit the amount of data samples when they are employed on huge datasets. For detailed results on each task, please refer to Appendix C.
sult in He of sentence pair tasks superior to that of single sentence tasks. We speculate the reason is that most candidate PLMs used in this work are from the BERT family and the corresponding Next Sentence Prediction (NSP) (Devlin al., 2019) pre- training task makes the [CLS] embedding more suitable for encoding a sentence pair, such that TE methods cannot fully understand the samples through the pre-trained features when meeting sen- tence classification task.
Robustness Fewer Samples The key strength of training-free methods lies no need to target model training, while the total time consumption can be further reduced if the method also performs well when using only small amount of data sam- ples, such that the encoding time of ignored sam- ples can be saved (Encoding for all GLUE datasets takes 1.73 hours in case). To examine the data efficiency, select top-performing training-free methods and conduct PL.M selection when different percentages of data are conditioned. Figure 2 illus- trates the overall performance variation on GLUE
Sensitivity to Task Type More detailed perfor- mances on three different types of tasks are also reported in Table 3. Generally, TE methods per- form better on sentence pair tasks (paraphrase and inference) than on single sentence tasks (sentence classification), where out of 14 TE methods re-

---

Euclidean
Cosine
Correlation
Method
MRR(t)
Pp(1)
Pet(f)
MRR(1)
PP(11)
(er(U)
(RR(1)
(o(()
Mer(()
DSE
1.00
0.46
7.2s
0.46
0.26
7.3s
0.44
0.20
9.2s
DDS
0.50
-0.10
129 4s
0.51
0.31
133. Is
0.51
0.32
135.9s
KNN
0.50
0.22
1.7s
0.59
0.34
8.4s
0.65
0.41
21.0s
LFC
0.41
-0.01
6.7s
0.45
0.09
8.6s
0.39
0.08
8.7s
PARC
0.69
0.35
83.6s
0.90
0.64
83.8s
0.90
0.67
87.5s
MSC
0.61
0.08
4.5s
0.50
0.22
4.6s
0.50
0.22
37.8s
Table 4: The performance comparison of methods on GLUE benchmark when different affinity functions (Euclidean distance, cosine distance, and correlation distance) are employed, where the MRR score, mean Spearman coefficient Up and mean estimating time Met reported.
KNN
Logistic
H-Score
SFDA
KNN
Logistic H-Score
H-Score LogME
SFDA
PARC
LogME
PACTan
PACTran
0.9
0.6
0.3
0.8
0.5
Mil
$
0.6
0.2
0.3-
0.5
10% 25%
5075
75%
1007
10%4
25%
50%
75%
100%
512 Feature Dimenston
Percentage
Percentage
Feature FeatureDimont Dimendion
Figure 2: The performance variation on GLUE tasks training- free methods when different percentages of tar- get data samples are used to conduct the transferability estimation, which the original class proportion will be kept when the sub-dataset is sampled.
Figure 3: The performance variation GLUE tasks of training-fiee free methods when pre-trained features' dimen- sions reduced different lengths PCA.
estimating time. As shown in Figure 3, list the performance variation of top-8 performing training- free methods on different feature dimensions. It observed that H-Score and regularized H-Score preferably enjoy original dimensions because the original feature space helps them to approximate the feature redundancy better, while the others all achieve the best results on smaller dimensions. For &NN, PARC, they need measure sample affinity which may encounter the curse of dimensional- ity high-dimensional scenes, thus performing better when the feature dimension is small mean- while dimensionality reduction will not lose too much original information. For Logistic, LogME, SFDA, and PACTran which assume the pre-trained features be linearly transformed, eliminating redundant feature dimensions results in better estimation results.
training-free methods as the data percentage changed. By employing shrinkage-based estima- tor of covariance, regularized H-Score exhibits sta- ble performance compared to the original H-score, And LogME also shows similar stability both MRR and 'P since it is based the marginalized likelihood which can alleviate the over-fitting prob- lem on a small dataset. We can observe that kNN also performs stable on Up while needs careful selections and sample affinity function. How- ever, a significant decrease in MRR occurred on almost all methods, indicating more advanced ap- proaches required to achieve accurate selection in low-resource scenarios.
Effect of Feature Dimensions If method per- forms best when conducted on the pre-trained fea- tures with the original dimension, then there is no need to employ dimensionality reduction and tune the reduced dimensions, which can further save the
Sensitivity to Different Target Models Al-
though model similarity-bart based methods only need

---

train one target model, we actually have rich choices of PLM train target models. An ideal method should produce similar results when differ- ent target models are implemented such that we can save the time required to try different target models. To examine the sensitivity to the target model, we conduct model amnilarity-based methods with dif- ferent target models and show the results in Table 3. We can observe different behaviors in which DSE performs stably while DDS is very sensitive to the type of target model. Since the main difference between DSE and DDS is that the former computes inter-sample affinities across models while the lat- ter compares the affinity graph across models, this observation reflects that the affinity graph from the target model may not well reflect the target task mechanism and directly measuring the affinity be- tween features across models is preferable.
(1) How to make the estimation approach aware of fine-funing strategies and experimental ppper-parameters? The fine-tuning strategy usu- ally needs to be determined under the acceptable computation burden, i.e. fully fine-tuning (optimiz- ing all model parameters) or purameter-efficient tuning (optimizing part of model parameters), The actual fine-tuning strategy not only affects the train- ing time and computation consumption but also the loss landscape of PLM which results in different target task performance (Bassignana et al., 2022). However, current approaches usually consider the situation of one strategy whose effectiveness can not be guaranteed in other situations. Therefore, making the estimation able to adapt to different fine-tuning strategies is worth further exploring. Besides, even if the best PLM can be accurately selected, one still needs exhaustive searching of training parammearsssters to produce desired fine- tuning performance. It is also interesting to con- sider other important hyper-parameter such as learning rate and temperature when estimating the transferability.
Effect of Sample Affinity Function As intro- duced in Section 3, the implementation of DSE, DDS, KNN, MSC, LFC, and PARC require certain sample affinity functions. We try Euclidean, cosine, and correlation distances for the above methods to examine whether different functions will affect the methods and report the results in Table 4. Com- pared to Euclidean distance, cosine distance and correlation distance conduct extra normalization operations and thus results in more estimating time. However, except when applied to DSE, cosine and correlation distances generally exhibit superior per- formance than Euclidean distance. This observa- tion reveals that the norm of the feature vector may result in anisotropic feature space and should not be taken into account to measure the sample affin- ity, which is also supported by (Su et al., 2021) that suggests the normalization operation can alleviate the anisotropy problem of PLMS.
(2) How to adapt TE methods to text genera- tion task? Although model similarity- -bsed meth- ods do not assume the type of target task since these methods only rely on the sample features, they neglect the information of the target label and thus the mapping from input space to output space is not well captured and the corresponding task can not be fully understood. However, taking label information into consideration for the text genera- tion task is challenging since the length of output text changes and the me-to-many issue exists (Bao et al., 2020; Zheng et al., 2021; Zhao et al., 2023). Since currently LLMs conduct all tasks in the way of text generation and the number of LLMs is con- tinually increasing, the TE method tailored for text generation is urgently needed.
7 Conclusion and Future Directions
(3) How to make estimation results consistent with specific evaluation metrics? In our experi- ments, the TE methods are asked to correlate just one evaluation metric for GLUE datasets, e.g., Acc for QNLI. However, some tasks may have diverse metrics, e-g., NDCG, R@ for ranking tasks, and sometimes one may focus on one of the metrics and the variations of these metrics are not necessarily consistent, such that the TE method's predictions can be confusing in these cases. Therefore, how to make TE methods aware of our interested metric is another direction worth exploring.
This paper reviews the recent advances in TE that can be applied to PLM selection and presents a method taxonomy based on a thorough analysis. Moreover, comprehensive qualitative and quantita- tive comparisons between different approaches are provided to help understand their applicability in a number of aspects. We hope this survey can help people for the purpose of research or industry to choose desired PLM by appropriate TE methods. Although lots of efforts have been made as sur- veyed, there still remain some directions that de- serve further investigation:

---

Limitations
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4171-4186
This work provides a comprehensive summary of existing TE methods. However, limited by our experimental conditions, we have to examine sur- veyed methods on a toy experimental setting where the following problems need to be improved: (1)
Nan Ding, Xi Chen, Tomer Levinboim, Soravit Chang- pinyo. and Radu Soricut. 2022. Pactran: Pac- bayesian metries for estimating the transferability of pretrained models classification tasks. In Proceed- ings of the 17th European Conference on Computer Vision, pages 252 252-268.
Only small-scale PLMs form the candidate pool, the effectiveness of TE methods to select the best- performing LLM is needed to be verified given the current popularity of LLMs. (2) Since most of existing TE methods only support target task of classification type, we determine the GLUE bench- mark as the evaluation datasets, while the TE per- formances regression task, structure prediction task and generation task are still under-explored.
Kshitij Dwivedi, Jiahui Huang, Radoslaw Martin Cichy, and Gemma Roig. 2020. Duality diagram similarity: A generic framework for initialization selection in task transfer learning. In Proceedings of the 16th European Conference on Computer Vision, pages 497-513.
Kshitij Dwivedi and Gemma Roig. 2019. Representa- tion similarity analysis for efficient task taxonomy & transfer learning. In Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recog- nition, pages 12387-12396.
References
Andrea Agostinelli, Michal Pándy, Jasper R. R. Uillings, Thomas Mensink, and Vittorio Ferrari. 2022. How stable are transferability metrics evaluations? In Proceedings of the 17th European Conference on Computer Vision, pages 403-321.
Shangwei Guo, Chunlong Xie, Jiwei Li, Lingjuan Lyu, and Tianwei Zhang. 2022. Threats to pre-trained language models: Survey and taxonomy. CoRR, abs/2202.06862.
Zaid Alyafeai, Maged Saced AlShaibani, and Irfan Ah- mad. 2020. A survey on transfer learning in natural language processing. CoRR, abs/2007. .00299..
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficien transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790-2799.
Siqi Bao, Huang Hc, Fan Wang. Hua Wu, and Haifeng Wang. 2020. PLATO: Pre-trained dialogue genera- tion model with discrete latent variable. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 85 96
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In Proceedings fo the 10th International Conference on Learning Representa- tions.
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas J. Guibas. 2019. An information-theonee ability task transfer learning. Proceedings of the 2019 IEEE International Conference on Image Processing, pages
approach to transfer-
Jiaji Huang, Qiang Qiu, and Kenneth Church. 2021. Ex- ploiting zoo of checkpoints for unseen tasks. In Proceedings of the 2021 Annual Conference on Neu- ral Information Processing Systems, pages 19423- 19434.
Elisa Bassignana, Max and Barbara Plank. 2022. Evidence > intuition: Transferability estimation for encoder selection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4218-4227.
Mike Zhang,
Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. 2022. Frustratingly easy trans- ferability estimation. In Proceedings of the 39th In- ternational Conference on Machine Learning, pages 9201 -225.
Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. 2021. Scalable diverse model selection for accessible trans- fer learning. In Proceedings of the 2021 Annual Con- ference on Neural Information Processing Systems, pages 19301-19312.
Shibal Ibrahim, Natalia Ponomareva, and Rahul Mazumder. 2022. Newer is not always better: Re- thinking transferability metrics, their peculiarities. stability and performance. In Proceedings of the 2022 European Conference on Machine Learning and Knowledge Discovery in Databases, pages 693- 709.
Zuohui Chen, Yao Lu, Wen Yang, Qi Xuan, and Xiao- niu Yang. 2021. Graph-based similarity of neural network representations. CoRR, abs/2111.11166
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

Mohammadreza Iman, Khaled Rasheed, and Hamid Arabnia. 2022. A review of deep transfer learning and recent advancements. CoRR, ahs/2201.096799
Jason Phang, Thibault Févry, and Samuel Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. CoRR, abs/1811.01088.
Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu. Huaiyuan Ying. Chuanqi Tan, Mosha Chen, Song- fang Huang, Xiaozhong Liu, and Sheng Yu. 2023. Biomedical question answering: A survey of ap- proaches and challenges. ACM Computing Surveys, 55(2):35:1335:36
Jo Plested and Tom Gedeon. 2022. Deep transfer learning for image classification: A survey. CoRR. 0hs/2205.09904.
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, Cédric Renggli, André Susano Pinto, Sylvain Gelly, Daniel Keysers, and Neil Houlsby. 2021. Scalable transfer learning with expert models. In Proceed- ings of the 9th International Conference on Learning Representations.
Simon Korblith, Mohammad Norouzi, Honglak Lee. and Geoffrey E. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, pages 3519-3529.
Sam T. Roweis. 1997. EM algorithms for PCA and SPCA. In Proceedings of the 1997 Conference on Neural Information Processing Systems, pages 626 632.
Nupur Kumari, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. 2022. Ensembling off fifethe-shelf models for GAN training. Proceedings of the 2022 IEEE/CVF
Conference on Computer Vision and Pattern Recog- nition, pages 10641-10652.
Sebastian Ruder, Matthew Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 15-18.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059.
Yandong Li, Xuhui Jia, Ruoxin Sang. Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. 2021. Ranking neural checkpoints. In Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition, pages 2663-2673.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf, 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilitics of large language mod- els mirage? CoRR, Abs/2304.15000
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, 1119997.11692
Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang. Xiaogang Wang. Ying Shan, and Ping Luo. 2022. Not all models are equal: Predicting model transferability in a self-challenging fisher space. In Proceedingy of the 17th European Conference on Computer Vision, pages 286-302.
Yifei Luo, Minghui Xu, and Deyi Xiong. 2022. Cog- taskonomy: Cognitively inspired task taxonomy is beneficial transfer learning in NLP. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 904-920.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. CoRR, abs/2103.15316.
Amiel Meiseles and Lior Rokach. 2020. Source model selection for deep learning in the time series domain. IEEE Access, 8:61 166190-2200.
Shuteng Niu, Yongxin Liu, Jian Wang, and Houbing Song. 2020. A decade survey of transfer learning (2010-2020). IEEE Transactions on Artificial Intelli- gence, 1(2):151-166.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière. Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and efficient foundation language models. CoRR, 0nn/2302.11977
OpenAI. 2022. Introducing ChatGPT. https:// openai wwwpblog/ccccoccccocccoooooooooo hommbbt.g/ccagpt...
Michal Pándy, Andrea Agostinelli, Jasper R. Uijlings, Vittorio Ferrari, and Thomas Mensink. 2022. Trans- ferability estimation using bhattacharyya class sep- arability. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9162-9172.
Anh Tuan Tran, Cuong Nguyen, and Tal Hassner. 2019. Transferability and hardness of supervised classification tasks. In Proceedings of the 17th IEEF/CVF International Conference on Computer Vision, pages 1395-1405.

---

Tu Vu, Brian Lester. Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Compulational Linguistics, pages 5039-5059.
next sentence prediction and mutual information. Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics, pages 562-574.
Chengkun Zheng, Guanyi Chen, Chenghua Lin, Ruizhe Li, and Zhi Chen. 2021. Affective decoding for em- pathetic response generation. In Proceedings of the 14th International Conference Natural Language Generation, pages 331-340.
Tu Vu, Tong Wang. Tsendsuren Munkhdalai, Alessan- dro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Ex- ploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7822-7926.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In 7th In- ternational Conference on Learning Representations.
Zekun Wang, Ge Zhang, Kexin Yang, Ning Shi. Wangchunshu Zhou, Shaochun Hao, Guangzheng Xiong. Yizhi Li. Mong Yuan Sim, Xiuying Chen, Qingqing Zhu, Zhenzhu Yang, Adam Nik, Qi Liu, Chenghua Lin, Shi Wang, Ruibo Liu, Wenhu Chen, Ke Xu, Dayiheng Liu, Yike Guo, and Jie Fu. 2023. Interactive natural language processing. CoRR, 0bs/2305.13246.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. 2021. Logme:
Practical assessment of pre- trained models for transfer learning. In Proceedings of the 38th International Conference on Machine Learning, pages 12133-12143.
Amir 7amir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik. and Silvio Savarese. 2019. Taskonomy: Disentangling task transfer learning. In Proceedings of the 28th Inter- national Joint Conference on Artificial Intelligence. pages 6241-6225.
Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma. Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. GLM- 130B: An open bilingual pre-trained model. In Proceedings of the 1 11th International Conference on Learning Representations.
Kun Zhao, Bohao Yang, Chenghua Lin, Wenge Rong,
Aline Villavicencio, Xiaohui Cui. 2023. Eval- uating open-domain dialogues in latent space with

---

A Fine-tuning Results GLUE
PLM Candidates
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
WNLI
bert cert-bse-unccnn
56.3
92.7
88.6
89.6
84.7
91.8
69.3
53.5
distilbert-base-unceee
51.3
91.3
87.5
88.5
82.2
89.2
59.9
56.3
pert-base-casse
58.2
91.7
87.8
89.2
83.9
91.0
66.1
46.5
distilbert-base-casse
47.2
90.4
85.6
87.8
81.5
88.2
60.6
56.3
roberta-base
63.6
94.8
90.2
91.9
87.6
92.8
78.7
57.7
distilroberta-ben base
59.3
92.5
86.6
89.4
84.0
90.8
67.9
52.1
Table The fine-tuning performances of candidate PLMs on GLUE dev datasets reported from HuggingFace, where the metries are Matthews Correlation Coefficient (MCC) for CoLA and Accuracy for the other datasets.
B Implementation Details
Our experimental machine contains Intel(R) Xeon(R) Silver 4110 CPU 2.10GHz and NVIDIA GeForce RTX 3090 24G GPU. For the implementation of TE methods, since some methods' performance heavily depend on the number of feature dimensions, we reduce the feature dimension to [16, 32, 64, 128, 256, 512, 768] by PCA for each method to find their most suitable dimensions. For DSE, RSA, KNN, MSC, LFC, PARC that need to compute the affinities between sample features, we also try different sample affinity functions including cosine, euclidean and correlation distances. The implementation details of surveyed methods are as follows:
DSE Among the averaged sample affinities (Luo et al.. 2022) and the affinity between the mean features (Vu et al., 2020), we found that the former performs better. And DSE achieves the best results when DeBERTa is taken as target model, Euclidean distance used to measure the sample affinity and the number feature dimensions 768. The corresponding computation is as Eq. 1.
S(D) = N Σ (1((())) ((;:)ll
DDS Among a number of instances of DDS framework (Dwivedi et al., 2020), RSA (Dwivedi and Roig. 2019) shows the best performance when DeBERTa is trained as target model, correlation distance is used and the number of feature dimensions is 512. Specifically, the pre-trained features and target features are first processed by z-score normalization. Then the pre-trained affinity graph and target affinity graph are computed by correlation distance. Finally, the lower triangular adjacent matrices of two graphs are compared by Spearman correlation coefficient transferability score.
MSC We use the code of silhouette score from scikit-learn implement MSC, which exhibits the best performance when cosine distance is used and the number feature dimensions 256.
ANN We use the code of KNeighhorsClassifir from scikit-learn and use the test accuracy of lcave-one- out cross-validation to quantify the transferability. We tune the k in [1, 3, 5, 71. the method exhibits the best performance when h 5, correlation distance is used and the number of feature dimensions is 64.
The computation process of PARC is similar that of RSA except that the target affinity graph is replaced by affinity graph derived from samples' one-hot labels. We use the code from here and the best performance is achieved when correlation distance is used and the number of feature dimensions is 512.
PARC
GBC It first uses Gaussian distribution to model each target class of samples which is parameterized by the in-class pre-trained features vectors. Then the averaged Bhattacharyya distance between every pair of different classes are used to measure the inter-class overlap L'qs. and 3:
BC = (o) (s(x))
(2)

---

S(D) Σ x
(pr PPg))
(3)
where v is a specific value of target classes. We use the code from here and the most suitable number of feature dimensions is 64.
Logistic We use the code of LogisticRegression from scikit-learn with the default typer-parameee classify the pre-trained features. The accuracy of leave-one-our cross-validation used to quantify the transferabilityy and the most suitable number of feature dimensions is 64.
H-Score As Eq. 4 shows, it first computes the covariance matrix over the feature dimensions of pre- trained features and that over the feature dimensions of each target class's S mean feature, then the trace of the dot- dot-product between the inverse matrix of former and the latter is used to approximate the optimal log-loss. We use the code from here and the most suitable number of feature dimensions is 768.
(DD tr(cov(6((2)) 1 cov (Ep((xx)))((2)))))))
(4)
Regularized H-Score Compared to H-Score, it further solve the statistical problem of covariance estimation by shrinkage-based estimator (Ibrahim al., 2022). The most suitable number of feature dimensions is 768.
WLEEP first uses Gaussian mixture model to fit the pre-trained features, then computes the Log Expected Empirical Prediction (LEEP) score between posterior distribution derived from fitted Gaussian mixture model and the target labels as Eq. 5:
1 S(D) N
Σ log( P(ylc)P(dé(z))
(5)
CEC
where e is the specific Gaussian component and C is the space of all components. We use the code from here where the number of Gaussian components is five times that of target classes and the most suitable number of feature dimensions is 64.
TransRate argues that the mutual information I()(X),)) H(((X)) (()())))) between the pre-trained features and the target labels serves as a strong indicator for the performance of model. Since the mutual information notoriously difficult to compute especially for continuous variables high-dimensional settings, the authors turn to utilize rate distortion that is closely related to Shannon entropy. The code can be found in (Huang et al., 2022) and the most suitable number of feature dimensions before computing the mutual information is 64.
LogME uses marginal evidence of the target task P(ylo(z)) f P(w) d(x), w)dw where is the weight of linear classifier. The prior P(w) defined as Gaussian and P(ylo(x), w) a Gaussian likelihood, then P(ylo(=)) can analytically estimated. We use the code from here, and the most suitable number of feature dimensions is 512.
SFDA It first projects the pre-trained features by regularized Fisher Discriminant Analysis such that the projected features have better separability of target classes which simulate the fine-tuning dynamics. Based on Bayes classification, the projecting weights can be used to compute the prediction probability of each sample on target label which further used compute the log likelihodoo as the transferability. Moreover, confidential mix noise is further added to examine the model's 's ability to classify hard samples. We use the code from here and the most suitable number of feature dimensions is 512.
PACTran It has three instances using the Dirichlet, Gamma and Gaussian prior distributions respec- tively. Since the first two priors require the pre-training task head layer non-negative features which are not the case in PLM selection, from here, and tune the and of in 10.1, 10] and [1, 10, 100, 1000]. The PACTran performs best when x 1, o 10 and the number of feature dimensions is 64.
implement PACTran with the more general Gaussian prior by the code

---

C TE Methods Performances on GLUE Tasks
Method
CoLA
SST-2
MRPC
QQP
MNLI
QNLI
RTE
WNLI
Reciprocal Rank
DSE
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
DDS
0.50
0.33
0,50
0.50
0.50
1.00
0.50
0.25
MSC
1.00
0.17
0.50
0.17
1.00
0.50
0.50
0.20
KNN
0.33
0.33
0.33
1.00
1.00
1.00
0.17
1.00
PARC
1.00
0.20
1.00
1.00
1.00
1.00
1.00
1.00
GBC
0.50
0.17
0.33
1.00
0.50
0.33
0.50
0.20
Logistic
0.33
0.17
1.00
1.00
1.00
1.00
1.00
1.00
H-Score
1,00
0.20
1.00
1.00
1.00
1.00
1.00
0.17
Reg. H-Score
1.00
0.20
1.00
1.00
1.00
1.00
1.00
1.00
NLEEP
0.50
0.17
0.33
1.00
1.00
1.00
1.00
1.00
TransRate
0.17
0.17
0.50
1.00
0.25
0.25
0.25
0.25
LogME
1.00
0.17
1.00
1.00
1.00
1.00
1.00
1.00
SFDA
0.50
0.50
0.50
1.00
1.00
1.00
1.00
0.25
PACTran
1.00
0.17
1.00
1.00
1.00
0.50
1.00
0.50
Spearman correlation coefficient
DSE
0.43
0.49
0.37
0.49
0.49
0.31
0.37
0.75
RSA
0.49
-0.09
0.20
0.60
0.26
0.71
0.43
-0.06
MSC
0.94
-0.89
0.31
-0.26
0.60
0.71
0.77
-0.41
ANN
0.60
-0.09
0.37
0.43
0.71
0.94
-0.26
0.57
PARC
0.94
0.43
0.54
0.83
0.77
0.60
0.89
0.38
GBC
0.60
-0.89
0.37
0.71
0.37
0.49
0.77
-0.67
Logistic
0.49
0.14
0.14
0.77
0.71
0.71
0.60
0.64
H-Score
0.77
0.14
0.49
0.66
0.83
0.83
0.94
-0.06
Reg. H-Score
0.83
0.14
0.77
0.66
0.83
0.83
0.71
0.06
NLEEP
0.66
-0.89
0.37
0.54
0.60
0.83
0.20
0.81
TransRate
-0.89
-0.49
0.77
0.37
0.09
0.43
-0.31
0.26
LogME
0.83
-0.14
0.66
0.66
0.77
0.83
0.83
0.52
SFDA
0.94
0.49
0.37
0.71
0.83
0.89
0.60
-0.14
PACTran
0.77
-0.20
0.66
0.60
0.71
0.77
0.66
0.09
Estimating Time
DSE
0.38s
2.90s
0.16s
17.86s
31.86s
4.48s
0.09s
0.03s
RSA
157.62s
221.01s
25.11s
224.61s
226.94s
220.50s
10.58s
0.63s
MSC
4.94s
7.30s
0.98s
7.32s
7.22s
7.35s
0.48s
0.06s
ANN
2.59s
2.52s
0.51s
117.75s
28.28s
15,99s
0.26s
0.04s
PARC
104. 13s
145.10s
17.32s
141.218
142.70s
142.18s
7.31s
0,45s
GBC
0.07s
0.34s
0.02s
1.70s
1.85s
0.50s
0.01s
0.01s
Logistic
0.55s
3.28s
0.31s
13.35s
20.02s
3.19s
0.19s
0.09s
H-Score
3,04s
11.78s
2.03s
125.86s
136.40s
17.47s
1.84s
1.32s
Reg. H-Score
3.84s
17.15s
2.55s
95.83s
118.07s
25,83s
2.30s
1.74s
NLEEP
18.49s
27.26s
4.83s
26.96s
32.49s
22.80s
3,22s
0.27s
TransRate
0.09s
0.42s
0.02s
2.16s
2.33s
0.69s
0.02s
0.01s
LogME
1.23s
5.07s
0.69s
37.80s
33.36s
7.55s
0.62s
0.55s
SFDA
9.37s
67.36s
4.22s
356.30s
385.94s
104.57s
3.07s
1.38s
PACTran
0.68s
4.37s
0.36s
19.52s
27.23s
4.82s
0.20s
0.09s
Table 6: The reciprocal rank scores, Spearman correlation coefficients and estimating time of TE methods on each GLUE dataset.


Reviews:
[{"review_id": "VZA2YQGkfx", "reviewer": "Reviewer_JTER", "summary": "", "questions": "1. The conclusion should be revised. How to select a valid PLM for a target task efficiently should be clearly summarized in the conclusion based on your analysis.\n2. The analysis does not have any problem, what do authors think about the novelty of this paper? It is an important factor to determine whether to receive this paper or not.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! \nThe novelty and contribution of this work are:\n       (1) summarizing as well as categorizing the previous works for model selection.\n       (2) providing a empirical study to compare current methods on GLUE benchmark.\nwihch are still lacking in the existing works.\n\nAnswer to question#1:\nWe will add the instructions to select PLM according to the empirical conclusion in the revised version.\n\nAnswer to question#2:\nAs a survey paper, the novelty of this work is to provide a taxonomy over model selection methods and also the empirical studies which have not been explored.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper lists existing transferability estimation methods and analyzes the performance of selected models. Experimental results show H-Score performs well with superiorities in effectiveness and efficiency.", "reasons_to_reject": "This paper focuses on the analysis of existing transferability estimation methods, but the novelty is limited. Although as a survey paper, proposing a new method or evaluation metric is not necessary, the contribution of this paper should be expanded to be qualified to be published.", "questions_to_authors": "1. The conclusion should be revised. How to select a valid PLM for a target task efficiently should be clearly summarized in the conclusion based on your analysis.\n2. The analysis does not have any problem, what do authors think about the novelty of this paper? It is an important factor to determine whether to receive this paper or not.\n", "paper_topic_and_main_contributions": "This paper makes a thorough survey of existing transferability estimation methods and conducts an empirical study on the GLEU benchmark.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "Y4HXbXiPsW", "reviewer": "Reviewer_oqBj", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for you comments! We will polish the paper to improve our writing to meet the standard of academic publication.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper's primary results represent a significant amount of work that could be useful to the community. Table 1 provides a compact comparison of many transferability estimation methods with references. Table 3 shares the results of an extensive comparison that highlights advantages and disadvantages of different methods. These two tables could be a good starting point for community members interested in getting started with transferability estimation for language models.", "reasons_to_reject": "While the paper's experimental results are potentially valuable, there are some major issues with the writing. One challenge of writing a survey paper is clearly and succinctly describing a range of existing work while adding useful cross-work comparisons. The survey should provide more value than a list of titles and abstracts. The current draft of this paper does not meet the standards of academic writing or fluent English. Descriptions and comparisons of methods are not presented clearly in a way that enables linear reading.\n\nRecommendation to the authors: Ask a fluent English speaker who has published papers at past ACL/EMNLP conferences for help revising the paper (including both organization and presentation of information).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors conduct a qualitative and quantitative analysis of many transferability estimation methods in the context of pre-trained language models. The paper's text compares and contrasts the methods while tables and graphs present experimental results for running the methods with several publicly available models and the GLUE benchmark. As a result, readers can gain a sense of the relative advantages and disadvantages of different transferability estimation methods for these types of language models and NLP tasks.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "This paper needs to be proofread by a fluent English speaker."}, "extra_text": {}}, {"review_id": "dfXH5Fgff3", "reviewer": "Reviewer_35ZS", "summary": "", "questions": "Question A: The models you compared seem to be quite similar. Would you think that including other models, like e.g. T5 could have changed results? What about LLMs?\n\nQuestion B: Basically, you want to find out which models are similar, right? Because we can expect similar models to perform similarly on the same tasks, correct? This might be a shot in the dark, but have you considered the alignment of different models' feature spaces? I think this could bring in an interesting (and further) dimension of transferability estimation (my hypothesis being that models with similar feature spaces perform similarly).\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for you kind comments!\n\n(1) We will polish the problem formalisation to facilitate the understanding of readers.\n\n(2) Answer to question A: We will explore more challenging scenarios of model selection including LLMs which also face the selection problem as the increasing number of LLMs.\n\n(3) Answer to question B: Thanks for you suggestion! The similarity between target model and candidate model has been utilized by Model Similarity-based Methods to score the transferability of candidate model, while training an extra target model might be time-consuming.  However, the utilization of similarity between candidate models is unexplored in Training-free Methods. As you pointed, I totally agree this is an impoortant consideration worthy of further exploration.\n\n(4) We will add the missing references.\n\n(5) Thank you for your patience in pointing out typos and writing improvements which made us learn a lot, we will improve these points in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The following are reasons to accept:\n\n- The paper presents an extensive evaluation of methods for transferability estimation and thus fills a gap in this specific research area.\n\n- The authors present a well-researched and mostly well-written survey. They present the issue and the different parts in digestible sections and follow a clear and reasonable methodology.\n\n- The authors draw valuable conclusions from their work and present interesting future directions.\n\n- The ablation study is particularly interesting, as is the feature dimension reduction.", "reasons_to_reject": "There are per se no reasons to reject this paper. However, there are a few points which could improve the paper:\n\n- include some examples in the problem formalisation. this is a bit \"dry\" to read.\n\n- the paper could have benefited from a more diverse model selection. See also Q1.", "questions_to_authors": "Question A: The models you compared seem to be quite similar. Would you think that including other models, like e.g. T5 could have changed results? What about LLMs?\n\nQuestion B: Basically, you want to find out which models are similar, right? Because we can expect similar models to perform similarly on the same tasks, correct? This might be a shot in the dark, but have you considered the alignment of different models' feature spaces? I think this could bring in an interesting (and further) dimension of transferability estimation (my hypothesis being that models with similar feature spaces perform similarly).\n\n", "paper_topic_and_main_contributions": "This paper presents a survey of transferability estimation to answer the question about which pre-trained language model should be used to maximise the performance for a specific downstream task. To that end, the paper compares 14 different methods and evaluates them with the help of six different pre-trained language models. As such, this survey presents a model selection approach which has the potential to contribute help and guide research in identifying suitable models for suitable tasks. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "110: For transfer learning, I'd probably include work by Ruder et al. https://aclanthology.org/N19-5004/\n\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I include general questions here. I do not expect a detailed answer for them, but are thought as food for thought for the authors. Some comments are high-level and I leave it to the authors if they want to consider them. Some are just common thoughts.\n\n015: \"make a survey\" does not sound like an idiomatic expression\n\n019: based on the GLUE\n\n033: especially on Large Language Models (LLMs)\n\n037: clear formulation of the problem --> good!\n\n040: variation goes more significant --> reformulate\n\n042: delete such\n\n044: work\n\n050: impossible rather than prohibitive\n\n050: Thus, \n\n052: what problem was solved --> maybe give an example?\n\n105: the H-Score\n\n139: explain LogME\n\n144: in what way are the researches remarkable?\n\nSection 3: While the problem formulation and formalisation are correct, it would help to have examples instead of mathematical expressions only.\n\n231: the target task\n\n274: the H-Score\n\nTable 1: good overview, maybe colorise hooks (blue) and crosses (red) to make them more distinguishable\n\n327: are also \n\nTable 2: explain Mcc.\n\nAppendix A/Table 5: highlight best-performing score per dataset/task\n\n887: As Eq. 4 shows, \n\n385: MRR?\n\n"}}]

Author rebuttal:

Meta-review: The main contribution of this work is creating a taxonomy of methods to estimate the "transferability" of Pretrained Language Models (PLMs) based on whether they require certain inputs, are task-agnostic, take into consideration the dynamics of the features during fine-tuning, and/or are training-free. However, the properties of the surveyed methods are not always adequately expounded. Moreover, the Authors provide a comparison of their performance on tasks from GLUE. However, the selection of PLMs is limited to variants of BERT and RoBERTa encoders; hence, it is hard to determine if the findings generalise to auto-regressive and encoder-decoder PLMs. Finally, despite computer vision being prominent in the abstract and introduction, no studies have been carried out in this domain. Hence, there is a mismatch between the motivation and the experimental setup. Overall, this paper might be valuable as a summary of previous work on transferability, but its novelty is somewhat limited, as it mostly compares existing methods in a standard NLP benchmark. This submission might be considered for acceptance to Findings.
Decision: Accept-Findings