k-NN as a Simple and Effective Estimator of Transferability
Authors: Moein Sorkhei
Christos Matsoukas
Johan Fredin Haslum
Emir Konuk
Kevin Smith
Venue: Accepted by TMLR ()

Abstract: How well can one expect transfer learning to work in a new setting where the domain is shifted, the task is different, and the architecture changes? Many transfer learning metrics have been proposed to answer this question. But how accurate are their predictions in a realistic new setting? We conducted an extensive evaluation involving over 42,000 experiments comparing 23 transferability metrics across 16 different datasets to assess their ability to predict transfer performance. Our findings reveal that none of the existing metrics perform well across the board. However, we find that a simple k-nearest neighbor evaluation -- as is commonly used to evaluate feature quality for self-supervision -- not only surpasses existing metrics, but also offers better computational efficiency and ease of implementation.

Published in Transactions on Machine Learning Research (10/2025)
NN as Simple and Effective Estimator of Transferability
Mocin Sorkhei
sorkbei@kth.se
KTH Royal Institute of Technology, Stockholm, Sweden Science for Life Laboratory, Stockholm, Sweden
Christos Matsoukas
matsou@kth.se
KTH Royal Institute of Technology, Stockholm, Sweden Science for Life Laboratory, Stockholm, Sweden
Johan Fredin Haslum
Shnsllm@@thh..m
KTH Royal Institute of Technology, Stockholm, Sweden Science for Life Laboratory, Stockholm, Sweden
Emir Konuk
ekonak@Wkth.cc
KTH Royal Instibute of Technology, Stockholm, Sweden Science for Life Laboratory, Stockholm, Sweden
Kevin Smith
ksmith@kth.cc se
KTH Royal Institute of Technology, Stockholm, Sweden Science for Life Laboratory, Stockholm, Sweden
Reviewed on OpenReview: https: openreview. net/ pormforum? id- PolkyPIHHc
Abstract
How well can one expect transfer learning to work in new setting where the domain is shifted, the task is different, and the architecture changes? Many transfer learning metrics have been proposed to answer this question. But how accurate are their predictions in realistic new setting? We conducted an exlensive evaluation involving over 42,000 experiments comparing 23 transferability metrics across 16 different datasets to assess their ability to predict transfer performance for image classification tasks. Our findings reveal that none of the existing metrics perform well across the board. However, we find that a simple k-nearest neighbor evaluation as is commonly used to evaluate feature quality for self-supervision not only surpasses existing metrics, but also offers better computational efficiency and ease of implementation.
1 Introduction
Transfer learning is a widely used technique for reusing knowledge learned in one domain the source to improve performance in another - the target. The question whether transfer learning will be bencficial in a particular setting, and to what extent, is not always clear. Many metrics designed for this purpose so-called transfcrability metrics have been proposed. They typically focus on only one of three factors that influence the transfer learning process: (1) the domain distance iee. what happens when the larget data differs from the source, (2) variations in task and task complexity f.e. what happens when the target task differs from the source task, and (9) the choice of architectural design i.e. what happens when the network architecture changes. We argue that a good transferability metric should simultaneously account for all these factors not only independently, but also in combination.
We further posit that, when prodicting transferability, one should not only consider the accuracy in predicting the final performance of the transferred model in the new setting, but also the ability predict performance improvements gained through the transfer learning process. The reasoning behind the former should be

---

Published in Transactions on Machine Learning Research (10/2025)
distance between source and target domains
complexity the tanged task
Figure 1: We investigate how well existing transfcrability metrics perform across three essential factors of transferability (Left): 1) shifts between the source and larget domains, 2) changes in larget lask, and 3) changes in network architecture. (Right) How effectively different metrics predict transferability, averaged across all the above settings. Surprisingly, we find that a simple nearest neighbor evaluation (I-NN) outperforms all existing metrics from the literature.
obvious interested in knowing what the final performance of the model will be once we apply transfer learning. But we should also be interested in the latter knowing how much benefit, if any, we will see from transferring knowledge from the source domain (i.e. the pretrained weights) as compared to learning from a random initialization. Carelessly transferring from an inappropriate data source can lead to biased or even inferior performance (Torralba & Efros 2011 Matsoukas et al. 2022 Zhang et al., 2022 Raghu et al. 2019 He et al. 2019). If random initialization can do a similar or better job, shouldn't we want know?
In this work, we conducted an extensive evaluation involving over 42,000 experiments comparing 23 transfer- ability metrics across 16 different datasets to assess their ability to predict transfer performance for image classification. We establish desiderata (the key fcatures or qualitics) of good transferability metric - it should accurately predict performance in the larget setting after transfer learning with robustness to domain shifts, changes in the task, and architecture changes (see Figure D Furthermore, the metric should be able to predict both the final performance the transferred model as well as the gains anticipated from transfer learning. Our experiments show that none of the existing metrics consistently performs well across these crileria.
In pursuit of a robust metric, we explore the application of nearest neighbor evaluation (K-NN) commonly used technique in self-supervision to evaluate the quality of learned feature representations - within the context of transferability estimation. We discover that. simple nearest neighhor cvaluation surpasses all the metrics from the literature (see Figure []] Furthermore, k-NN possesses attractive properties such as being cheap, optimization-free and closely resembling the target classification task. Additionally, its straight implementation adds to its appeal. Together, these factors make k-NN a compelling choice as a transferability metric.
2 Methods
In this work, we ask: to what extent are established metrics of transferability useful across changes in domain, lask complexity, and archileclure? Is there a single metric that performs well across all scenarios? We begin by describing the metrics used in our study.
Transferability metrics Transfcrability metrics are designed to predict the efficacy of transfer learning, taking into account task and architectural differences. Below, we briefly describe 17 such metries, starting with metrics that model the relationship between source and target tasks:
2

---

Published in Transactions on Machine Learning Research (10/2025)
NCE
2019 focuses task differences and estimates transferability by calculating the negative conditional entropy between the source and target labels.
(Tran et al.
LEEP Nguyen et al. 2020) models the joint empirical distribution between the source labels, predicted by the pre-trained model, and the target labels and calculates the average log- likelihood of the target labels given the predicted source labels.
ОТСE (Tan et al. 2021 models task and domain differences. Domain difference is measured using optimal transport between source and target domains, and task difference is measured using conditional entropy between source and target labels.
NumC (Agostinelli ct al. 2022 additionally utilize the inverse of the number of classes as trivial transferability metric, implying that a lower number of classes signifies higher transfer performance.
In addition, we consider transferability metrics designed to predict transfer efficacy under architectural changes:
TransRate (Huang et al. 2022 estimates the mutual information between the target embeddings, extracted from the pretrained model, and the target labels.
LogME (You et al. target label evidence given the target features extracted from the pro-trained model.
2021 models transferability through estimating the maximum value of the
GBC (Pándy et al.
2022 models each target class with a Gaussian distribution in the pre-trained feature space and calculates the Bhattacharyya distance Bhattacharyya (1946) between all pairs of classes.
N-LEEP et al. 2021) is an extension of LEEP with the difference that it removes the classification head of the pretrained model and instead fits a Gaussian Mixture Model (GMM) on the target embeddings and uses such cluster assignments place of source labels to compute the LEEP score.
H-score (Bao et al. 2019) estimates the feature redundancy and the inter-class variance of the target. embeddings extracted from the pre-trained model. Low feature redundancy and high inter-class variance (high H-score) suggests good transfer lcarning performance.
SFDA (Shao et al. 2022) projects target features, extracted from the pre-trained model, into a space with high class separability and employs Bayes' rule to assign instances to different classes, assuming a normal distribution for each class.
NCTI (Wang et al. 2023) ganges the proximity between the current state of the pre-trained model and its hypothetica ical state the terminal stage of fine-tuning. Closer proximily signifies superior transfer performance.
ETran (Gholami et al. 2023 akin to SFDA, projects target features into a class-separable space and employs Bayes' rule, following the assumption normal distribution for each class. also computes an energy score on target features to determinc whether the target data in- or distr-diistribttion for the pre-trained model.
PACTran (Ding et al. 2022 assesses transferability by quantifying the pre-trained model's gen- eralization to the target task, utilizing the PAC-Bayes bound (McAllester 1998 Germain al. 2009).
PARC (Bolya et al. 2021 computes the dissimilarity between pairwise target features and the dissimilarity between corresponding pairwise labels. It then calculates the correlation between these two measures, operating on the premise that images with distinct labels should exhibit dissimilar feature representations.
TMI Xu & Kang 2023) assesses transferability through measuring intra-class variance of the target features, arguing that higher intra-class variance leads to higher transfer performance.
3

---

Published in Transactions on Machine Learning Research (10/2025)
LFC Deshpande et al. 2021] uses linearized framework to approximate finetuning and measures Jabel-Feature correlation for estimating transferability.
EMMS (Meng et al. 2023 estimates transferability by establishing a linear regression between target features and labels derived from a foundation model.
Domain distance metrics The distance between two domains is often crucial for successful transfer learning - larger domain distance is associated with reduced net gain from transfer learning.
While not explicitly designed to predict transferability, we consider 5 widely recognized methods for measuring domain distance:
The FID score models each dataset with Gaussian distribution and measures the distance between the two datasets using the Fréchet distance.
Fréchet Inception Distance (Heusel et al. 2017 Fréchet 1957
Earth Mover's Distance (Cui et al. 2018 Rubner et al. 2000 : EMD views each dataset as a set of clusters represented by the class mcan features, weighted by the number of images per class, and calculates the Wasserstein distance between the two sets of clusters.
Kernel Inception Distance [Binkowski et al. 2018 Gretton et al. 2012 KID is a non- con-parametric distance measure based on the maximum mean discrepancy the features after applying a polynomial kernel.
Image Domain Similarity (Mensink al. 2021 IDS is asymmetric distance measure that takes samples from the source and target domains and calculates the average of the distances between each target sample to its closest source sample.
Intrinsic Multi-scale Distance (Tsitsulin et al.) 2019): The IMD metric measures the discrepancy between two distributions by approximating the underlying data manifolds and the distance is calculated using the spectral Gromov- Wassersteen inter-manifold distance.
Mean-dist: In addition to the 5 metrics described above, we also compute the the average of all domain distance metrics, denoted as "mean-dist".
k-NN transferability metric In this work, we propase use nearest neighbor evaluation (k-NN)
metric of transferability. Drawing inspiration from its use in self-supervised tasks to measure the pretrained
model's ability to partition the data based on its relevance to the target task (Caron et al. 2021 Wu et al.
2018
He el al. 2020), we employ A-NN to predict transferability.
Specifically, we split the training set (S) of the target in two disjoint subsets S1, S2 of 80%-20% of the training set. Subsequently, k-NN classification was performed on S2 using nearest neighbors from S1. The resulting N-NN accuracy served as the transferability score (to ensure reliability, we repeated the same procedure with 3-fold vols-validation similarity of extracted image features was used to measure distance between data points. We chose default value of k = 200, and found that its performance remained consistent w.r.t. lo (Figure 9b in the Appendix).
the training set, yielding identical results). Following common practice, cosine
3 Experiments
Our experiments are conducted as follows. Given source, target, and architecture, obtain scores from 23 metrics applied to the training set of the target data. We then use transfer learning to train networks on the larget dataset and measure the absolute and relative performance. Transfer learning involves initially training a network on a source domain, followed by fully fine-tuning it on the target domain. To assess the performance of metric, we compute the correlation between its transfcrability score and the observed performance after transfer learning. In addition to the main experiments, we measure transferability prediction performance isolated for domain shift, change of task, and change of architecture.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Datasets We apply transfer learning across diverse set of 16 image classification datasets. For the source domains, we selected IMAGENET (Deng et al.] 2009), 2N17 (Van Horn et al. 2018). PI.CCES365 (Zhou
et al. 2017). and NABirds (Van Horn et al. (015). As target datasets, we include well-known benchmarks such as CIFAR-10 and CIFAR-100 Krizhevsky et al. (2009), Caltech-101 (Fei-Fei et al. 2004). Caltoch-256
GGrillin et al. 2007), StanfordDogs KKoosla el al.) 2011); Aircraft (Maji et al. 2013), NABirds Van Horn et al.
(2015), Oxford-III Pet. (Parkhi et al. 2012)), SUN397 (Xiao et al., 2010),, DTD (Cimpoi et al. 20144) AID
(Xia et al. 2017), and APTOS2019 Karthik (211 These datasets cover super-ordinate object recognition,
ain-graineddeee classification, scene recognition, texture classification, aerial imagery, and medical imaging (details
in Table 3 in the Appendix).
Architectures To investigate the effect of architectural changes, we utilize set of 11 architectural variations drawn from five model familics. In detail, we utilize ResNet-50 (He et al.) 2016). Inception-Vis (Szegody 2018), RegNet Y -
et al. 2016), DenseNet-121 (Huang et al.) 2017), MobileNet-V2 (andler et al.
Radosavovic et al. 2020), ResNeXt-50 (Xie et al., 2017), WideResNeet550- (Zagoruyko & Komodakis 2016). DeiT-Small (Touvron et al. )021), Swin-Tiny (Liu et al. 2021), PiT-Small (Heo et al. 2021), and PVT-Small [Wang ct al.] (021). Domain distance metrics rely on a network extract features upon which the distance is calculated. For this, we consider five representative IMAGENerT-reereen architectures: ResNel-50, Inception- V3, DenseNet-121, DeiT-Small, and Swin-Tiny.
Implementation details For each dataset, either the official train/val/tent splits were used, or we made the splits following (Korublith et al. 2019). Images were normalized and resized to 256 x 256, after which augmentations were applicd: random color jittering, random horizontal flip and random cropping to 224 x 224 of the rescaled image. The Adam optimizer {Kingma & Ba 2014] was used for CNNs and AdamW (Loshchilov
& Hutter 2017) for ViT-based architectures, and the training of models was done using PyTorch (Paszke et al. 2019 We provide details regarding the training procedure in the Appendix.
What performance should we measure? When applying transfer learning, one may wish to predict:
absolute transfer performance the performance of the transferred model mcasured in the target domain, in absolute terms;
relative transfer performance - how much performance improvement was gained in the target domain as a result of transfer learning.
The absolute transfer performance is essential to measure how well the transferred model can address the target task/setting. Throughout this work, absolute transfer performance is denoted ferf(P).
While PP) gives prediction about the expected performance, it offers limited insight into the effectiveness of the knowledge transfer. Consider, for example, a task that is easy to solve. Even though the (PP may be high in absolute terms, it may not necessarily imply that transfer learning was beneficial - in fact, it often isn't (He et al., 20119 Neyshabur et al. 2020 Raghu et al.) 2019). We must also consider the performance gained as compared to the same model initialized with random weights (He et al. 2015), denoted as (RT). Following Neyshabur et al. (2020), we define the net gain from transfer learning as the Relative Tronsfer Performance (RIP) between pretrained network P and randomly initialized model RI, expressed as: RTP: :- pPrr When applying this to predictions from transferability metrics, PPP)(P)
we adopt a similar formulation RTPp = PP)(P) where est(P) and es((R)) denote the transferability scores for the pretrained and handomly-initialzen models respectively. We also introduce transfer gaz the complement of RTP, represented by TG RTP.
We judge the performance of the transferability metrics by checking how well the transferability scores correlate with the performance observed from actually applying transfer learning (both absolute or relative).
The complement, TC, captures the inverse relationship between domain distance and net benefits from transfer learning. It is used when measuring the correlation between domain distance metrics and the gain from transfer learning We invert. the domain distance estimates to predict absolute transfer performance, accounting for their inverse relationship.
5

---

Published in Transactions on Machine Learning Research (10/2025)
modorelii meticc
2833 pe perfomuuts
3.5
Conreaatin foo with zarere pe-tormare
Joran 1a By n
-25
Tresstig
Loongg
grmer
onde
2-00
Transhade
orcE
Lrc
(a) Correlations with absolute transfer performance.
(b) Correlations with relative transfer performance.
Figure 2: Effectiveness of different metrics when both target domain and task complexity change. Results are averaged across four source datasets described in the text. Full break-down of results appears in Tables 4 and in the Appendix.
This is quantified using weighted Kendall's tau (Tw) Vigna 2015), in line with previous studies (You et al,, 2021 Shao et al. 2022: Wang et al. 2023 Pándy al. 2022 Weighted Kendall's tau measures the correlation between ranked lists by assessing the number of correctly ordered pairs, particularly emphasizing the accurate ranking of high-performing models. We further employ other common evaluation measures in Figure 10 of the Appendix. We use accuracy to compute transfer performance for all lasks unless otherwise specified.
4 Results
We begin by reporting our main results, where the factors of transferability are considered simultaneously, followed by experiments isolating cach factor.
Which metrics are general predictors of transferability? Our main results consider the scenario where both the target domain and the target task change from the source. In this setup, a network is pre-trained on each of the four source datasets and subsequently transferred to each of the 12 target datasets (as detailed in previous section). For each network, we compute both the absolute and relative transfer performance, and then measure the correlation between these and the transferability metrics (including domain distance metrics).
The results are reported in Figure 2 Metrics designed to measure domain distance appear in pink, while metries designed to estimate transferability appear in blue. When predicting absolute transfer performance, the transferability metrics generally outperform the domain distance metrics, with the exception of LogME and H-score (Figure 2a). Among the transferability metries, GBC stands out as the top-performing metric. Domain distance metrics appear to have a negative correlation with absolute transfer performance. This may be attributed to thoir inability to consider task complexity or network architecture, two crucial factors directly influencing absolute transfer performance.
When examining Relative Transfer Performance (RTP), as depicted in Figure 2b the picture changes. In contexts where the focus is quantifying the benefits gained from transfer learning, domain distance metrics exhibit good performance (with the exception of TMD). Mean-dist, while being less effective than PACTran, proves to be particularly adept at assessing the net gains from transfer learning, albeit at a high computational cost.
Setting aside k-NN for now, several metrics fail perform well in either or both absolute and relative prodiction. GBC is the lcading metric for prodicting absolute transfer performance, while PACTran excels in terms of predicting relative transfer performance. This suggests that there is no universally superior metric in this setting- the most suitable choice may vary based on the intended use and the source dataset, elaborated in Tables in the Appendix.
Which metric is better when only the domain changes? We next consider how the metrics perform when only the domain distance changes the target task and network architecture remain the same. In this experiment, we use IMAGENET as the source domain and we fine-tune models on DomainNet (Peng) et al. 2019). DomainNet comprises six distinct domains, each featuring the same set of 345 categorics. As an additional experiment, we set each individual domain, within DomainNet, the source and use targets
6

---

Published in Transactions on Machine Learning Research (10/2025)
condittion isd wh 18031 tert
0.30
frwh radiis
010
325
-025
Ereen FaCTres
orcE
A-d0
330
Tranodate
Logig
aed
Emair
crce
(a) Correlations with absolute transfer performance.
(b) Correlations with relative transfer performance.
Figure 3: Effectiveness of different metrics when only the target domain changes. Results are averaged over the seven configurations described in the text. Full break-down of results appears in Tables 6 and in the Appendix.
Or 3ett 1oth roouus
05
23
.05
Traskuata
Deg
mreer oll
Lopelr
ten
Fi
(a) Correlations with absolute transfer performance.
(b) Correlations with relative transfer performance.
Figure 4: Effectiveness of transferability metrics when only larget task complerity changes. Results are avcraged over NABirds, SUN397, and Caltoch-256. Full results appear in Tables g and 9 in the Appondix.
the remaining domains. Condensed results appear in Figure 3 Detailed results are in Tables 6 in the Appendix.
In terms of absolute transfer performance (Figure 3al, LogME and H-score, which exhibited poor correlation in the previous setting, now emerge as the top-performing metrics. Interostingly, GBC is no longer the leading metric for predicting absolute transfer performance in fact, it is among the weakest. The correlation between the domain distance metrics (in pink) is better than in Figure 2a possibly because domain distance is the only varied factor. Perhaps surprisingly, transferability metrics perform better at predicting absolute transfer performance in this scenario (with the exception of TransRate and TMT).
When we consider relative transfer performance in Figure 3b the domain distance metrics show their superiority over the transferability metrics. EMD emerges as the top performer, albeit only by a narrow margin compared to FID and KID. PACTran, which was the best metric to predict relative transfer performance in the previous setting, exhibits negative correlation in this selting. k-NN's superior performance will be discussed later.
Which metric is better when only the task changes? Next, we consider the case of changing the task. Using IMAGENET the source, we apply transfer learning to subsets of NABirds, SUN397, and Caltech-256. More specifically, we vary the task complexity by varying the number of classes (as a proxy) in accordance with Deng et al. [2010): Konuk & Smith (2019). In detail, we sequentially remove classes from the target dataset to create target tasks of varying complexity. The results are provided in Figure 4 as averaged correlations, and the detailed results appear in Tables in the Appendix. Again, we refrain from discussing k-NN until a later section.
In terms of absolute transfer performance, it is evident from Figure 4a that the transferability metrics (blue) are generally superior to the domain distance metrics (pink). Domain distance metrics exhibit a strong negative correlation with absolute performance (except from IDS and IMD). Interestingly, LogME and H-score, the best metrics to predict absolute transfer performance in the previous setting (Figure 3a now appear ineffective.
The results on relative transfer performance (Figure 4bj follow a similar pattern as in Figure 4a for the transferability metrics (except for TransRate and LFC). However, the domain distance measures reveal a contrasting pattern. In general, both transferability and domain distance metries are either very highly
7

---

Published in Transactions on Machine Learning Research (10/2025)
bunddr saferaats
modte Settmare
Commintion som
Conneoto (1.) with
6.0
02
-R.b
Randues
Logag
Lee
nac
uc
mep
-Eat
Trenfaate
Logas
QBc
NCT
Erren
PACTeR
arc
TM
Derg
(a) Correlations with absolute transfer performance.
(b) Correlations with relative transfer performance.
Figure 5: Effectiveness of transferability metrics when only network architeclure changes. Results for each
metric denote average correlation over 12 downstream datasets. Full results for individual datasets appear in Tables and
correlated with relative transfer performance, negatively correlated. While there is no clear winner in this test, there are some clear losers (TransRate, LogME, H-score, LFC, IDS, and IMD).
Which metric is better when only the network architecture changes? We also examine the effect of isolating network architecture as factor of transferability. We consider 11 distinct network architectures starting from IMACENET pre-training and perform transfer learning on each of the 12 targets. We compute correlations between the transferability metries and the absolute and relative transfer performance. In this setting, we omit the domain distance metries since they are not applicable in this setup (as both the larget domain and task remain the same). The results are presented in Figure 5 and we provide detailed, per-dataset correlations in Tables 1 and 2
When it comes to predicting absolute transfer performance in Figure 5a N-LEEP outperforms the other transferability metries from the literature, followed by NCTI. This result unsurprising, as both of these metrics are specifically designed to characterize architectural variations. For relative transfer performance, shown in Figure 5b nearly all the existing transferability metrics fail, with many exhibiting a negative correlation. EMMS, followed by H-score and PARC, stand out by significantly outperforming the other metrics, suggesting that these are the only reliable transferability metrics when predicting relative transfer performance when architecture the only factor that changes.
k-NN as a reliable predictor of transferability Finally, we comment on the transferability predictions of k-NN. In Figures 2 through 5 k-NN results were reported in turquoise. In Figure 2 which showed the general ability to predict transferability, A-NN performed remarkably well - it performs comparably to the best transferability metrics when measuring absolute performance, and outperforms all other metrics in terms of relative transfer performance by a widc margin. This trend persists when the individual factors are isolated (Figures Notably, when isolating domain change (Figure 3 k-NN outperforms all the other metrics in predicting both absolute and relative transfer performance.
Compared to the methods from the literature, k-NN performed consistently well. In fact, k-NN was the only method among the 24 to maintain positive correlation throughout all the experiments. Combined with its optimization- and hyperparameter-free nature, as well as its low computational cost and case of implementation, our findings suggest that k-NN can be seamlessly employed reliable transferability metric.
Computational efficiency The total running time versus the overall performance of the metrics is depicted in Figure Da in the Appendix. It is evident that k-NN provides computational cost comparable to the fastest metrics while, at the same time, outperforming them.
5 Discussion
We argue that a metric for transferability should accurately predict both absolute and relative transfer performance, as each answer different but important questions (Lee what performance level can I expect? vs. what benefit will transfer learning provide?) Not only that, but an ideal metric should also be reliable across different transfer learning scenarios: changes in the domain, task, architecture.
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Effectiveness of transferability metrics to predict absolute transfer performance when only network
architecture changes, shown for individual target datasets.
Dogs
SUN397
DTD
CTF-100
CIF-10
Cal-256
Cal-101
Pet.
Aircraft
Birds
APTOS
ATD
TransRate
0.353
0.183
0.261
0.040
0.234
-0.066
-0.109
0.202
0.101
0.378
0.136
0.032
LogME
0.578
0.090
0.023
0.535
0.681
0.321
0.751
0.576
0.263
0.257
0.589
-0,260
GBC
0.765
A-LEEP
0.783
0.835
0.647
0.821
0.574
0,816
0.537
0.940
0.257
0.392
0.420
0.330
LEEP
0.576
0.648
-0.046
0.768
0.341
0.494
0.192
-0.152
0.462
0.187
0.080
-0.362
H-Score
0.515
0,211
-0,205
0.218
0.288
0.061
0.085
-0.096
0.141
-0.177
-0,012
-0.161
NCE
0.712
0.711
0.442
0.904
0.744
0,842
0,340
0.771
-0.173
0.359
0.254
-0,657
SEDA
0.530
-0.305
-0.388
0.419
0.623
0.119
0.077
-0.352
0.556
-0.160
0.800
0.640
NCTI
0.782
0.883
0.613
0.801
0.682
0.444
0.326
0.728
0,080
0.405
0.428
0.505
0.658
PACTran
-0.079
0.334
0.017
0.555
0.024
0.062
0.032
-0.305
0.703
0.088
0.212
0.223
PARC
0.803
0.631
0.173
0.813
0.673
0.125
0.410
0.356
0.008
0.716
0.309
0.357
ОТСE
0.398
0,235
0.003
0.253
-0.191
0.512
0.668
0.651
-0.
-0.480
-0.210
-0.191
TMI
0.158
0.250
0,237
0.054
0.037
0.369
0.043
0.050
0.733
-0.149
-0.072
-0.166
LFC
0.663
0.700
0.628
0.730
0.502
0.571
0.257
0.580
-0.372
0.642
-0.141
0.744
EMMS
0.566
-0.101
-0.351
0.450
0.684
0.010
0.393
0.236
0.594
0.079
-0.261
-0.437
E-NN
0.786
0.836
0.355
0.309
Table 2: Effectiveness of transferability metrics to predict relative transser performance when only network architecture changes, shown for individual target datasets.
Dogs
SUN397
DTD
CIF-100
CIF-10
Cal-256
Cal-101
Pet.
Aircraft
Birds
APTOS
AID
TransRate
-0.332
0.046
-0.402
0.242
0.229
-0,251
-0.515
-0.343
-0.059
-0.524
-0.507
0.094
LogME
0.344
0.167
0.121
0.307
-0.098
0.196
-0.060
0.266
0.341
-0.104
-0.075
-0.061
GBC
0.347
-0.322
U.649
-0.540
0.101
N-LEEP
0.310
0.168
-0.471
-0.530
-0.630
0.387
-0.598
-0.106
-0,226
0.328
0.248
-0.276
LEEP
-0.022
-0.257
-0.
-0.647
-0.336
-0.319
H-Score
0.704
0.273
0.104
0.416
0.629
0.767
0.527
-0.018
0.877
0.657
0.104
0.417
NCE
-0.431
-0.301
-0.725
-0.790
-0.435
-0.606
-0.722
-0.464
-0.642
-0.297
0.326
40.508
SFDA
0.383
0.215
0.547
0.165
0.149
0.391
0.290
0.447
0.460
0.385
0.273
0.388
NCTI
-0.204
-0.020
0.036
0.211
-0.127
0.573
0.234
0.221
-0.068
-0.238
-0.269
0.418
PACTran
-0.025
0.141
-0.110
-0.214
-0.176
0.112
0.297
0.317
-0.033
-0.312
0.338
-0.271
0.372
0.526
0.762
-0.252
0.673
0.840
0.524
-0.154
0.073
ОТСE
-0.328
-0.388
-0.608
-0.385
-0.636
-0.591
-0.610
-0.401
-0.531
-0.654
-0.569
-0.374
TMI
-0.157
-0.183
0.095
-0.153
0.091
-0.015
0.491
0.112
0.077
0.199
0.060
0.289
LFC
0.403
0.258
0.165
0.419
-0.005
0.100
0,680
0.261
0.393
0.181
0.347
0.252
EMMS
0.520
0.600
0.610
0.430
0.370
0.570
0.410
0.470
0.440
0.410
0.440
0.350
ENN
0.464
Which metrics should we use when we want to predict absolute transfer performance? If one cares about the final performance of the transferred model, irrespective any gain/loss from transfer learning, our experiments show that transferability metrics are superior domain distance metrics, but the choice of the best metric is not clear. While GBC is best when the domain and task change together (Figure 2a LogME and H-score demonstrate superior performance when domain changes in isolation (Figure 3a W When only the task changes, NCE, LEEP, N-LEEP, ETran, and PACTTra emerge as the top-performing metrics (Figure 4ap. When considering variations in network architecture, N-LEEP stands out as the most effective metric FFigure
5ap.
Domain distance metrics prove deficient, often anti-correlated with absolute transfer performance. This, combined with their inability to adequately account for task and architectural differences renders them unsuitable estimators of absolute transfer performance.
k-NN, on the other hand, consistently demonstrates good performance in every scenario comparable to or better than the best transferability metrics. When the domain changes in conjunction with the task, k-NN is comparable to GBC. When domain changes in isolation, k-NN stands out as the top-performing metric. When task variation is considered, j-NN is on par with NCE, LEEP, A-LEEP, ETran, and PACTran. Considering only architectural differences, performs comparably against A-LEEP while outperforming all the other metrics.
9

---

Published in Transactions on Machine Learning Research (10/2025)
Which metrics are better at predicting the relative transfer performance? While the ability to predict the end performance of transferred models is undeniably valuable, it leaves unanswered a critical question: how much did transfer learning actually help? Our investigations reveal that existing transferability metrics do not provide a satisfactory answer to this question. Transfcrability metrics often failed to yield consistent positive correlations when domain distance, lasks, or architecture were isolated. Domain distance metrics, on the other hand, exhibited mostly a positive correlation with the relative transfer performance. When only the domain changes, they were superior to transferability metries (Figure 3b), on par with them when only the task changes (Figure 4b), and slightly better when task changes along with the domain (Figure 2b
However, not all domain distance metrics are equally adept at estimating these gains. IMD and IDS appear to falter as predictors when individual factors vary (Figures 3b 4b while IDS seems reliable when all factors are simultancously altered (Figure 2b The average of all domain distances (mean-dist) appears more reliable than any of the individual metries (Figures 2b
alboit with a significant computational overhead.
K--N consistently outperforms both transferability and domain distance metrics when measuring relative transfer performance across all settings (Figures
2b
3b
4bb.
Considered together with its consistently k--N is clearly the most reliable
good ability to predict absolute transfer performance (iigures Figures transferability metrie.
2a
3a
4a
Why existing metrics fail to meet all criteria? Our analysis concludes that, aside from N-NN, there is no existing metric that consistently satisfics the criteria for predicting transferability. While transferability metries are generally capable of modeling absolute transfer performance, they tend to be unreliable when the experimental setting changes. This inconsistency is perhaps unsurprising, given that existing transfcrability metrics primarily concentrate on individual factors within transfer learning. Most are designed to consider variations in task and architecture. It is then perhaps not surprising that these metrics may fall short when confronted with changes in domain distance. On the other hand, domain distance metrics are explicitly crafted to gauge data similarity. This design inherently limits their ability to accommodate the task or the architecture.
On the success of K-NN transferability metric Throughout our experiments, k-NN demonstrated the ability to reliably predict absolute and relative transfer performance. It performed consistently well when considering the factors (domain, task, architecture) both in combination and in isolation.
But why is K-NN so effective? To begin with k-NN classification Fix 1985 is an optimization-free approach that measures separability in the target domain using pretrained weights from the source domain. The intuition is that, if the source-pretrained weights already do a good job of separating the target classes, even before they are adapted using fine-tuning, there is a good chance the knowledge transfer will be beneficial. k-NN is cheap proxy for performing the fine-tuning on the target task mask/domain - which makes it directly useful as transferability metric. Furthermore, the simplicity of the method may add to its reliability, as other transferability metrics use approaches in a similar spirit but are sensitive to hyperparameters. In addition, A-NN shares desired properties of other prominent transferability metrics, such as GBC and N-LEEP. However, A-NN eliminates the need for pre-defined heuristics, such as the assumption of a Gaussian distribution in the case of GBC, Furthermore, unlike N N-LEEP, it does not require any form of training on the target data to assess transferability.
Related work Estimating the potential benefits of transfer learning is not trivial. While we advocate that transferability metric must be simultancously robust to domain shifts, task changes, and different architectures several prior works have identified these individual factors as important (Yosinski et al. 2014 Azizpour et al. 2015 Kornblith et al. 2019). There is a large body of work on measuring distance between domains. Many approaches involve fitting distributions to data samples and then comparing the distributions, such as Fréchet Inception Distance (Heusel et al. 2017 Fréchet 1957), Kernel Inception Distance (Bińkowski et al.) 2018 Gretton et al. 2012), and Intrinsic Multi-scale Distance (Tsitsulli et al.) 2019). . While these works focus solely on the problem of measuring domain distance and ignore the problem of predicting transferability, some
others do consider transferability, including Earth Mover's Distance (Cui et al. 2018 Rubner et al. 2000),
Image Domain Similarity (Mensink et al. 2021), and Optimal Transport Dataset Distance (Alvarez-Meli
& Fusi, 2020). Other works have focused on the impact of the architectural and task differences when
10

---

Published in Transactions on Machine Learning Research (10/2025)
2020 and its variant (Song et al. 2019). model task differences assuming there exits trained model for each task. Other works such Taskonomy (Zamir et al. 2018) and Task2Vec (Achille et al. 2019) require re-training in order to model task relations. Others, like NCE (Tran et al., 2019) and LEEP Nguyen et al.
employing transfer learning. Some of them such as DEPARA (Song et al.
2020) model the relation between also consider the similarity of the domains but requires learning of auxiliary tasks, making less practical. Finally, several prior works have explored predicting transferability of pretrained models, focusing on the effect of architectural differences within transfer learning. These include TransRate, LogME, GBC, N-LEEP, H-score, SFDA, ETTran, NCTI, PACTran, PARC, TMT, LFC, and EMMS.
source and target. labels, and only a lew, such as OTCE (Tan et al. 2021
Contributions w.r.t previous works Our work brings important contributions on top of relevant prior works. Unlike prior works, we explicitly consider and analyze the individual factors transferability, offering deeper insights into metric robustness. We introduce RTP and emphasize signíficance in additional absolute transfer performance, revealing new insights. Our broader evaluation includes a comprehensive set of domain distance metrics, showing instances where these metrics surpass transferability metrics (Figure 3b We consider significantly more metrics (24) compared to all existing works: 6 in Agostinelli et al. 2022 in Bolya et al. (2021), and 7 in |Rngggli et al. (2022). We demonstrate k-NN's consistent performance across various transfer learning scenarios, along with its robustness w.r.t. and superior performance-runtine trade-off compared to all other metrics.
6 Conclusion
In this paper, we established the criteria for reliable transferability metric, considering domain distance, target task complexity, and architectural differences. Assessing 23 existing metrics in this context, we observed that no existing single metric consistently performs well, or outperforms others. Metrics tailored for a specific transferability factor perform well when that factor changed isolation but struggle with varied factors, highlighting that good metric should account for all factors be useful. We proposed k-NN as simple, robust alternative. k-NN proved effective in asscssing the performance gains from transfer, and emerged the best metric when considering all factors. It is cheap, optimization-free and provides interpretable scores making it an attractive choice for in practice.
Acknowledgements. This work was supported by the Wallenberg Autonomous Systems and Software Program (WASP) and MedTechLabs (MTL). We acknowledge the Berzelius computational resources pro- vided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre and the the computational resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Rescarch Council through grant agrecment no. 2022-06725.
Impact Statement
This paper aims advance the field of machine learning. One potential socictal impact is improving resource efficiency in AI deployment, redueing the need for extensive computational experiments and lowering the environmental footprint of large-scale model training. While our focus is methodological, it can contribute to sustainable Al development by reducing resource requirements.
References
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pictro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF international conference on compuler vision, pp. 6430 6439, 2019.
Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metrics evaluations? In Computer Vision-ECCV 2022: 17th European Conference, Tel Avie, Israel, October 23 23-27, 2022, Proceedings, Part XXXIV, pp. 303 321. Springer, 2022.
11

---

Published in Transactions on Machine Learning Research (10/2025)
David Alvarez-Melis and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428-21423 2020.
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. Factors of transferability for a generic convnet representation. IEEE transactions on pattern analysis and machine intelligence, 38(9):1790 1802, 2015.
Yajic Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Lconidas Guibas. An informationn-heorelie approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2309-22 2313. IEEE, 2019.
Anil Bhattacharyya. On measure of divergence between two multinomial populations. Sankhyū: the indian journal of statistics, pp. 401 406, 1946.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint 1rx::1881.00.p1 2018
Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning. Advances in Neural Information Processing Systems, 34:19301-1999 193122, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervieed vision transformers. In Proceedings of the TEEE/CVF international conference on computer vision, pp. 9650 9660, 2021.
Mircca Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4109-4118, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pottern recognition, pp. 248 255. Ieee, 2009.
Jia Deng, Alexander C Berg, Kai Li, and Li Fei-Fei. What does classifying more than 10,000 image categories tell us? In Computer Dision-ECCY 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, Seplember 5-11, 2010, Proceedings, Part V 11, pp. 71 84. Springer, 2010.
Aditya Deshpande, Alessandro Achille, Avinash Ravichandran, Hao Li, Luca Zancato, Charless Fowlkes, Rahul Bhotika, Stefano Soatto, and Pietro Perona. A linearized framework and a new benchmark for model selection for fine-tuning. arXie preprint (XXi:2102.00088, 2021.
Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks. In European Conference Compuler Vision, pp. 252-26010 268. Springer, 2022.
DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-4555 1982.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pollern recognilion workshop, pp. 178 178. IEEE, 2004.
Evelyn Fix. Discriménatory analysis: nonporametric discrimination, consistency properties, volume 1. USAF
school of Aviation Medicine, 1985.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Maurice Fréchet. Sur la distance de deux lois de probabilité. Comptes Rendus Hebdomadaires des Seances de L Academie des Sciences, 244(6):689 692, 1957.
Tomer Galanti, Lior Wolf, and Tamir Hazan. A theoretical framework for deep transfer learning. Information and Inference: Journal of the IMA, 5(2):159 2091 2016.
Pascal Germain, Alexandre Lacasse, François Laviolette, and Mario Marchand. Pac-bayesian learning of linear classifiers. In Proceediiggs of the 26th Annual International Conference Machine Learning, pp. 353-360 2009.
Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etran: Energy- based transferability estimation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 18613 18622, 2023.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773) 2012.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance imagenet classification. In Proceedings of the IEEE international conference computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4918-4927, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 9729-9738, 2020.
Byeongho Heo, Sangdoo Yu,, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF /CF International Conference on Computer Vision, pp. 11936-11945, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger, Densely connected convolutional networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly casy transferability cstimation. In International Conference on Machine Learning, pp. 9201-9225. PMLR, 2022.
Sohier Dane Karthik, Maggie. Aptos 2019 blindness detection, 2019. URL https://kaggle.comm
Maurice Kendall. A new measure of rank correlation. Biometrika, 30(122):81-9)) 93, 1938.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Proc. CVPR workshop on fineggrined visual categorization (FGVC), volume Citeseer, 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar preprint 1412:412.600 6980, 2014.
13

---

Published in Transactions on Machine Learning Research (10/2025)
Emir Komuk and Kevin Smith. An empirical study of the relation between network architecture and complexity. In Proceedings of the IEEE/CVF International Conference on Compuler Vision Workshops, pp. 0-0, 2019.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-26777 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
Yandong Li, Xuhui Jia, Ruoxin Sang, Yukum Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CV /CFF Conference on Computer Vision and Pattern Recognition, pp. 2663-2673, 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference compuler vision, pp. 10012- 10022, 2021.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprinl ar Xiv: Nix:1711.000 171.05100, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-gree tine-grained visual classification of aircraft. arXiv preprint 13K6:1300.5111 2013.
Christos Matsoukas, Johan Fredin Haslum, Moein Sorkhei, Magnus Söderberg, and Kevin Smith. What makes transfer learning work for medical images: Feature reuse & other factors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9225-92344 2022.
David A McAllester. Some pac-bayesian theorems. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 230, 1998.
Fanqing Meng, Wenqi Shao, Zhanglin Peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Foundation model efficient multimodal multitask model selector. orXiv preprint or arXiv:2308.0622 2023.
Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari. Factors of influence for transfer learning across diverse appearance domains and task types. arXiv preprint arXiu:2103 00..:188,,31,, 2021.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Cuong Nguyen, Tal Hassner, Matthias Soeger, and Cedric Archambeau. Leep: A new mcasure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp. 7305. PMLR, 2020.
Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using bhattacharyya class scparability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 9172 9182, 2022.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 3505. IEEE, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Karl Pearson. Vii. note regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240 242, 1895.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406 1406-1415, 2019.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Svetlozar T Rachev. The kantorovcc mass transference problem and its stochastic applications. Theory of Probability 8 Tls Applications, 29(4):4476676 1985.
Tija. Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces, In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10428 10436, 2020.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
Cedric Renggli, André Susano Pinto, Luka Rimanic, Joan Puigcerver, Carlos Riquelme, Ce Zhang, and Mario Lučié. Which model to transfer? finding the needle in the growing haystack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 9203-9214, 2022.
Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as a metric for image retrieval. International journal of computer vision, 40(2):99 121, 2000.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual recognition challenge. International journal of compuler vision, 115:211 252, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 4510 4520, 2018.
Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: predicting model transferability in self-challenging fisher space. In Europeon Springer, 2022.
Conference on Computer Vision, pp.
Jie Song, Yixin Chen, Xinchao Wang, Chengchao Shen, and Mingli Song. Deep model transferability from attribution maps. Advances in Neural Information Processing Systems, 32, 2019.
Jie Song, Yixin Chen, Jingwen Ye, Xinchao Wang, Chengchao Shen, Feng Mao, and Mingli Song. Depara: Deep attribution graph for deep knowledge transferability. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 3922-3930, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey Loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the incoption architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pallern recognilion, pp. 2818 2826, 2016.
Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-lask representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 15779 15788, 2021.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521 1528. IEEE, 2011.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Auh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. -495-1105, 2019.
Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Alex Bronstein, Ivan Oscledets, and Emmanuel Müller. The shape of data: Intrinsic distance for data distributions. arxiv preprint 1rXi::1005.111 11141, 2019.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911), 2008.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building a bird recognition app and large scale dataset with cilizen scientists: The fine print in fine- greeggraied dataset collection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 595-604 604, 2015.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection datasct. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 8769 8778, 2018.
Sebastiano Vigna. A weighted correlation index for rankings with ties. In Proceedings of the 24th international conference on World Wide Web, pp. 1166 1176, 2015.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 168-578, 2021.
Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre-trained models are from neural collapse on the target dataset informs their transferability. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 5549 5558, 2023.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-paraettric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733 3342-,72,22 2018.
Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfci Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data for performance evaluation of acrial scene classification. IEEE Transactions on Geoscience and Remole Sensing, 55(7):3965 33665-9-981,, 2017.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE compuler sociely conference compuler vision and IEEE, 2010.
pattern recognition, pp.
Saining Xie, Ross Girshick, Piotr Dollár, Zhnowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1492-1500, 2017.
Huiwen Xu and U Kang. Fast and accurate transferability measurement by evaluating intra-class feature variance. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 11474-11482, 2023.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained
models for transfer learning. In International Conference on Machine Learning, pp- 12133-12143. PMI.R, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXie:1605.0711 2016.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Disentangling task transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 3712-3722, 2018.
Taskonomy:
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1452- 1464, 2017.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Appendix for k-NN as a Simple and Effective Estimator of Transferability
Appendix overview. In this supplementary material, we provide more experimental details and results, as outlined below:
more details regarding the implementation and datasets used in this work.
the break-down of the results when domain and task complexity change together.
the break-down of the results when only the domain changes.
detailed results regarding when only the task complexity changes.
additional results regarding the computational cost of different metrics, the effect of ls in k-NN, along with performance of metries evaluated with different evaluation measures.
example visualizations of feature representations
concise mathematical formulations of representative transferability metrics
A More implementation details
For each dataset, either the official train/vallttest splits were used, or we made the splits following Kornblith et al. (2019). Images were normalized and resized 256 x 256, after which augmentations were applied: random color jittering, random horizontal flip and random cropping to 224 224 of the rescaled image. The Adam optimizer (Kingma & Ba, 2014) was used for CNNs and AdamW (Loshchilov & Hutter 2017 for
ViT-based architectures, and the training of models was done using PyTorch (Paszke et al. 2019 After a grid search, the pretrained and the maudomly-initialiane models were trained with a learning rate of 10 and 3 x 10-4 respectively, following an initial warm-up for 1,000 iterations. During training, the learning rate was dropped by a factor 10 whenever the training saturated until it reached a final learning rate of 10-6 3 10-6 for pre- trained or randomly-initialize, models respectively. The checkpoint with the highest validation performance was finally chosen for final evaluation.
B More details about the datasets
This section presents more details regarding the datasots used this work. Starting with the source domains,
IMAGENET [Deng et al. 2009 Russakovsky al. 2015 contains around 1.2M training and 50,000 validation image from 1,000 classes. The Pl.ccss665-Standar dataset (Zhou et al. 2017 used in this work contains
1.8M train and 36,500 validation images with 365 scene classes. iNat2017 (Van Horn et al. 2018) includes 675,170 train and validation images consisting of 5,089 classes of fine- fine-grained species. NABirds (Van Horn et al. 2015 consists of 555 fine-grained bird classes and has 48,562 images total.
Regarding the 12 target datasets utilized this study, Table 3 provides the details of each, encompassing the number of images, number classes, and their respective domain Finally, Figure illuutrrates an example of each target dataset.
C Additional results when domain and task complexity change
This section entails the detailed results for the scenario when both the target domain and task change from the source, corresponding Figure the main paper. In this setup, network is pre-trained on each of the four source datasets and subsequently transferred to each of the 12 target datasets. For each network,
17

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Summary of the downstream datasets used in this work.
Dataset
Reference
Domain/tack
Classes
Train size
Test size
Metric
StanfordDogs
Khosla et a.
2011
Fine-grained classification
120
12,000
8,580
'Top-
Aircraft
Map a..
Pine--ratee Fine-grained classification
6.667
Mean per class
NABirds
Van Horn et
2015
Fine-grained classification
555
23,929
21,633
TTpp1
Oxfordll1-Pet
Tarkhi et al. 020
Fine-grained classification
37
3,680
3,669
Mean per class
CIFAR-10
Krizhessky
2009
Superordinate apperordinate-evell classification
10
50,000
10,000
TTp-1
C-FAR-100
Krizhevsky et al.
2009
Superordinate-level classification
100
50,000
10,000
Top-1
Caltech-101
Fei-Pej et
20121
Superordinate-levy classification
3,030
5,617
Mean per class
Caltech-256
Grihia et al
2007
Superordiunte- classification
257
15,420
15,187
Mean per class
DTD
Chipoot
Texture classification
3,760
1,880
Top-1
SUN397
et
Scene classification
397
19,850
19,850
Top-1
AID
Xia cb al
Acrial imagery
30
5,000
5,000
Top-1
APTOS2019
2019
Medical imaging
3,113
549
Qudratic Kappa
Calteci-101
Dugg
Ox
NABirdo
Aiscraft
AID
CIFAR-100
CIFAR-10
APTOS2019
Figure 6: Overview of the downstream datasets used in this paper. These datasets cover superordinate-lveel object classification, fine-grained classification, scene recognition, texture classification, acrial imagery, and medical imaging.
we compute both the absolute and relative transfer performance, and then measure the correlation between these and the transferability metrics (including domain distance metrics). For each configuration, Tables and [ show the performance of different metrics in predicting absolute and relative transfer performance respectively. Figure Segureeshoww shows the inter-correlation of different transferability metrics. Figure shows the same for domain distance motries.
When predicting relative transfer performance (RTP), Table 5 shows that domain distance metrics are good predictors of the relative transfer performance (except for IMD).
However, the choice of the best distance metric is not consistent when the source dataset is different. Specifically, while EMD is the best domain distance metric when IMAGENET is the source domain (Tw: 0.68), IDS is the best when iNat2017 the source (Tw: 0.69), and FID the best when NABirds the source (Twi 0.74). This signifies that no single definition of distance works best all the time and the best distance metric could vary depending
the source domain. However, the differences in the distance definitions appears to be complementary. In fact, we see that "mean-dist", defined the average of distance estimates, performs average (Tw: 0.7) better than each individual domain distance metric, although at higher cost.
Additional results when only the domain changes
This section presents the detailed results regarding the scenario when only the domain distance changes the target task and network architecture remain the same. This corresponds to Figure 3 the main paper. In this setup, we use IMAGENET as the source domain and we fine-tune models on DomainNet (Peng al.) 2019). Net comprises six distinct domains, each comprising the same set of 345 categories. As
18

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Effectiveness of different metrics to predict absolute transfer performance when target domain and task complezity change. In this setup, a network is pre-trained on each of the four source datasets (shown the top row) and subsequently transferred each of the 12 target datasets, Quantitative correlations (Tw) of different metrics with absolute transfer performance in each configuration is shown below.
ImageNet
1Nac2017
Places365
NABirds
Average
TransRate
0.33
0.24
0.12
0.03
0.18
LogME
-0.38
-0.48
-0.55
-0.48
-0.47
GBC
0.60
0.72
0.67
0.70
0.67
A-LEEP
0.49
0.52
0.53
0.47
U.50
LEEP
0.54
0.69
0.64
0.69
0.64
H-Score
-0.40
-0.46
-0.42
-0.44
-0.43
NCE
SFDA
0.30
0.20
0.22
0.14
0.22
NCTI
0.39
0.66
0.59
0.59
0.56
ETran
0.47
0.55
0.59
0.49
0.52
PACTran
0.39
0.62
0.58
0.54
0.58
PARC
0.31
0.39
0.59
0.37
0.42
OТCE
0.65
0.33
0.66
0.69
0.58
LFC
0.43
0.57
0.43
0.43
0.46
EMMS
0.35
0.58
0.56
0.60
0.57
NumC
0.35
0.40
0.67
0.58
U.55
FID
-0.25
-0.24
-0.09
-0.21
-0.20
KID
-0.13
-0.25
-0.23
-0.22
-0.21
EMD
-0.33
-0.26
-0.29
-0.35
-0.31
-0.33
IMD
0.10
-0.23
-0,21
-0.31
-0.10
mean-dist
-0.22
-0.37
-0.29
-0.33
-0.30
K-NN
0.34
0.62
0.64
0.61
0.60
Table 5: Effectiveness of different metrics to predict relative transfer performance when largel domain and task complezity change. In this setup, a network is pre-trained on each of the four source datasets (shown in the top row) and subsequently transferred each of the target datascts. Quantitative correlations (Tw) of different metrics with relative transfer performance in each configuration shown below.
ImageNNtt
iNac2017
Places365
NABirds
Average
TransRate
0.36
0.29
0.27
0.51
U.36
LogME
-0.14
-0.18
-0.20
-0.01
-0.13
GBC
0.71
0.68
0.66
0.70
0.09
A-LEEP
0.46
0.17
0.18
0.54
0.11
LEEP
H-Score
0.32
0.35
0.17
0.52
0.34
0.74
SFDA
0.00
-0.11
-0.27
-0.14
-0.13
NCTI
0.21
-0.03
-0.18
0.01
0.00
ETran
0.69
0.66
0.75
0.65
PACTran
0.75
0.72
0.71
0.77
0.74
PARC
0.46
OТСE
0.73
0.69
0.68
0.70
0.7
TMI
-0.05
0.54
0.28
0.10
0.22
LFC
-0.03
0.02
-0.25
0.12
-0.04
EMMS
0.52
0.49
0.37
0.53
0.48
NumC
-0.67
-0.66
-0.63
FID
0.39
0.64
0.39
0.74
U.59
KID
EMD
0.68
0.57
0.43
0.67
0.59
0.65
0.69
0.65
0.60
0.65
TMD
0.23
0.20
0.11
0.19
0.18
mean-dist
0.69
0.71
0.59
0.81
0.70
K-NN
0.95
additional experiment, we set each individual domain, within DomainNet, the source and use as targets the remaining domains. For each configuration, Tables 0 and show how well different metrics perform when predicting absolute and relative transfer performance respectively.
Considering absolute transfer performance, Table demonstrates that k-NN (Twi 0.65) appears be the best metric, followed by LogME (rw: 0.63) and H-score (Tw: 0.63). Unlike the previous setting, GBC (Tui 0.36) is longer good metric in this setting. When predicting relative transfer performance, Table shows
19

---

Published in Transactions on Machine Learning Research (10/2025)
IttgeNNt
1N11
Placca365
NABin
MDD
IDS
IMD
F1D
KI0
MD
10$
IMD
RID
EDD
ID$
IMD
FID
KiD
EDD
ID5
IMD
1.0
0.94 0.91
0.35
0.15
FID
1.0
0.97 0.88
0.42
0.14
1.0
0.87 0.89
0.47
0.12
FID
1.0
0.78 0.73
0.6
0.18
KID
0.94
1.0
0.84
0.34
0.2
KID
0.97
1.0
0.9
0.38
0.12
KID
0.87
1.0 0.83
0.58
0.07
KID
0.78
1.0
0.76
0.28
0.1
EMD
0.91
0.84
1.0
0.41
0.2
EMD
0.88
0.9
1.0
0.28
0.21
EMD
0.89
0.83
1.0
0.47
0.18
EMD
0.73
0.76
1.0
0.29
0.22
IDS
0.35
0.34
0.41
1.0
0.08
IDS
0.42
0.38
0.26
1.0
0.06
IDS
0.47
0.58
0.47
1.0
0.25
IDS
0.6
0.28
0.29
1.0
-0.0
IMD
0.15
0.2
0.2
0.08
1.0
IMD 0.14 0.12
0.21
0.06
1.0
IMD
0.12 0.07
0.18
0.25
1.0
IMD
0.18
0.1
0.22
-0.0
1.0
Figure 7: Cross correlations of different domain dislance definilions. The distances are calculated from each source (shown on top of each panel) to 12 target datasets.
Table 6: Effectiveness of different metrics to predict absolute transfer performance when only the target domain changes. In this sctup, IMAGENET is chosen the source and fine-tuning is done on DomainNet Peng et al. 2019 Additionally, we set each individual domain within DomainNet as the source (shown in the top row) and use the remaining domains as largets, resulting in a total of seven configurat ions. Quantitative correlations (Tw) of different metrics with absolute transfer performance in each configuration shown below.
ImageNet
Clip.
Info.
Paint.
Quick.
Real
Sketch
Average
TransRate
-0.40
0.00
-0.15
-0.05
LogME
0.87
0.74
0.11
0.74
0.51
0.65
0.81
0.63
GBC
-0.37
N-LEEP
0.36
0.39
0.20
0.45
0.33
0.38
0.67
0.40
LEEP
0.33
0.20
0.16
0.45
0.68
0.31
0.67
0.40
H-Soore
0.72
0.53
0.30
0.67
0.74
0.63
0.81
0.63
NCE
0.33
0.20
0.20
0.15
0.68
0.31
0.67
0.41
SFDA
NCTI
0.70
0.61
0.05
0.26
0.35
0.78
0.29
0.43
0.35
PACTran
0.05
0.10
-0.28
0.11
0.52
0.34
0.55
0.20
PARC
0.28
-0.11
0.40
0.76
0.76
0.87
0.40
0.48
ОТСE
0.26
-0.16
0.46
0.53
0.27
TMI
-0.19
-0.46
-0.31
-0.45
0.32
-0.17
-0.55
-0.31
LEC
EMMS
0.61
0.59
0.20
0.60
0.74
0.46
0.79
0.57
-0.02
0.45
0.02
0.24
KID
0.37
-0.03
0.06
0.15
0.50
0.21
0.24
0.26
EMD
0.19
-0.19
-0.16
0.32
-0.12
0.21
0.17
0.06
IDS
-0.17
-0.38
0.15
-0.18
-0.05
IMD
0.26
-0.42
-0.54
-0.41
0.63
0.26
-0.34
-0.08
0.01
0.29
ENNN
0.64
0.55
0.20
0.82
0.79
0.71
0.84
0.65
that k-NN (Tw: 0.59) is again the strongest metric on average, followed by mean-dist (Tw: 0.55) and EMD (Twi 0.54). In contrast to the previous sctting, PACTran (Tw: -0.01) shows negative correlation on average. Overall, these result further confirm that the choice of the best transferability metric is not consistent and changes depending the experimental setup. Further, it suggests that k-NN matches outperforms the best metric in cach setting.
E
Additional results when only the task complexity changes
This section presents the detailed results corresponding to the case of changing only the target task the domain and architecture remain the same. This corresponds to Figure in the main paper. Using IMAGENET as the source, we apply transfer learning to subsets of NABirds, SUN397, and Caltech-256. More specifically, we vary the task complexity by varying the number classes (as a proxy) in accordance with Deng et al. (2010); Konuk & Smith (2019). In detail, we soquentially remove classes from the target datasct to create target tasks of varying complexity. For each configuration, Tables and I display the performance of different. metrics when predicting absolute and relative transfer performance respectively.
20

---

Published in Transactions on Machine Learning Research (10/2025)
ImageNet
1Nat2017
032
100 2020
132
4445
323
000
322
0:95
4.30
-.89
0.4
031
133
0.50
330
6344
4.22
61
4:35
126
469
0.30
-333
692 16
64
43
4.
0.4
-335
442
-35
-333
CUE
037
4.22
3.55
6:22
08.5
-375
6004
10
0.37
-134
3.26
4.3L
0.03
6.15
4.00
61
600
4.22
332
457
4.06
-223
Toovoo 614 43
10
64X
2:24
11
511
11
EMA
0.22
0.22
0.64
10
545
0.02
4:11
0:01
0.55
5.0
0.14
JKLE -30
143
13
18
02
35
03
0/22
0.3
1/0
0.34
0.72
932
0.00
-333 4034
0.45
3.0
073
1/4
0.50
0.78
0411
032
0:49
0.34
933 042
0.51
0.24
66 026
01
0.6
0.22
.6.3
0.34 0.34
000
0.43
02
0.56
027
67
0.54
214, 6-12 00.55 0:1 6/9 6-37 14 647
0.27 6:44 622 0.34 0.24
1.6 0:97 0.34
S0A 0331 5:00
09/2 0:46 0:11 0.89
0.04
0:07 0.84 021 0:4
0.09
0.54
6:97
0.24
MAC 0.45 -044 1.55 03 0.45 12 111
10
0.43 955 038 0.37
0.42
9.53
6.32
0.54
053
0.7
PNNC 0335 4056 0.53 9:4 0.5 041 0.44
3.0
0:35 0.56 033
0.0
0.55 044
3:50
0:43
0.43
8.40
FATH -0.2 -25 034 6.2 0.50 0.44 0:2 649
8.8
0.03
9.03
0.37
878
PACH -374 46.5
0.32 0:28 0.26
0.43
9.2
1.0 0.49 031
076
0.31
6.41
6BC
ATIN 0.27 942 669 6.79
0.12
9.A7
0:93
0.24
sTav 605
0.53
0.75
deTe 644 00.5 5M D/M 5.37 1M
642 0.97
5.59
9:91
0.34
0:41
OTre 1. 0:22
0.37 0:4 D01
0:14
0-4
0.29
TM 6.12 00.95 024 1:6k 0:44
0043 0.8
3.4
5:34
0.41 0.22
-28
1414 0:2 0.00 0.27
0.44
0.22
938 0.44 6.70
032
0.37
9.45
1.0
0.89 937
UC .032
0.38
08 0.75
0.31
033
3.8
0.97
AM 6000 034
144 0.00 0.32
0.
587
0.73
9.40
0.35
0.23
132
1-M4 -122
4.38
031
1
1.40
02
0.54
823
1.00
0.51
616
P 365ces365
NABirds
857
2.58
Treskox 0:59 0.37
1.0 0:22 8:35 88
925
800
44 932 D2T 012 07 8.1
6:32 0.7 923
6.00
Teeskee
6.72
3.0
5:7 0.45 8:30 03 246
637 0.99
022
2.45 0.03 586 435 0:57
-625
-0.3
332 13
6.75 0.3 34 661 033
60 G.77 86
0.44 3.37 6.73 0.77 844
6.70
-236
-606
0.37
3.0
0.56 3.55 055
1.21
6/47 0.53 B82 007
3.4 626
3.06
0.55
0.25
154
00.66
3.6 6.76 10 0.7 0.41 663 0.2 085 6/6
4.51 6.67
08
606
-272
6.58
+6.99
555
14
0.54
0.95 032 0.41
0.22 5M
5.71 6.71 0.74
-04 -0.2 -382 0/22 0.54
0.55
932 0.50
0.59
9.50
0.43 0.53
AM6--342 -0.4
-0.45
027 0.8 0.5L
0.23
0.59
055
1.40 0.43 0.52
FACTee 664 40.2 66 6-28 6:62
654
0.5
3.66
5.04 0.23 065
PACher 0.22
00.77 0.43 100
0.20
057
048
0/6 6.34 0.54
SBC 055 .0.35
932
01 0.55 0 0. 6772 0.39 3.0 0.76 88
0. 9.00
8:13
0.32 0:3
0.00
0:48
.0.9 1.3 0.05 0.0 0.43 9:00 0.41 34 038 0:22 0.77 8:88
5.76 0.54 0.73
MCT -622 4002
037 6.77 0.75 0:3 3:20 050 0.14 875
L3
0.35
9.30
0.5L
0.23 834
NCI -3.00
004
01 8802 032 0.92
0.23
0.24
0.57 0.88
0.74
0.37
032
3.00 0.61
OFCE
098 1171 055 0:51 8.57 659 0,59 OK 0:55
3.5
0111
0.58
0.14
00955 0.54 0.66
0.43 0.78
0.85
1.0
0.51 0.88 0.34
LIC -600 0.55 932 672 0.57 0.0 330 649 033 363 0.5 0.75 0.09
9.20
1.0
0.58
62
6.72
LKC 40 .030
-0.5 8556 0.72 075 0.46 1/4 6.53 0.76 88 6.75 3.53 855
3.8
0.0
84
852
CMNE (656 46.5 012 6.71 88 037 664 0.3 512 6/2 0.27 0.56
10
DNC -331 6.23 4616 066 6.79 3.06 0.08 6.34 0.4 064 6:56
062
18
NAYC 1223
059
Movt
029
Figure 8: Inter-correlations of different transferability metrics. The transferability scores are calculated from each souree (shown on top of cach pancl) to all the target datasets. Evidently, some metrics demonstrate stronger correlations than others. For instance, LogME and H-score display high correlation with each other but a low correlation with the other metrics. GBC, A-LEEP, and N-NN exhibit strong correlations with one another.
When predicting absolute transfer performance, NCE, LEEP, A-LEEP, GBC, ETran, and PACTran emerge as the top performers (Tm: 0.98 on average). k-NN proves to be an ive estimator of transferability in this setting, and performs on par with the best transferability metrics (Tw: 0.98 on average).
When predicting relative transfer performance, NCE. LEEP, GBC, NCTI, ETran, and PACTran perform best among the transferability metrics (Tw: 0.93 on average), and FID and mean-dist show similar performance
21

---

Published in Transactions on Machine Learning Research (10/2025)
Table Effectiveness of different metrics to predict relative transfer performance when only the target domain changes. In this setup, IMAGENET chosen as the source and fine- tinettnnngg is done on DomainNet (Peng et al. 2019). Additionally, we set each individual domain within DomainNet row) and use the remaining domains as targets, resulting in total of seven configurations. Quantitative correlations (Tw) of different metrics with relative transfer performance in each configuration is shown below.
the source (shown the top
ImageNet
Clip.
Info.
Paint.
Quick.
Real
Sketch
Average
TransRate
0.20
0.21
0.28
0.10
0.08
0.14
0.11
0.16
LogME
0.71
GBC
0.08
0.44
0.43
0.52
-0.19
0.08
0.39
0.25
.-LEEP
-0.10
-0.43
-0.14
-0.04
0.19
-0.93
-0.43
-0.14
LEEP
0.45
-0.07
0.49
0.46
0.34
0.54
-0,09
0.30
H-Score
0.26
0.16
0.41
0.34
0.31
0.47
0.24
0.31
NCE
0.24
-0.15
0.41
0.34
0.47
0.42
-0.09
0.23
SFDA
-0.25
-0.24
-0.09
-0.14
0.21
-0.18
-0.39
-0.15
NCTI
ETran
-0.14
-0.20
-0.04
0.51
0.57
0.18
0.46
0.19
0.05
0.38
-0.44
-0.08
PARC
0.15
0.19
-0.46
-0.47
0.45
-0.32
0.19
-0.04
OTCE
-0.48
-0.66
-0.50
-0.54
0.14
-0.52
-0.67
-0.46
TMI
0.41
0.47
0.24
0.27
LEC
0.24
-0.13
0.49
0.22
0.39
0.28
-0.07
0.2
EMMS
FID
0.30
0.49
0.64
0.46
0.45
0.76
0.55
0.52
0.51
0.46
0.44
0.72
0.55
EMD
0.49
0.60
0.40
0.15
0.60
0.54
IDS
0.45
0.46
0.32
0.10
0.11
0.28
0.10
0.26
IMD
0.50
0.07
mean-dist
0.49
0.58
0.67
0.56
0.33
0.61
0.59
0.55
K-NN
0.62
0.51
Table 8: Effectiveness of different metrics in predicting absolute transfer performance when only the targel task complerity changes. In this setup, IMAGENET used as the source, and finc-tuning is done subsets of NABirds, SUN397, and Caltech-256, where each subset consists of varying number of classes (see main text for details). Quantitative correlations (Tw) of different metrics with absolute transfer performance in each configuration shown below.
NABirds
SUN397
Caltech-256
Average
0.58
LogME
-0.35
-0.23
-0.14
-0.24
GBC
1.00
0.97
0.96
0.98
A-LEEP
1.00
0.97
0.96
0.98
LEEP
1.00
0.97
0.96
0.98
H-Score
-0.71
NCE
1.00
0.97
0.96
0.98
0.97
0.95
NCTI
0.92
0.97
0.61
0.83
ETran
1.00
0.97
0.96
0.98
PACTran
1.00
0.97
0.96
0.98
PARC
0.88
0.82
0.83
0.8
OTCE
TMI
1.00
0.97
0.96
0.98
1.00
0.97
0.97
0.98
EMMS
0.99
0.98
0.97
0.98
NumC
1.00
0.97
0.96
0.98
FID
-0.97
-0.91
-0.88
-0.92
KID
-0.79
-0.68
-0.88
-0.78
IDS
0.54
0.30
0.29
0.38
IMD
0.85
0.76
-0.48
0.38
mean-dist
-1.00
-0.91
-0.88
-0.93
A-NN
1.00
0.97
0.96
0.98
(Tw: 0.93 on average). Notably, H-score and LogME show negative correlation when predicting both absolute and relative transfer performance in this sctup, indicating their inability to account for task complexity. k-NN shows strong performance (Tw: 0.93 on average), on par with the best transferability and domain distance metries. Overall, these results show that k-NN can successfully account. for task complexity well.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Table 9: Effectiveness of different metrics for predicting relative transfer performance when only the target task complexity changes. In this setup, IMAGENET is used as the source, and finettnnigg done on subsets of NABirds, SUN397, and Caltech-256, where each subset consists of varying number of classes (see text for details). Quantitative correlations (Tw) of different metrics with relative transfer performance in each configuration is shown below.
NABirds
SUN397
Caltech-256
Average
TransRate
-0.63
-0.68
-0.77
-0.69
LogME
-0.42
-0.35
-0.18
-0.32
GBC
0.89
0.93
0.97
0.93
A-LEEP
0.71
0.86
0.97
0.85
LEEP
0.89
0.93
0.97
0.93
H-Score
0.11
-0.01
-0.14
-0.01
NCE
0.89
0.93
0.97
0.93
SFDA
0.69
0.44
0.83
0.65
NCTI
0.89
0.94
0.96
0.93
ETran
0.89
0.94
0.97
0.93
PACTran
0.89
0.93
0.97
0.93
PARC
0.68
0.60
0.53
0.6
ОТСE
0.88
0.94
0.97
0.93
TMI
0.63
0.40
0.01
0.35
LFC
-0.75
-0.81
-0.60
-0.72
FMMS
0.84
0.89
0.94
0.89
NumC
-0.77
-0,85
-0.94
-0.85
0.89
0.93
0.97
0.93
KID
0.86
0.88
0.97
0.9
EMD
0.89
0.90
0.97
0.92
IDS
-0.58
-0.55
-0.47
-0.53
IMD
-0.69
-0.75
0.27
-0.39
mean-dist
0.89
0.93
0.97
0.93
k-NN
0.89
0.93
0.97
0.93
All together, these findings highlight the capability of k-NN as a robust transferability metric, which can successfully account for crucial factors of transferability better than all the current transferability metrics from the literature.
F Further ablation studies
In this section, provide additional results when using evaluation measures, such as Kendall's (Kendal) 1938 Pearson's p (Pearson] 1895). and Rel@1 (Li et al. 2021 (Figure 10 We further include ablation
studies on the transferability metrics' total runtime (Figure 9a and robustness of k-NN to the choice of k (Figure 9bl.
G A theoretical formulation for transferability measures
For more unified understanding, we can frame transferability measures in terms of probably approximately correct (PAC) learning. Our goal is not derive tight generalization bounds for the measures in the literature but provide theoretical framework to formulate the problem for better insights.
Supervised learning aims to approximate the true conditional distribution, Pr, with a Icarned distribution Pr, usually by minimizing the KL-divergence between them, (xr.(Pr|(Pt) whereas in transfer learning we minimize the divergence, (Prl|P]|Prss starting from a source-prete pretrceeee model, Pris Here, we slightly abuse the notation with 5 indicating pretrained model and not just the pretraining datasct. Unlike NCE Tran al.) 2019) we are not limiting the problem to setting where the input instances are shared and only the conditional distributions differ across tasks.
23

---

Published in Transactions on Machine Learning Research (10/2025)
PAC interpretation of transfer learning task for small and 5 can be given as;
P(Jerr(Pris) - errg - inf err(h)] e) 2 1 -
(1)
bES
where errn is the irreducible Bayes error of the target distribution and S indicates the hypothesis class of model. Note that errb is sometimes omitted in
target models initialized with a given source-pretraing PAC formulation since it is generally not feasible to achieve, Le. (xz(PrllPrio * O. We include it here to highlight the conceptual distinction between the goals of transfer learning and the notion of transferability, as discussed below.
In the strictest sense PAC addresses all possible target distributions and the infinum operator is blind to the training method thus different initializations has no impact on the hypothesis class. PAC framing simply states the aim of transfer learning as the estimation of the true data distribution, Pr, i.e. approximation of Bayes error, eTTB, ideally without using many samples (or low sample complexity). Instead, the concept of non- non-trivial transfer introduced in Galanti et al. (016)) together with empirical observations that protrained modcls require lewer labeled examples to perform well (Azizpour et al. 2015 Raghu et al.) 2019 Matsoukas et al., 2022), support the hypothesis that for the subset of target distributions relevant computer vision, a randomly initialized model typically requires more samples to achieve error rates comparable to a well-initialized model using pretrained weights.
The aim of transferability methods is to predict how well a tuned model will perform, i.e. to provide an approximation of a fine-tuned model's error, ideally without training said model. In other words, instead of estimating the true distribution, e.g. using empirical risk minimization based standard deep learning methods, such that the learned model approximates the error of a (hypothetical) Bayes optimal model, we aim estimate given fine-tuned model's error, err(Pr(s). Following similar PAC formulation as transfer learning,
P((err(for(S)) (rr(()ss) inf mr ( 2 6
(2)
mEM
where (S) indicates the transfcrability error from source domain S to target task T and M is the model family used for estimating transferability. It would be safe to assume that M should be simpler hypothesis class than S.
This framing formalizes the intuition behind the standard empirical evaluation approaches, i.c. to calculate a transferability measures' correlation (or anti-correlation to downstream accuracy. After all, the best transferability measure the transfer learned model itself. Unfortumately, to the best of our knowledge, there are no tight generalization bounds proposed in the literature for this setting, and even in a limited problem setting TTan et al.) 2019) bounds require strong assumptions.
Regardless, even without theoretical bounds, PAC framing of transferability can inform our understanding of various approaches in terms of how tightly they can approximate the transferred model. Below, we compare and contrast A-NN with its most similar metrics, GBC and N-LEEP, to understand why it may offer a better estimate of transferability. We examine these metrics through the lens of estimated error. Transferability is naturally inversely related to the estimated error.
GBC aims to measure class separability an indicator of classifier performance. They propose fitting Gaussians to the target classes in the souree embedding space. They measure the Bhattacharyya distance between these larget classes as measure of their separability, using this indicator for how transferable the source model is. This separability also be understood the error rate of a (Gaussian) naive Bayes classifier. For simplified binary problem, the crror of naive Bayes classifier is given by,
eTGG = P(Y = 1) / 1 p(z|Y 1)dx
1)dx + P(Y = P0 St tt00
p(x|Y = 0)dr
where the decision boundary B dircctly controls the error rate and easy to see that low Bhattacharyya distance between two classes leads B to be determined with more entangled classes, yielding high error rate.
24

---

Published in Transactions on Machine Learning Research (10/2025)
0.75
DPMA
LSB
0.70
PNRC
0.35
SPOA
Averae
0.00
0.55
0.50
300
200
260
254
Rumning
(a) Run-time transferability metrics.
(b) Performance of k-NN w.r.t. different values of k.
Figure (Left) comparison of run-time against performance for different metrics. k-NN to the choice of k.
(Right) robustness of
X-LEEP is explicitly defined a classifier. This method a bit more relaxed than GBC the sense that the N-LEEP measure, i.e. log error rale of this classifier, based not single Gaussian per class but Gaussian mixture model in the source embedding space. We can intuitively understand N-LEEP as a more expressive hypothesis family than naive Bayes classifiers.
k-NN is our proposed method, which climinates the need for Gaussian or mixture of Gaussian assumptions. Instead, directly leverages the inherent distribution of features extracted from the source model for classification, making it inherently more flexible and expressive.
Fine-tuning is the gold-standard classifier but is computationally expensive and impractical for real-world use. Tt the most flexible approach, as enables network weights adjust to the target distribution.
As an additional advantage of using an expressive yet computationally efficient non-paramotric classifier like A-NN, we note that k-NN can be more robust to outliers compared to Gaussian or GMM models. Notably, k-NN serves as a Bayes-optimal classifier for uniform distributions, whereas (Gaussian) naive Bayes is naturally optimal for normal distributions [1]. However, drawing definitive conclusions based on these theoretical properties may not entirely meaningful, neither assumption can be reliably validated in the high-dimensional representation spaces of deep learning models. Instead, empirical observations should guide further research in this area.
H Feature visualization
In this section, we present t-SNE (Van der Maaten & Hinton 2008
visualizations of feature reprosentations P pre-trained on ImageNet, evaluated across
extracted from the penultimate layer of a ResNet-50 model
several downstream tasks. The results, shown in Figure 10 illustrate that higher k-NN transferability
estimates est(P) indicate higher transfer performance per ((P).
Mathematical formulations
In this section, we present concise mathematical formulations of several representative metrics of transferability considered in this study.
NCE (Tan al. 2019) assumes the source and target datasets share the same input examples. It treats the source and target labels as random variables and uses their conditional entropy quantify how much
25

---

Published in Transactions on Machine Learning Research (10/2025)
0.
64
6.1
up
Hokore
Trenslat
coe
4ro
arca
(a) Evaluation measure: Kendall's
(b) Evaluation measure: Kendall's
640
Transees
Logeg
dacs
Leee
Kourr
Evaluation measure: Pearson's p.
(d) Evaluation measure: Rel@1.
Figure 10: Transfcrability metrics evaluated using common correlation cocfficient measures.
CIFAR-10
Oxford-Pet
AID
DTD
Transfer perf.: 0.96
Transfer perf.: 0.91
Transfer perf.: 0.89
Transfer perf.: 0.68 A-NN: 0.61
k-NN: 0.82
k-NN: 0.90
K-NN: 0.74
Table 10: t-SNE visualizations of feature representations from the penultimate layer of a ResNet-50 model P pre-trained on ImageNet, evaluated on various downstream tasks. Each color represents distinct class. For each task, we report the k-NN transferability estimate est(P) prior to fine-tuning, alongside the absolute transfer performance per ((P) after fine-tuning. Tasks with higher k-NN estimates have better transfer performance.
information the source task label provides about the target task label. It considers single sequence of training samples X = (I1, I2, Zn) E X",, along with two corresponding label sequences Y = (31 ....... Vn.) E ) and 7 = (21, ....., 2n) E 2", whore y: and 2; are labels assigned to I; under two different tasks: the source task Tz = (X, 2) and the target lask Ty (X,Y). Here, denotes the domain of the values of X, while y and represent the sets of unique labels in and Z, respectively.
Transferability between the two tasks is estimated by computing the conditional entropy (CE) between the two label sequences. First, from the label sequences Y and Z, the empirical joint distribution P(y, 2) for all (v. 2) E y x Z is computed as:
1
P(u. 2) 1i 9: y and 2; =)1,
where |-| denotes the cardinality of the set. Then, the definition of CE between two random variables Cooer 1999) is employed to compute the CE between the two label sequences Y and Z:
P(y, 2) H(Y 2) = ΣΣ 2(y,2) 2)) log
P(z))
2E2
26

---

Published in Transactions on Machine Learning Research (10/2025)
where P(2) is the empirical marginal distribution over 2, given by:
1 { : 2, = 23/. n2
P(=) P(y.2) NEV
Since lower conditional entropy indicates better transferability, the negative conditional entropy, -YY 2), is used as the final transferability metric for predicting transfer performance.
Log Expected Empirical Prediction (LEEP) Nguyen et al. 2020 estimate transfcrability of source model 0 pre-trained on classification lask with outpul label space E to largel classification task with dataset D = {<11,3)) (æ2, . (In, Bn)), where I; X' and E y. For each target input Zi, the model outputs a softmax distribution 0(z,,, where 0(z;) denotes the probability of assigning I; to class 2 E Z. First, the empirical joint distribution between source and target labels is computed as:
P(u,z)
Σ 0(1)22. 1
where summation Eimmy indicates that we sum over all indices E (1. Ys equals This is then used to compute the empirical conditional distribution:
,n} for which the target label
P(y,2) 1(y12)= Ever 2))
For any input instance X, a label y is predicted by first sampling 0(x), then y P(yl2), which is equivalent to prodicting y using the distribution:
pyy x;
0,
= Σ
Ply ()))).
ZEZ
The final LEEP score, representing the average log- licelhodd of the target labels under this expected prodictor, is defined as:
LEEP (0, D) log Σ P(us =) =))(()))))) .E2
1
N-LEEP Li et al. (2021) extend the original LEEP score to enable estimating the transferability of models pre-trained via unsupervised tasks, without requiring source classification head. Given target dataset D = (11181)) (æ2, . (In, Bn),, they extract the feature representation of input using the pre-trained model. Thoy then a Gaussian Mixture Model (GMM) on the set {s
P(s) Σ no N(s pog En),,
VEV
where V is the collection of Gaussian components and n, is the mixture weight. Instead of using source model softmax predictions (()) as in LEEP, they the GMM posterior distribution:
N. Moy Ev) P(v x) = P(u EvEr Te N(s Mur, E.r)
Substituting ((x:); the LEEP formulation with P(v 24), all remaining steps the N-LEEP computation proceed identically to L.EP..
27

---

Published in Transactions on Machine Learning Research (10/2025)
GBC (Pándy et al., 2022) assumes per-class Gaussian distributions and assesses transferability by computing the Bhattacharyya coelficient. (Bhattacharyya 1946) between pairs of classes in the source model's embedding space. The probabilistic model for each class Pe is defined as a Gaussian distribution Pe = (Me, Ee), with parameters estimated as follows:
No Σ 1 (ur=c)
No Σ 1 [m== fa(
1
Pe
Ec
No 1n - (mo))(()) - -Me)"
where zt denotes a target input, fx() denotes the feature representation extracted from the penultimate layer of the source pre-trained model, and [y.=e is the indicator function selecting samples belonging to class Thanks to the Gaussian assumption of the target class distributions, the Bhattacharyya distance DB between two classes C and oj be computed in closed form as:
1 TE (Mc. Me,) + In DB(c,, (Me, Me,)
1
Σ E
where Y 3 De,). The Bhattacharyya coefficient between two class distributions is then computed as: BC(G, 0j) exp (xp(-DD(c,cc))..
The final transferability score from source target estimates the overall class overlap by summing the pairwise Bhattacharyya coefficients between all distinct target class pairs:
GBC = - Σ BC(c,,))
iデ
where the negative sign ensures that higher GBC values correspond to lower overlap, indicating better transferability.
Fréchet Inception Distance (FID) (Heusel et al.; 2017: Fréchet 1957 measures the distance between two probability distributions, typically used to compare sets of images. Assuming the distributions follow multivariate Gaussian distributions, the distance takes a closed-form expression (Dowson Landau 1982 In practice, FID computed on the encoded fcature representations of the two datasets using a feature extractor (e.g., pre-trained Inception network). Let px. Σx and μy, Ey be the means and covariance matrices of the encoded features from the two distributions F and G, respectively. The Fréchet distance is given by:
dF(F.G) 1|ux py + Σx + Ey - 2 2xxx
This formulation combines feature mcans and covariances providing holistic measure of similarity between the two distributions. In transfer learning, it can serve lo estimate the distance between source and target domains Matsoukas et al. 2022 as proxy to quantify domain similarity.
Earth Mover's Distance (EMD) Cui et al. (2018) estimate domain similarity between the source domain and target domain T using Earth Mover's Distance (EMD) (Rachev 1985 Rubner et al. 2000). To make computations tractable, they denote the source domain as 5 ((s,, Ws, ) and the target domain as T {(ty, Wi, 11 where s; and t, represent the i-th and j-th categories in the source and target domains, respectively, and wa Wr the normalized number of images in each category such that.
28

---

Published in Transactions on Machine Learning Research (10/2025)
Σ We; = Wh = They also define g(s,) and g(ty) as the mean of image features extracted from generic feature extractor (e.g. the penultimate layer of a ResNet). The distance between the two domains is then computed using their Earth Mover's Distance (EMD):
>i Th Ej
-
d(s,T)
where Jig is the optimal flow from category 8; to 1y by minimizing the EMD optimization problem.
Image Domain Similarity (IDS) Mensink et al. (002)) define a similarity metric between source $ and target T datasets by extracting image features using a feature extractor. The domain difference computed as the average distance from each target image closest source image:
1 D(T 8) = T Σ min d(S..
where [T] is the number target images, ft and fa are the target and source image features, and d(:,-) denotes the Euclidean distance. They compute this metric using 1000 randomly sampled images from each dataset.
29


Reviews:
[{"review_id": "hYdEZI7tWG", "reviewer": "Reviewer_MQRB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Extensive experiments show the effectiveness of KNN on image classification.\n2. The definition of the ideal transfer learning metric is meaningful.\n\nWeaknesses:\n1. Although extensive studies show the effectiveness of KNN, this paper still lacks the empirical analysis on explanation (e.g., examples' visualization in the low dimension), and the theoretical analysis is also not complete on many metrics.\n2. This paper cannot show how KNN further pushes the transfer learning community. For example, how to guide the application in practical scenarios.\n3. The finding is interesting, but the contribution to methods is limited.\n4. Is this conclusion effective on other tasks, such as text classification?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the most effective transfer learning metric. The authors argue that the good transfer learning metric should not only consider the accuracy in predicting the final performance of the transferred model in the new setting, but also the ability to predict performance improvements gained through the transfer learning process. Based on this demand, the authors conduct an extensive evaluation involving over 42,000 experiments comparing 23 transferability metrics across 16 different datasets to assess their ability to predict transfer performance and reveal that none of the existing metrics perform well across the board. Furthermore, k-nearest neighbor evaluation is found to be the best metric.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4NS2uRK20O", "reviewer": "Reviewer_jtYo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper presents a large-scale empirical validation of over 42,000 experiments, covering 16 datasets and 23 mainstream transferability metrics. The large scale and breadth of the evaluation are compelling. \n2. The authors provide a well-structured taxonomy of transferability metrics, categorizing them into domain distance metrics, metrics under architectural changes, and metrics between source and target tasks. \n3. The authors propose to employ k-NN as a transferability metric and demonstrate its outstanding performance in predicting both absolute and relative transfer outcomes.  The proposed algorithm shows strong robustness to hyperparameter choices. \n4. The paper distinguishes and investigates three different transfer factors—domain, task, and architecture—and provides a thorough analysis of how existing metrics perform under different factors.\n5. The authors simultaneously consider both absolute and relative transfer performance and especially highlight the importance of comparing transfer learning against random initialization. This perspective significantly enhances the validity of the conclusions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper systematically benchmarks 23 transferability metrics and introduces k-NN as a simple yet superior alternative. The authors demonstrate k-NN’s effectiveness across diverse settings and provide a clear understanding of transferability in deep learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Although the empirical results are strong, the theoretical explanation for why k-NN outperforms other metrics is inadequate. Despite being discussed within a PAC learning framework, the work still lacks a solid theoretical foundation.\n2. All experiments are confined to image classification. The study does not cover broader transfer learning tasks such as object detection, segmentation, or text domains, which may limit the generality of the conclusions.", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0xr2VqiTNd", "reviewer": "Reviewer_yBPZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \nThe authors have conducted extensive experiments to investigate the predicability of transfer metrics in the literature. \nThe finding that the $k$-NN predictive performance serves as a transfer metric is interesting and could be practically useful given its simplicity. I consider this finding worth sharing. \n\n## Weaknesses \nTwo points affecting the significance of the findings: \n1. The paper only reports the correlation between the absolute/relative transferability performance and transferability metrics. \nIt would also be helpful to have the target values of absolute/relative transferability performance, since using these, we can identify in which problem settings, conducting knowledge transfer is useful. \nWhile unlikely, it is conceivable that all the problem settings in the paper do not require any transfer (with low relative transfer performance), and the positive correlation of a transferability metric might not indicate that it is useful to predict when to transfer. \n\n2. This point is related to the sample efficiency of transferability metrics, which is related to the practicality of transferability metrics. Transfer learning might be useful in a setting where the target domain does not have sufficiently large data to create a performant model (a situation where the relative transfer performance could be high). \nIt would be useful to know whether the $k$-NN method (or other metrics) could serve as a proxy of transferability in such low-target-data scenario --- the availability of the target data might affect transferability metrics differently.\nWhile I realise that the transfer predictability is useful even outside this scenario (e.g., the target data is abundant, but the user might just want to have a performance boost), addressing this question would strengthen the paper. \n\nWhile the paper is well written overall, there is room for improving the clarity\n* It is ambiguous what kind of transfer learning is performed. My understanding here is that 'transfer learning' refers to using the neural network (up to the final layer) trained in the source domain as the initialisation for the network in the target domain. This point requires clarification. \n* The words 'task' and 'domain' appear to be loosely defined. This is clarified somewhat in the paragraph in Section 4 (which metric... when only the domain changes), but in the first paragraph what these words mean. The authors might want to move the first paragraph (general predictors of transferability) to the end of the section. \n* On a related note, I find the term \"domain\" confusing in the paragraph \"Which metric is better when only the task changes?\". Since in transfer learning, the domain always shifts (from the source to the target), the sentence 'domain remains the same' is a bit confusing. The point of this section is that the task complexity varies, and it seems redundant to say 'domain remains the same'. \n* It seems all the experiments are based on computer-vision datasets. The paper's presentation should reflect on this point (as it can read like the findings are universal across different data types). \n\n\n## Minor comments\n* Typo: \"it my not\"\n* Transfer gap is introduced and seems to have never been used in the paper.\n* There are some words spelled in American English and others in British.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The setup of the paper is transfer learning, where informally one is interested in applying knowledge gained from data in one domain to improve the performance in another domain. \nThis work considers various metrics that are used to predict the transferability of a data domain. \nSpecifically, authors consider computer-vision datasets and compare the correlation between the transferability metrics and actual predictive performance (performance gain) in classification tasks. \n\nThe main finding of the paper is that the predictive performance of the $k$-NN classier in the target domain, whose similarity measure is formed by the features learned in the source domain, is positively correlated with the transfer performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* There are a lot of textual citations where there should be parenthetic (e.g., all the metrics introduced in Section 2). These should be fixed. \n* It would be more helpful to explain the math expressions (where possible) for the considered transferability metrics in appendix, since the reader needs to look at all these citations.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)