How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation.
Authors: Prabhant Singh
Sibylle Hess
Joaquin Vanschoren
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Ananlysis
Transferability
Primary area: datasets and benchmarks

Abstract: Transferability estimation metrics are used to find a high-performing pre-trained model for a given target task without fine-tuning models and without access to the source dataset. Despite the growing interest in developing such metrics, the benchmarks used to measure their progress have gone largely unexamined. In this work, we empirically show the shortcomings of widely used benchmark setups to evaluate transferability estimation metrics. We argue that the benchmarks on which these metrics are evaluated are fundamentally flawed. We empirically demonstrate that their unrealistic model spaces and static performance hierarchies artificially inflate the perceived performance of existing metrics, to the point where simple, dataset-agnostic heuristics can outperform sophisticated methods. Our analysis reveals a critical disconnect between current evaluation protocols and the complexities of real-world model selection. To address this, we provide concrete recommendations for constructing more robust and realistic benchmarks to guide future research in a more meaningful direction.

Published as a conference paper at ICLR 2026
How NOT TO BENCHMARK YOUR SITE METRIC: BE- YOND STATIC LEADERBOARDS AND TOWARDS REALIS- TIC EVALUATION.
Prabhant Singh
Sibylle Hess s.c.hess @ nue.n TU Eindhoven
Joaquin Vanschoren
p.singh @ nue.nl
j. j.vancchoeen@ @ tue.nl AMOR/e Lab TU Eindhoven
AMOR/e Lab TU Eindhoven
ABSTRACT
Transferabilityy estimation metrics are used to find nigh-performing pre-trained model for given target task without fine-tuning the models and without access to the source dataset. Despite the growing interest in developing such metrics, the benchmarks used measure their progress have largely gone unexamined. In this work, we empirically show the shortcomings of widely used benchmark setups for evaluating transferability estimation metries. We argue that the benchmarks on which these metrics are evaluated are fundamentally flawed. We empirically demonstrate that their unrealistic model spaces and static performance hierarchies artificially inflate the perceived performance of existing metrics to the point where simple, dataset-agnostic heuristics can outperform sophisticated methods. Our analysis reveals critical disconnect between current evaluation protocols and the complexities of real-world model selection. To address this, we provide concrete recommendations for constructing more robust and realistic benchmarks guide future research in a more meaningful direction as well as provide a benchmark based the proposed best practices for vision classification.
INTRODUCTION
Using models that are pre-trained on large datasets like ImageNet (Deng et al. 2009) has become standard practice in real-world deep-learning scenarios. However, performance gains can vary considerably depending on the model architecture, weights, and the dataset on which it was pre- trained (the source dataset). This leads to the pre-trained model selection problem. This raises the question: "How can we find nigh-performing pre-trained model for a given target task without fine-tuning our models and without access to the source dataset." " Source Independent Transferability Estimation (SITE) metrics address this question by computing candidate model, that used to rank models by their predicted downstream performance. This research area is growing rapidly, with papers appearing at major Al venues such as ICML, NeurIPS, and CVPR. Progress in the field has so far been measured primarily against a small set of widely adopted benchmarks.
-ecllllt- score for each
While these benchmarks have been useful for driving early advances, we argue that they fail to capture the complexities of real-world applications. To address this gap, our paper offers critical empirical analysis of the most commonly used benchmark setup for SITE metric evaluation. We identify fundamental flaws in its design that call into question the validity of reported results. Our contributions are to:
1. Empirically demonstrate how current benchmarking practices give misleading performances of SITE metrics which questions their reliability,
2. Show that simple, static ranking heuristic can outperform sophisticated metrics, exposing the trivial nature of the task posed by the benchmark which further highlights the weakness of the current benchmark, and
3. Propose a set of actionable best practices for constructing more robust and meaningful benchmarks suitable for practical challenges of real-world model selection.

---

Published as a conference paper at ICLR 2026
Ranked Models
Model Zoo
Target Dataset
$
Score-1.0
Labels
(p(x))))))
2
Feature Extraction
Transforability Estimation
Score=o.7
1
Score-0.5
Figure 1: Illustration of Source Independent Transferability Estimation (SITE): Given a set of pre- trained models (on the left), SITE metric computes score Tm based on extracted features on target dataset. The scores Tm are used to rank the pre-trained models according to their transferability.
Despite significant progress in the development of novel SITE methods, the empirical evaluation standards within this emerging research area have yet to reach the maturity seen in other machine learning domains. To address this issue, we propose best practices rectify them, leading to the SITE benchmarking and evaluation checklist (Appendix A
TRANSFERABILITY ESTIMATION: BACKGROUND
The idea behind transferability estimation is simple: estimate which model from ZoO of models would perform best after fine-tuning. Transferability estimation as a research area is fairly young; the H-Score (Bao al.) (209)) and NCE (Tran al. (2019) can be considered early works on this topic, introducing the evaluation of transferability and the assignment of models corresponding to an estimate of their transferability for given target task. We illustrate how transferability estimation metrics work in Figure T
There two widely accepted problem scenarios for transferability estimation: -ource-dependent transferability estimation (where one has access to the source and target dataset) and source- independent transferability estimation (where one does not have access to the source dataset), will be focusing on the latter.
2. SOURCE DEPENDENT TRANSFERABILIT ESTIMATION (SDTE)
The SDTE scenario assumes access to the source datasets which the models have been pre-trained. Apart from the fact that this assumption is often not met, a drawback of common SDTE metrics is the use of distribution matching methods like optimal transport (Tan et al. 2021). which are typically very expensive to compute. In addition, SDTE metrics are not reliable when the discrepancy between the source and target datasets is very high(Braccaio et al. (2025; for example, when comparing the entire ImageNet21 (Deng et al. 2009) to the Cars (Krause et al. 2013) or Plants (G. & J. 2019) datasets.
2

---

Published as a conference paper at ICLR 2026
2.2 SOURCE INDEPENDENT TRANSFERABILITY ESTIMATION (SITE)
The Source Independent Transferability Estimation (SITE) assumes access to the source model but not the source training data. This is more realistic scenario, as we might not always have access to the source dataset, nor have the capacity to store the typically very large source datasets like ImageNet (Deng et al. 2009) or LAION (Schuhmann et al. 2022) in our local setup. SITE methods typically rely on evaluating the feature representation of the source model on the target dataset and its relationship with the target labels. There are several SITE metries inspired by various viewpoints. For instance, LogME (You et al. 2021) formalizes the transferability estimation as the maximum label marginalized likelihood and adopts a directed graphical model to solve it. SFDA (Shao et al. 2022) proposes self-challenging mechanism; it first maps the features and then calculates the sum of log-likeldooo as the metric. ETran (Gholami et al., 2023) and PED (Li et al., (023) treat the problem of SITE with an energy function and act pre- pre-processor for other SITE methods. ETran uses energy-based models to detect whether a target dataset is in-distribution or out of distribution for a given pre-trained model, whereas PED utilizes a potential energy function to modify feature representations to aid other transferability metrics like LogME and SFDA. LEEP (Nguyen et al. 2020) the average log- likeiiihoood of the log-expected empirical predictor, which is nonpparametric classifier based on the joint distribution of the source and target distribution. N-LEEP (Li et al. 202T) is further improvement on LEEP by substituting the output layer with a Gaussian mixture model. TransRate (Huang et al. 2022) treats SITE from an information theory point of view by measuring the transferability as the mutual information between features of target examples extracted by pre-trained model and their labels. There have been applications of transferability estimation and proposed metrics for different domains such as medical imaging (Juodelyte et al. (0244) Singh
et al.
2025 Yang et al., (2023) and speech recognition (Chen et al. 2023). We suggest the survey by Ding et al. 2024) for complete view of transferability metrics.
3 SITE: PROBLEM STATEMENT
We assume that given a target dataset D (xx,, (x of N labeled points and M pre- trained models {Lm, = (om, 2mm) ) m= Each model 4m consists of a feature extractor Φn (z) E Rd that returns a d-dimensional embedding and the final layer or head v'm that outputs the label prediction for the given input (I). The task of estimating transferability is to generate a score for each pre-trained model so that the best model is the top of ranking list. For each pre-trained model 2m. a transfcrability metric outputs a scalar score Tm that should be coherent in its ranking with the performance of the fine-tuned classifier dn.. That is, the goal is to obtain scores such that score Tm, 2 T if and only if the fine-tuned model & has higher probability to predict the correct labels the target dataset than model immp
N
N
1
Σ
p(yn Xni 2m))
Σ 1=
(nn Pmz))
where p(yn Rn; ( indicates the probability that the fine-tuned model i. predicts label yo for input Xn. Hence, a larger Tm should correspond to better performance of the model target data D.
4 TRANSFERABILITY ESTIMATION: STANDARD SETUP
Widely adopted transferability estimation methods, such as LogME You et al.) 2021 Tran-
<Rate (Huang et al. 2022). NCTI (Wang et al., 2023) LEEP (Nguyen et al. 2020). SFDA Shao
al. (022), ETran Gholami et al. 2023), GBC (Pándy al.) (0022 and LEAD (Hu et al. 2024 are evaluated on benchmarks sharing similar models and datasets.
The setup used in LogME, TransRate, H-Score, SFDA, ETran, and NCTI uses pre-trained ResNets (He et al. 2015) (ResNet34, ResNet50, ResNet101, ResNet151), DenseNets (Huang et al. 2016) Net16,, DenseNet121, DenseNet201). MobileNet (Howard et al. (017). Inceptionv3 (Szegedy ct al. 2013). MNASNet TTan et al. 2018) and GoogleNet (Szegedy et al. (014). These models are fine-tuned CIFARIO (Krizhevsky et al. 2009), CIFAR 100 (Krizhevsky al. 2009), Pets (Parkhi

---

Published as a conference paper at ICLR 2026
al. 2012), Aircraft MMji et al. 2013a), Food (Bossard et al. 2014) and DTD (Cimpoi et al. 2014).
In this work, we focus on this widely adopted SITE benchmark setup.
Fine-tuning details
To obtain test accuracies, these models were fine-tuned with a grid search over learning rates {10-1,10) 10-2, 10-30 10 and a weight decay in {10-a, 10 10-6 10-6, 0). The best hyperpa- rameters are determined based on the validation set. The final model is fine-tuned on the target dataset with the selected parameters. The resulting test accuracy is used the ground truth score
Gm for model pm This way, we obtain a set of scores {Gm m=1 as the ground truth to evaluate our pre-trained model rankings.
Evaluation Protocol
The evaluation of transferability metries reflects the correlation between the ground truth accuracy and the achieved SITE score. Currently, weighted Kendall's Tau (Vigna) 2015) prevails as a measure estimate the correlation. Earlier transferability works like H-score used Pearson correlation as an evaluation metric, but Pearson's 's is considered as too sensitive to scale, as two scorings will induce the same order can differ in their evaluation merely due calibration. Kendall's interpretable rank statistic, as it counts the number of swaps that bubble sort would have to perform to put one list in the same order as the other one. More precisely, Kendall's T returns the ratio of concordant pairs minus discordant pairs when enumerating all pairs of transferability estimations {Tmmm= and ground truth transferability scores {Gm m== as given by:
more
2
I
E ISKKjMM
sgn G; G5) sgn - -Ty)-
M(M -
The sgn function returns the sign of the input value and zero the input value zero. Since practical applications rely mainly on the correct order for the top performing pre-trained models, a weighted variant Kendall's Tw is used. Weighted Kendall's Tw is computed as
Σ ISKjjM
GG - -G) sgn (T; - (T))
Tw = M(M 1)
where p(t) returns ranking of indices, audices,ssarting at zero. As a default choice, weighted Kendall's tau uses hyperbolic weighing : w(r, 8) s+I + r+I In transferability estimation, the ranking p reflects the ranking of the ground truth accuracy. As result, any pair that involves of the top transferring models gets a large weight. In the rest the text, we will refer to this benchmark (datasets, models, evaluation protocol) standard benchmark.
AN EMPIRICAL CRITIQUE OF THE STANDARD BENCHMARK
While the standard benchmark is widely used, we contend that it is built flawed foundations that lead to an overestimation of the true capabilities of SITE metrics. We identify and empirically validate three critical limitations: unrealistic model space that does not reflect practical challenges
transfcrability estimation, benchmark that is solved by a static ranking for all considered datasets, and misleading differences in SITE scores that do not meaningfully correlate with performance gaps. For the purpose of this study, we examine the following metrics on the standard benchmark: LogME You et al., 2021). TransRate (Huang et al. 2022). GBC (Pándy et al. (022). NLEEP (Li
ct al. 2021). (Shao et al. 2022), and H-Score (Bao et al. 2019)
5.1
CRITIQUE 1: THE MODEL SEARCH SPACE IS UNREALISTIC
We argue that the model space of the standard benchmark's model pool unrealistic, because dominated by models of varying sizes from only two architectural families (ResNets and DenseNets). In real-world scenario, practitioners are not interested in knowing whether they should use "bigger vs. smaller" ResNet, but rather which architecture performs under particular constraints such as size. training speed, inference speed, and ease of access. For example, larger vision classification models known to predictably outperform their shallower counterparts (He et al. 2015), and mixing small and large variants of the same family of vision classification models reduces the complex task of model selection to trivial detection of the largest model.
4

---

Published as a conference paper at ICLR 2026
CIFARIO
(b) CIFARI00
Aireraft
(d) DTD
(c) Pets
f Food
Figure 2: Performance of transferability metries when we remove architectures of the same families.
We sequentially remove the architectures denoted on the horizontal axis and report the achieved Twp- pattern performance decrease with every ablation is observed.
The inclusion of MobileNet and MNASNet alongside ResNets and DenseNets is likewise misaligned with the benchmark's 's use case. MobileNet and MNASNet are designed for edge computing environ- ments. In the experimental evaluation, MobileNet and MNASNet consistently occupy the bottom ranks (cf. Figure 3). Hence, removing those models from the search space has little effect on the performance evaluation. However, advocate against using these models in the standard benchmark to avoid unnecessary inflation of the search space.
We therefore recommend (i) excluding edge-oriented models from the benchmark and (ii) using at most one representative architecture per family, ensuring that the compared models have similar sizes.
Validation via Model Ablation To test the robustness of SITE metrics under search space, we ablate the largest models from overrepresented families (ResNet- 15NN, ResNet- 101, DenseNet-169, DenseNet-201) and recompute weighted Kendall's Tw- Figure 2 plots Tu progressively remove these models, indicated on the horizontal axis. The rightmost points correspond setting with one model per family, containing only of the initial 11 models.
realistic
Most SITE metrics exhibit sharp drop in after ablation. Except on DTD and Pets, all SITE metrics fall below 0.6 once the oversized variants are removed. The results also show that none of the metrics are robust to changes in the model space. For every metric, dataset exists where the removal of single model results in steep decrease performance. In this more realistic setup, no metric reliably predicts transferability across datasets. This demonstrates that the high performance of these metrics brittle and heavily reliant on correctly ranking few over-represented models in flawed benchmark. Most importantly, none of the existing SITE metrics are able to reliably estimate transferability a setup where models similar sizes are compared.
5.2 CRITIQUE 2: THE BENCHMARK SOLVED BY STATIC RANKING
Dependencies between the candidate models (having larger and smaller variants of the same archi- tecture the model search space) and lack of diversity in the evaluated datasets lead static leaderboard, where few high-capacity models, such ResNet-152, consistently occupy the top ranks regardless of the target dataset. We visualize the ranking of models Figure 3 The figure shows that the top performing model, Re5Net-152, occupies the first rank for 8 of the 10 datasets in the benchmark. The second place is always occupied by one of the top 3 models (ResNet-152, DenseNet-201, and ResNet-101).
As a result, we question whether the standard benchmark is suitable for assessing transferability estimation if SITE metric with data- iata-indeeendeet ranking achieve nigh-performance mea- surement. We probe this question by introducing a controlled experiment that tests whether a fixed, dataset-agnostic ranker rival SITE metrics. This leads validate the hypothesis through
5

---

Published as a conference paper at ICLR 2026
Distribution of Rank Pasitions Across Datasets (Gradient Ranko
min4t152
dersenet0)
darsenct221
Pan te0
Model
grogennt
x
Count accumences
Figure 3: Visualization of the ranking distribution of models in the standard (ordeed by fine tuned performance). The models at the top occupy the first ranks most datasets.
Dataset Metric
Aircraft
CIFARIO
CIFARI00
DTD
Food
Pets
Average
GBC
-0.12
-0.02
0.09
0.14
0.10
-0.15
0,007
TransRate
0.14
0.51
0.20
0.20
-0.05
0.17
0.195
SFDA
-0.22
0.85
0.79
0.63
0.30
0.34
0.448
H-Score
0.60
0.91
0.80
0.04
0.59
0.37
0.552
NLEEP
-0.51
0.76
0.84
0.70
0.69
0.84
0.553
LogME
0.41
0.85
0.72
0.66
0.39
0.41
0.573
Static Ranking
0.84
0.91
0.98
0.99
0.80
0.94
0.91
Table Comparison transferability estimations, computed by weighted Kendall's tau, for a static ranking versus SITE metrics on the standard benchmark. The static ranking achieves the highest overall.
static ranker that exploits the leaderboard's inherent bias without leveraging any task-specific information.
Validation via Static Ranking Heuristic We test naive static ranker in the standard benchmark, which orders models according fixed sequence. The static ranking follows simple heuristic based on the model size and an alternation of ResNet and DenseNet model families, resulting in the following order:
We report the performance of this static ranking in Table We observe that the static ranking significantly outperforms the sophisticated SITE metrics, achieving the highest Tu on every dataset. On average, the static ranking achieves Tu = 0.91, while the best performing SITE metric, LogME, achieves Tu 0.57. This finding questions the insight gained from the standard benchmark, as rewards the memorization of fixed model hierarchy rather than the ability to perform true task-specific transferability estimation.
5.3 CRITIQUE 3: SITE METRICS ARE NOT EVALUATED TOWARDS FIDELITY
Beyond ranking, practical transferability metric should provide scores whose magnitudes meaningful. That is, a large gap in metric scores should correspond to large gap in downstream accuracy, allowing assess whether selecting higher-scoring model is worth the potential

---

Published as a conference paper at ICLR 2026
increase in computational cost. The standard benchmark evaluation protocol, focused solely rank correlation, overlooks this crucial property.
We formalize this property follows. Let Ace be the difference in accuracy between two models on a target dataset D, and let år be the difference in their transfcrability scores:
Ace X, ;,;
Ac, D) Ac) AT(X, T(X) ((X)-(() T(Y)
An ideal metric should preserve the ordering of differences: for any four models A, B, C D from the model space M: VA, B, C,D.M, (A, (C, = AT(A, (C,
Validation via Pairwise Difference Correlation To quantify this property, which we term fidelity accuracy differences, we formalize it the correlation between pairwise differences in accuracy (Aac) and pairwise differences in the metric's score (Ar). A high correlation would indicate that score differences are meaningful proxies for performance gaps.
We compute the Pearson correlation between all (Aace] and {Ar} pairs for each metric and dataset. The resulting heatmap is shown in Figure 4 We observe that nearly all metrics exhibit weak correlation with accuracy differences. For example, our analysis reveals that a LogME score difference 0.09 can correspond to an accuracy gap as large as 2.5% or small 0.5% the Pets dataset. A more detailed plot of the relationships from scores to ground truth accuracies can be found in Appendix E The lack of reliable mapping between score gaps and performance gains severely limits the practical utility of these metrics for end-users, who cannot confidently interpret the scores make informed decisions.
Aircratt
-0.14
0.08
-0.08
0.37
-0.07
-0.17
CIFAR 10
00.17
0.39
0.33
0.32
0.06
-0.09
CIRRR 190
Dataset
DTD
-0.14
-0.10
0.03
0.06
0.48
-0.25
Food 101
-0.12
0.09
-0.11
0.01
0.51
-0.22
-0055
Pets
0.31
GBC
H-Score
LogME
NLÉEP
SFDA
Transhate
Figure 4: Heatmap correlation of A and Ar
5.4 TRANSFERABILITY ESTIMATION OUTSIDE OF COMPUTER VISION
In this work, we focus on popular computer vision and image classification setups; however, but other fields have similar benchmark issues. For example, MEAF (Hao & Wang 2025) is SITE metric for spiking neural networks, where our critique applies. The SEW-ResNet- - 152 model dominates the top rankings, and the performance differences can be as small as 0.2% the second rank (which can occur due to fine-tuning differences). LogME experiments winner every time as shown Table Critique 2, the comparison models with varying sizes from the same families, can also be applied to recent work in object detection Zhang et al. (2025). Here, varying sizes of YOLO models were compared to one another for the same task, in which YOLOv5m won in 4 out of tasks. EMMS (Meng et al.) 2023) conducts investigation into the transferability of ViT models while comparing ViT-S to VIT-B, where ViT-B outperforms every model and holds the first rank in 8 out of 11 tasks.
NLP tasks also reveal one static
7

---

Published as a conference paper at ICLR 2026
6 RELATED WORK
Previous work has conducted large-scale analysis of Source Independent Transfcrability Estimation (SITE) metrics. For instance, AAostinelll al. (2022) perform comprehensive evaluation across 700,000 experiments, showing that the effectiveness of a transferability metric is highly dependent on the specific experimental scenario. Although the setup proposed by AAgsstnnelli al. (2022) is thorough, its scale makes it impractical to reproduce in typical research settings without substantial computational resources. Similarly, Ibrrhim et al. (2021) highlight the instability of SITE metrics in the presence of class imbalance and propose an adaptation of the H-score tailored to their setting. 2024) recommend different SITE metries for different scenarios.
instance,/goostined
Baker & Handmann
Chaves et al. (2024). Kazemi et al. 2025), and Bolya et al. (202)) mention some drawback: of the standard benchmark continue utilize an impractical model search space (Critique and may
encounter similar challenges those associated with static leaderboard (Critique 2). Kornblith
(et al. (2019) show that higher ImageNet accuracy predicts better transfer on web-scraped target datasets, which largely overlap with the datasets of the standard benchmark. [ang et al. (2023) demonstrate that this correlation breaks down for real-world datasets that are not web-scraped. Our work complements these findings by exposing dataset and model search space synergies that result in benchmark limitations.
In contrast to these studies, goal is not to evaluate particular SITE metric to recommend which method to use and when, but rather to examine the benchmarks and recommend best practices for evaluating SITE metrics. We introduce framework for assessing the effectiveness of SITE benchmarks and offer practical insights into constructing experimental setups that enable more reliable evaluations and better inform the practitioner, as the end use of these metries is provide improved evaluations for users.
7 DISCUSSION
In this section, we move from critique to construction and propose a set of actionable recommenda- tions designed to foster the development of more robust and practically relevant benchmarks. We show how we can address the limitations identified the previous section and move towards building more reliable benchmark setup for SITE metrics. The goal is ensure that future transferability metries are validated against challenges that mirror the complexities faced by practitioners, thereby providing trustworthy guidance for real-world model selection. We therefore polish recommendations now on "How to build better benchmarks for your transferability estimation metric."
Best Practice I: Release code and data for your transferability estimation metric.
In this work, we were only able to examine a handful of metrics and datasets due to the limitations of publicly available data. good practice to ensure reproducibility and reduce noise in score discrepancies is to release the code for the metric, the datasets with links on which it was trained, the score metrics obtained, final accuracies, and the pretrained models on which the score was computed (for different task transfer scenarios). This, firstly, ensures full reproducibility and, secondly, allows for the incorporation of inconsistent scores different works for the same task.
Best Practice 2: Construct Diverse and Non-Trivial Model Space
To create a more robust evaluation, the model space must be intentionally diversified present a non-trivial selection problem. This involves curating a set of models from architecturally distinct paradigms, such as Convolution Neural Networks (e.g., ConvNeXt), Vision Transformers (e.g., VIT, Swin), and MLP-based models (e.g., MLP-Mixer). Furthermore, to ensure fair comparison that isolates the impact of architectural inductive bias, comparable models should be chosen according to practitioners constraints, such computational budgets (e.g., parameter counts speed, training speed, etc. This forces the evaluation of the metric to move beyond simple scaling rules and to make nuanced judgments about which architecture is best suited for the specific downstream task.
Best Practice 3: Ensuring Diverse and Challenging Dataset Space
The suite of downstream tasks is as critical as the model space. A metrics utility is defined by its ability to generalize across wide range of applications. A benchmark that relies a narrow set of
8

---

Published as a conference paper at ICLR 2026
similar overly simplistic datasets cannot provide this assurance. We identify two essential axes of diversity for the dataset space:
Task Diffiicltyy and Performance Headroom: Many existing benchmarks use datasets where modern pre-trained models achieve near-saturated performance (c.g., 99% accuracy). This "performance ceiling" makes meaningful rankings impossible, as the marginal differ- ences between top models often statistically insignificant and fall within the noise of the fine-tuning process. A robust benchmark must include challenging datasets that provide sufficient performance headroom for even the strongest models, ensuring that a clear and statistically significant performance gap can be measured.
Domain and Task Variety: metric's ' robustness tested by its performance across varied domains and task types. Consequently, the benchmark ought to incorporate datasets that cover a variety of visual domains, including fine- ane-ggained classification (for instance, FGVC Aircraft (Maji et al. 2013b), Stanford Cars (Krause et al.] 2013)), medical imaging, satellite imagery, and texture analysis (such as DTD Cimpoi et al. (2014)). Additionally, datasets that are not sourced from the internet should be included, consistent with the findings of Fang et al. (2023). This evaluates the metric's capacity to handle varying degrees of domain shift from the original source pre-training data. We acknowledge that the property of diverse datasets is generally present in the majority of the SITE papers.
Best Practice 4: Engineering for Performance Spread and Rank Dispersion
critical but often overlooked flaw standard benchmarks is the persistence of static model hierarchy, where one or two models dominate nearly all tasks. This "static leaderboard" undermines evaluation validity: a transferability metric can achieve a high weighted Kendall's Tau simply by favoring the top model rather than by accurately predicting task-specific suitability. To isolate architectural inductive bias, models should be matched on computational budgets (e.g., parameter counts or FLOPs). This design forces transferability metrics to move beyond simple scaling laws and, instead, to make nuanced judgments about which architecture best fits a given task.
Equally important, effective benchmark must exhibit high rank dispersion: model rankings should vary substantially across tasks, with different architectures excelling on different datasets. Achieving this requires a careful co-design of the model pool and dataset pool so that unique inductive biases are rewarded in different contexts. Such a setup provides a far more rigorous test of transferability metrics, ensuring they identify the right model for the right task rather than defaulting the same top performer.
We view these guidelines first step toward evolving standards for benchmarking transferability estimation, which may need to adapt to different deployment settings, such as edge devices versus large-scale computing environments. While our guidelines are essential steps, we also provide a benchmark for image classification that takes into account Best Practices 2,3, and 4 in the next section.
BENCHMARK BASED ON BEST PRACTICES
We built robust benchmark based on the best practices described in the previous section. Our proposed benchmark consists of the following models: Twins-SVT (Chu et al.) 2021). XCiT (El
Nouby et al. 202T), CoaT (Xu et al. 2021), DeiT (Touvron al. 2021), Max VTT (Tu al. 2022). and MVIT v2 (ii et al. 2022). This selection covers complementary model design paradigms, while all models have a similar parameter range, and no model a direct improvement over the other. Fine-tuning these selected models on datasets from the standard benchmark results in multiple models obtaining 99% accuracy, which is not meaningful for SITE. To address this limitation, we expand the benchmark's 's datasets with selected datasets from the Meta-Album (Ulah et al. 2022). From the 30 datasets of the Meta-Album, we do not select those where multiple models achieve very high or close to 100% accuracy.
Our resulting benchmark consists of the following datasets: Sports (Piosenka), Plant Village (G. & J.
2019), RESISC (Cheng et al.] 2017), Insects [Wu et al. 2019). PanNuke (Gamper et al. 2019). MPI
Human Pose Dataset (Andri uuka et al.) 20nn). (Picek et al. (021). RSD (Long et al.
2017
Boats Gundogdu et al. 2017). Plant Doc (Singh et al. 2020), Stanford Actions ( (ao al.
201 1

---

Published as conference paper at ICLR 2026
DTD Cimpoi et al., 2014), Subcellular Human Protein Dataset(PRTA) (Thul al. 2017). Insects SPIPOLL (Wu et al. 2019). and Dogs Khosla et al. (201T). Based on Critique 2, we evaluate our benchmark against static ranker. We create this statie ranker based the fine-tuned performance of models; more information on this can be found in the Appendix
We show the performance of SITE metrics and the static ranker observe that not a single SITE metric can perform consistently well for our benchmark. The static ranker achieves comparably low values of Tur E [-0.3, 0.77] with mean of 0.31. While following best practices, we propose that this benchmark should be used as an example to build a better and more reliable benchmark for SITE metric evaluation for different tasks, such as computer vision, NLP, and information retrieval.
benchmark in Table We
Dataset
TransRate
LogME
NLEEP
SFDA
HScore
GBC
Static
Sports
0.39
0.25
0.30
0.70
-0.08
0.38
0.46
PlantVillage
RESISC
0.24
0.11
0.14
0.76
0.23
0.36
0.55
Stanford Actions
0.16
0.37
0.28
0.07
0.01
0.03
0.27
Insects
0.72
0.57
0.84
0.53
0.52
0.87
0.56
DTD
-0.33
-0.37
-0.37
-0.48
-0.33
-0.42
0.01
PanNuke
0.14
Dogs
00.11
.1.1
0..62
0..3
-0.30
00.99
40.15
Fungi
0.44
0,70
-0,22
0.77
0.40
0.34
0.77
Plant
0.54
0.30
0.30
0.10
0.10
0.48
SPIPOLL
-0.15
0.10
-0.15
-0.28
-0.18
-0.15
0.32
RSD
0.02
-0.20
-0.34
-0.07
-0.06
-0.04
0.60
PRIA
Beats
0.20
-0.49
0.38
-0.32
-0.33
0.27
0.09
Average
0.13
0.061
0.04
0.15
0.06
0.17
0.31
Table 2: Tu performance 1.5 benchmark datasets from Meta-Album, showing the performance of LogME, SFDA, NLEEP, TransRate and Static Ranker.
LIMITATIONS AND FUTURE WORK
Our work examines the standard SITE benchmark for image classification; our investigation can serve blueprint for future studies in other setups and can be further expanded NLP, object detection, and medical image classification domains as well. A limitation that current benchmarks and metrics suffer from the integration of different finetuning strategies, optimizers, and hyperparameters the SITE metric evaluation. Currently, the methods and benchmarks have not been developed to take these hyperparameters into account while predicting transferability, whereas research has shown that they play a significant role in model performance. A promising direction can be the incorporation of learning from social choice theory (Zhang & Hardi (0224) to improve the development of more reliable benchmarks for SITE.
10 CONCLUSION
Benchmarking is the comerstone of machine learning research, allowing researchers to develop robust ideas and enabling scientific progress. Without robust benchmarks, trust in research methods can erode; to prevent this, we have shown how to ground benchmarks more realistic scenarios for SITE metrics. Our experiments highlighted critical failures in current benchmarking practices. To aid future research, we provide actionable best practices and a SITE benchmark and evaluation checklist for constructing robust benchmarks in the Appendix A inspired by the NAS Checklist (Lindauer & Hutter 2020). Our set of recommendations and proposed experiments is actionable and concrete, ensuring robust benchmarking of SITE metrics. This work serves a call to action for the community to adopt more rigorous standards; by doing so, we can foster the development of transferability metrics that genuinely useful to practitioners and provide truly predictive and reliable guidance.
10

---

Published as a conference paper at ICLR 2026
11
ACKNOWLEDGGEEENTS
This work was supported by European Union's Horizon Europe research and innovation programme under grant agreement number 101214398 (ELLIOT).
12 REPRODUCIBILITY STATEMENT
To enable reproducibility, provide the code, data, and execution scripts in the supplementary files. We also provide additional code for our experiments and Jupyter notebooks to reproduce the figures.
REFERENCES
Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metrics evaluations? In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XXXIV, 2022. doi: 10.1007/
978-3-031-19830-4.1
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3686-3693, 2014. doi: 11111//V...00..4111
Nermeen Abou Baker and Uwe Handmann. One size does not fit all in evaluating model se- lection scores for image classification. Scientific Reports, 14, 2024. URL https://api semanticscho
Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang. Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theoreti approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 2309-2313, 2019. doi: 10. 109/ICPI 4019.8803726.
Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning, 2022. BRLhhttps:/ /ttp://t......rg/aaaaa1111.1.11. http://///.....aa/////////////./
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Lorenzo Braccaioli, Anna Vettoruzzo, Prabhant Singh, Joaquin Vanschoren, Mohamed-Rati Bouguelia, and Nicola Conci. Meta-learning transformers to improve in-context generalization, 2025. URL /tttts://aaxiiv wwtpassi.rrg/as//////ea 2507 http:///aaab22255500000
Levy Chaves, Eduardo Valle, Alceu Bissoto, and Sandra Avila. Back to the basics on predicting transfer performance, 2024. URL httltttss//sxiiii org/obogg//..ooo wwss/202/ 240.//22222220
Zih-Ching Chen, Chao-Han Huck Yang. Bo Li, Yu Zhang, Nanxin Chen, Shuo-Yin Yiin Chang, Rohit Prabhavalkar, Hung yi Lee, and Tara Sainath. How to estimate model transferability of pre-trained speech models? In Interspeech 2023, pp. 456 460, 2023. doi: 10.2 10.21457/mtersppecc.2000 1079.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. CoRR, abs/1703.00121, 2017. URL
00121
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xiaolin Wang, Chunhua Wei, Huaxia Xia, Jiange Shen, and Chunhua Luo. Twins: Revisiting the design of spatial attention in vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10.1 W0W.CCVPR.000.5200000

---

Published as a conference paper at ICLR 2026
Yuhe Ding, Bo Jiang, Aijing Yu, Aihua Zheng, and Jian Liang. Which model to transfer? survey transferability estimation, 2024. URL https::/axxxi...ott htt.ooggg/bb////////
Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Armand Joulin, Matthijs Douze, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jegou. Xeit: Cross-covariance image transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Alex Fang, Simon Kornblith, and Ludwig Schmidt. Does progress on imagenet transfer to real- world datasets? In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023, htth:tps::/ wwwprevieeeeeeeeeeeeeeeeee //tptttsss///p/peeeeee/eeee net/ forum - SS3CK3yx52
Geetharamani G. and Arun Pandian J.
Identification of plant leaf diseases using nine-layer deep convolutional neural network. Computers and Electrical Engineering, 76:323-338, 2019. ISSN 0045-7906. doi: http://oo..oor//o1..looocoooceeo sciencedirect com/ htt.a//cww.....o..oooo.o.///////
URL https: //www
Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot. Pannuke: an open pan-cancer histology dataset for nuclei instance segmentation and classification. In European Congress on Digital Patholoy, pp. 11-19. Springer, 2019.
Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang, Etran: Energy-based transferability estimation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 18613-18622, 2023.
Erhan Gundogdu, Berkan Solmaz, Veysel Yucesoy, and Aykut Koc. Marvel: A large-scale image dataset for maritime vessels. Shang-Hong Lai, Vincent Lepetit, Ko Nishino, and Yoichi Sato (eds.), Computer Vision - ACCV 2016, pp. 165-180, Cham, 2017. Springer International Publishing. ISBN 978-3-319-54199-..
Haiqing Hao and Wenhui Wang. Bayesian transferability assessment for spiking neural net- works. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
2016 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
URL https::t///.m. icsp
org/Corrps.ID: 206594692
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017. URL ftttt:://aaaattt. org/ /abb/1104 04861
Zixuan Hu, Xiaotong Li, Shixiang Tang, Jun Liu, Yichun Hu, and Ling- Yu Duan. Lead: Exploring logit space evolution for model selection. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. doi: 10. 10.110//CVPR2233..222..0777
Gao Huang, Zhuang Liu, and Kilian Q Weinberger. Densely connected convolutional networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2269, 2016. TRLLttts:://aat http://////ppp.aaaaaaeaeeeeeeeee
9433631
Long-Ke Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy trans- ferability estimation. In International Conference on Machine Learning, pp. 9201-9225. PMLR, 2022.
Shibal Ibrahim, Natalia Ponomareva, and Rahul Mazumder. Newer not always better: Rethinking transferability metrics, their peculiarities, stability and performance. In ECML/KKDD, 2021. URL https://apt www.ntettii.c icscho ar. www.sccchhh.....ooooooooooeeeeee ID: 2o
Dovile Juodelyte, Enzo Ferrante, Yucheng Lu, Prabhant Singh, Joaquin Vanschoren, and Veronika Cheplygina. On dataset transferability in medical image classification, 2024. URL https
12

---

Published as a conference paper at ICLR 2026
Alireza Kazemi, Helia Rezvani, and Mahsa Baktashmotlagh. Benchmarking transferability: A framework for fair and robust evaluation, 2025. URL https::/axx...xxx//////////
20121
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Tine-Grained Visual Calegorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Simon Kornblith, Jonathon Shlens, and Quoe V. Le. Do better imagenet models transfer better? In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2656-2666, 2019. doi: 10.1109/CVPR.20 2019.00277.1
Jonathan Krause, Michael Stark, Jía Deng, and Fei-Fei. object representations for ine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Xiaotong Li, Zixuan Hu, Yixiao Ge, Ying Shan, and Ling- Yu Duan. Exploring model transferability through the lens of potential energy. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023. doi: 10. 109/1CCCC111000002000 0000000
Yandong Li, Xuhui Jia, Ruoxin Sang. Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. doi: 10.1 10.1109/CVP444437.... 2022 202000069.
Yanghao Li, Chao- Chan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Piotr Dollar, and Ross Girshick. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Marius Lindauer and Frank Hutter. Best practices for scientific research on neural architecture search. Journal of Machine Learning Research, 21(243): 2020. URL http://jml
Dxc/papprrs/21/22-5555 html
Yang Long, Yiping Gong, Zhifeng Xiao, and Qing Liu. Accurate object localization remote sensing images based convolutional neural networks. IEEE Transactions Geoscience and Remote Sensing, 55(5):2446-244 2498, 2017. doi: 20.11099/RRS..201266666666
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. ArXiv, 1bs/1306.5151, 2013a.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. ArXiv, abs/1306.5151, 2013b. URL https://apt
semanti corttiischolass wwwcchollar.or//oooruuu. TD: 021221180000
Fanqing Meng, Wenqi Shao, zhanglin peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, and Ping
Luo. Foundation model is efficient multimodal multitask model selector. In Thirty-seventh Conference on Neural Information Processing Systems. 2023. URL httL:tttt///operevaaee net/ htte//ooumm?i.. wwt.mmdddddeeppppeeeeem
Cuong V. Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: new measure to evaluate transferability of learned representations. In Proceedings of the 37th International Conference on Machine Leurning, ICML'20. PMLR, 2020.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012. URL
https: /tppiapp.../////// www.sttttninncccoeco. ar wtto http//orppss::::: : 279227999
Lukas Picek, Milan Sulc, Jiri Matas, Jacob Siilmann-Clauseee Thomas S. Jeppesen, Thomas Laessoe,
and Tobias Froslev. Danish fungi 2020 not just another image recognition dataset. 2021.
Gerald Piosenka. 100 sports image classification.
URL https:/ www. www.w.wwgge..ooo
@ataaetts/ggiosenka//pp..t aiatiiiatiiiiooooooooococ
13

---

Published as a conference paper at ICLR 2026
Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Trans- ferability estimation using bhattacharyya class separability. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. doi: 10.1109/VVPRS228822222222
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixii Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL htt.httss///pperevieieee
net fttr:/ooumma2 43377MM3777
Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang. Xiaogang Wang, Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in self-challenging fisher space. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel. October 23-27, 2022, Proceedings, Part XXXIV, pp. 286-302, Berlin, Heidelberg, 2022. Springer- ISBN 978-3-031-198291 doi: 10. 0000779788-3001-11993-----101100
URL http:::tp///.mmm...mm /doi- 3-031 19830 17
dianus. 1ibr. tue. n1 10. 1007 978
Davinder Singh, Naman Jain, Pranjali Jain, Pratik Kayal, Sudhakar Kumawat, and Nipun Batra. Plantdoe: A dataset for visual plant disease detection. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, CoDS COMAD 2020, pp. 249-253, New York, NY, USA, 2020. doi: 10.1 155553337115.3371199
Association for Computing Machinery. ISBN 978 450377386.
URL ponttps://don wrg. 10, 014111433380 337 58. 3371196
Prabhant Singh, Yiping Li, and Yasmina Al Khalil. Analysis of transferability estimation metrics for surgical phase recognition. In 3rd Workshop in Data Engineering in Medical Imaging (DEMI) MICCA/ Workshop 2025, 2025.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1 1-9, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern ieciognnnttioonCPRP pp. http:/ooppl1:::205550
2015. URLhttps: /api www.ettttiicch.o.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2815-2823, 2018.
Yang Tan, Yang Li, and Shao-Lun Huang. Otce: A transferability metric for cross-domain cross-task representations. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Peter Thul, Lovisa Akesson, Mikaela Wiking, Diana Mahdessian, Aikaterini Geladaki, Hammou Ait Blal, Tove Alm, Anna Asplund, Lars Bjork, and Lisa M Breckels. A subcellular map of the human proteome. Science, 356(6340), 2017.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 10347-10357, 2021.
Anh Tran, Cuong Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In 2019 IEEÉ/CVF International Conference on Computer Vision (ICCV), pp. 1395-1405, 2019. doi: 10. N0.109//CCV22219000044
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Humphrey Shi, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
14

---

Published as a conference paper at ICLR 2026
Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Metu-album: Multi-domain meta-dataset for few-shot image classification. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: //meta -a a lbum github. ic/
Sebastiano Vigna. weighted correlation index for rankings with ties. In Proceedings of the 24th International Conference on World Wide Web, WWW 15, Republic and Canton of Geneva, CHE, 2015. International World Wide Web Conferences Steering Committee. doi: 10.1 10.1145/2736277. 2741088.
Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre-trained models are from neural collapse on the target dataset informs their transferability. In 2023 IEEF/CVF International Conference Computer Vision (ICCV), 2023. doi: 10.1109//CCV510000 2023.00511.
Xiaoping Wu, Chi Zhan, Yukun Lai, Ming-Ming Cheng, and Jufeng Yang. Ip102: A large-scale benchmark dataset for insect pest recognition. In IEEE CVPR, pp. 8787-8796, 2019.
Weijian Xu, Yifan Wang, Xizhou Zhang, Yue Cao, Kai Chen, Chunhua Shen, Han Hu, and Yichen Wei. Co-scale conv-attentional image transformers (coat). In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Yuncheng Yang. Meng Wei, Junjun He, Jie Yang, Jin Ye, and Yun Gu. Pick the best pre-trained model: Towards transferability estimation for medical image segmentation. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, and Russell Taylor (eds.). Medical Image Computing and Computer Assisted Intervention MICCAI 2023, pp. 674-683, Cham, 2023. Springer Nature Switzerland.
Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. Human action recognition by learning bases of action attributes and parts. 201 International Conference Computer Vision, pp. 1331-1338, 2011. doi: 10.1 109/CCCV2201 S.CCV.0011.6126390 1.6126386.
Kaichao You, Yong Liu, Mingsheng Long, and Jianmin Wang. Logme: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, 2021.
URL (Rt/https:/ / /api. semant www.aotcccccllllcc ar org/ Corpus 231 985863
Guanhua Zhang and Moritz Hardt. Inherent trade-off's between diversity and stability in multi-task benchmarks. In International Conference on Machine Learning, 2024.
Tengxue Zhang, Yang Shu, Xinyang Chen, Yifei Long, Chenjuan Guo, and Bin Yang. Assessing
pre-trained models for transfer learning through distribution of spectral components. Proceedings of the AAAI Conference Artificial Intelligence, 39021))2250--22566 Apr. 2025. doi: 10.
1609/aaai.v330 33414. URL http::tp/////....... //ojs/////s/o/./. aaai, orq/ wwwprrggnnneeeeeeee index.nexeec php/ AAAI/ hhtp/AAA//rtticle/ arricce/ view/ 34414
15

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIAL
A SITE BENCHMARK AND EVALUATION CHECKLIST
To aid future research, we provide a concrete checklist for constructing robust benchmarks inspired by NAS Checklist (Lindauer Hutter 2020
Best practices for Building benchmark.
Ensure Diversity (Models should be from different families).
Match computational budgets (Models should under similar parameter range).
Avoid Trivial hierarchies (Models should not be in incremental improvements over each other Inceptionv2, Inceptionv3, Inceptionv4) where one improvement is proven improvement over other.
Include datasets with room for improvement. (Do not include datasets where all scores 99%).
Include datasets from multiple domains.
Engineer for rank dispersion (so different models win in different tasks) to avoid a static leaderboard. If this is not possible then examine if the following task requires transferability estimation.
Best practices for reporting experiments and evaluation
Report Tw with ablation over every model.
Report correlation of Ar and Acc
Best practices for releasing code For all experiments you report, check if you released:
Code for the training pipeline used to evaluate the final architectures.
Code for computing SITE metric for specialized tasks well like object detection and Regression.
16

---

Published as a conference paper at ICLR 2026
B BENCHMARK BASED ON BEST PRACTICES
B.1 STATIC RANKER
The final static ranking determined by sorting the models based on these histograms. model is ranked higher if it has more first-place finishes. If two models have the same number of 1st place finishes, the tie is broken by comparing their number of 2nd place finishes. This process continues lexicographically through all rank positions, ensuring stable and deterministic ordering.
Distribution of Rank Positions Across Datasets (Stacked Rank)
Rock
Purk
DelT
CoaT
Nodd
Twird
MaxVIT
12
14
Count accumences
Figure Visualization of the ranking distribution models the improved benchmark. Most the models share top spots and model is always top bottom rank.
C ADDITIONAL ANALYSIS
C.I TOP-K MODELS ANALYSIS
The practical importance of fidelity is most pronounced when comparing top-performing models. therefore conducted a focused analysis on model pairs involving at least one from top-3 ranks. The results show that even in this critical subset, the correlation between score gaps (Ar and accuracy gaps Ace remains weak and unreliable across all metrics. Large differences in SITE scores between top-ranked models frequently corresponded insignificant gaps in fine-tuned accuracy, confirming that these metrics fail to provide meaningful guidance for selecting the best model. This lack of fidelity for the most relevant models underscores their limited practical value for real-world decision-making We provide a detailed breakdown of this analysis in Figure 6 and Figure
C.2 AGGREGATED PEARSON CORRELATION FOR METRIC FIDELITY
In Table 3, list the average Pearson correlation of every metric between AT and A Acc. this metric can be useful for practitioners to get average idea of fidelity different SITE metrics.
D LISTING FLAWED TABLES FROM PREVIOUS STUDIES
In Table we can also observe that the RoBERTa model always outperforms other models, and BERT-D always underperforms, which causes similar biases discussed in our work. We observe similar benchmark practices here, such as standard benchmarks with multiple different parameter ranges of models coupled together and static leaderboard.
17

---

Published as a conference paper at ICLR 2026
CIAR Aircatt
0.77
0.82
-0.36
0.58
0.22
0.37
0.91
0.
CITAR 100
0.06
Datase
DTD
-0.05
0.22
-0.46
0.01
-0.26
-0.49
rood 101
-0.40
-0.73
-0.48
-0.38
-0.30
-0.38
-0.5
Rets
0.31
H-Score
LogME
NLEEP
SFDA
TransRate
Figure 6: Heatmap correlation of A Acc and Ar for Top-4 Models
Aircrott
-0.34
-0.52
0.35
.0.66
-0.44
-0.56
CITAR 10
05
CITAR 2000
0.23
Dataur
DTD
-0.38
-0.25
-0.13
-0.15
-0.41
-0.99
Food 301
-0.43
-0.36
0.23
0.08
-0.28
-0.42
-0055
Pess
GBC
H-Score
LogME
NLÉEP
SFDA
TransRate
Figure 7: Heatmap correlation of Ace and Ay for Bottom-4 models.
E SCORE VS ACCURACY FOR ALL SITE METRICS
We plot the SITE scores against the achieved ground truth accuracy to provide more detailed picture of the fidelity of the scores the differences accuracy. Ideally, we observe linear relationship such that we can infer from large gap between scores (comparatively) large gap between accuracies.
Figure plots the GBC score against the accuracy. We observe that the distances between scores have no meaningful translation to the obtained accuracies. The majority of scores cluster in very small range, and some outliers obtain visibly distinctive scores. Figure g shows that LogME is able to reflect meaningful distances in accuracy with its score for the CIFAR-10, and Cifar-100 and Food-101 datasets at least for the top performing models. This tendency also observed in the correlation heatmap (cf. Figure NLEEP indicates somewhat linear relationships for the CIFAR-10, CIFAR-100 100, Pets, and Food-101 dataset (cf. Figure For the Aireraft dataset we observe an inverse relationship (the higher the score, the lower the accuracy). For SFDA (Figure 11).
imagine
18

---

Published as a conference paper at ICLR 2026
Table 3: Average Pearson correlation between accuracy differences and metric score differences across datasets
Metric
Avg. Pearson r
SFDA
0.430
NLEEP
0.299
H-Score
0.101
LogME
0.047
GBC
-0.147
TransRate
-0.188
Table 4: Original results from LogME NLP Experiments.
tack
RoBERTa
RoBEKTaDD
uncased DERTDD
cased BERTD
ALBERT-vl
ALBERT-v2
ELECTRA-bus buse
ELECTRA-small
To
MNLI
Loghtt:
Accuracy
40.55
87.6
-0399
$4.0
40003
32,2
4002
81.5
4.04
816
4039
34.5
20A60
858
0.60
QNLI
LogME
Accuncc
0.565
92.8
-0603
908
0613
89.2
0668
88.2
1.00
SST
LogMlt
Accunacy
403122
94.8
40.30
92.5
40331
91.3
4033
90.4
432
903
40477
92.9
0.08
CaLA
LogME
Accuncy
-0,999
63.6
-0.536
59.3
40 0.568
$1.3
40572
47.2
1.00
MRPC
LogMt
Accunccy
90.2
0135
86.6
87.5
40004
85.6
Accunccy
60.6
-0.733
-0.22
linear mappings that fit to most datasets. This is also reflected by comparatively large correlation between the SFDA score and the accuracy in Figure 4 Transrate's 's scores are all over the place (cf. Figure [2], and the H-score exhibits some linear relationships for the Cifar- 10, Cifar-100 and Food-101 datsets (cf. Figure T3
19

---

Published as a conference paper at ICLR 2026
GBC Score Accuracy
(3408 10
(BRAA) 100
85
725
675
-0,0044
-3.300
4.0022
-0.0001
8.000
-66
-35
-2.9
-1.0 -3 35 -1.00
-6.7
-C22 3.33
GBC
MTb
340
22.8
24
74
12
CBC sc0e
03C
CDC SOOrE
Figure 8: Plot of GBC scores against the ground truth accuracy.
Log Score Accurasy
C1A120
0140-00
Absret
10.0
50.5
4
Acseaay 760
72.5
74
m00
615
3.05
1.3
L12
3.34
6,28
3322
0.221
3923
3320
6.02
Lig
Ido
BTD
01.
24
32.8
91.5
Log
L0gM0 scora
Figure 9: Plot of LogME scores against the ground truth accuracy.
F LLM USAGE
LLMs have been used for proofreading, and finding typos.
20

---

Published as a conference paper at ICLR 2026
NLEEP Score Accuracy
0110-10
CEPA-100
Alcroh
12.5
34
18.6
72
175
0.62
3.54
8.06
0.00
C.0
0.2
0.34
0.30
0.22
0.44
0.2
3.3
3.34
NLBEP
Pets
DTD
Feed-L21
76
310
22.8
11.9
71
METER sowe
NLPEP sooe
M.REP-rvv
Figure 10: Plot of NLEEP scores against the ground truth accuracy.
SFDA Score Accuracy
C1A3.20
C1AN100
Aicreh:
Accusayy Th
2
24
700
3.3
585308 31000/3300 000.0000000000000000000000000000 C2300
0.00911
31000
40300
STDA
Fiza
CTD
Te06-10L
05
34.
12.5
32.6
21
015
STON 5c3m
SEDA
Figure 11: Plot of SFDA scores against the ground truth accuracy.
21

---

Published as a conference paper at ICLR 2026
TransRate Accuracy
CFAR10
Alc
24
700
289
304
630
3000
2033
3006
4030
1549
2000
200
2033
2000
Paaskon
Przz
CTD
Food 101
74
310
22.
71
Transknne Scom
Figure 12: Plot of Transrate scores against the ground truth accuracy.
H-Score Score
CPA122
CPAD-220
Altra's
24
70.0
72
30.0
11-cooe
Tdoa
2TD
Foed
75
36
341
32.5
32.6
01.5
71
24
14
It Scom scare
1- W-Score score
115com scom
Figure 13: Plot of H-score against the ground truth accuracy.
22


Reviews:
[{"review_id": "EeSxg6nI6p", "reviewer": "Reviewer_uC4E", "summary": "This paper argues that the standard evaluation protocol for Source-Independent Transferability Estimation (SITE) is misspecified and overstates progress. Unlike Agostinelli et al., this work targets the protocol rather than metric design.\n\nThe authors show three failures:\n\n- **1. Unrealistic model space.** Common SITE pools are dominated by scaled variants of a few CNN families (chiefly ResNet and DenseNet). Removing the largest models sharply drops Kendall’s τ_w across methods, exposing brittleness and a “bigger-is-better” effect that drives rankings.\n\n- **2. Static leaderboard.** A trivial, dataset-agnostic ranking -- argely by model size and family -- matches or beats SITE metrics across datasets. The same few large models (e.g., ResNet-152) repeatedly top the charts; a fixed size-aware ranker attains higher τ_w than all evaluated SITE metrics.\n\n- **3. No fidelity checks.** SITE scores are not tested for decision fidelity -- whether score gaps map to meaningful accuracy gains. Pairwise ∆score correlates weakly with pairwise ∆accuracy, making score differences hard to interpret for cost–benefit trade-offs.\n\nThe paper ends with best-practice recommendations and a checklist for better evaluation.", "questions": "Please address weakness 1 and 2.\n\nOverall, a well-argued, well-executed critique of current SITE evaluation with compelling evidence -- solid work in my opinion. The work would be much stronger if it shipped a concrete repaired benchmark (diversified model pool, harder datasets, and a fidelity metric), rather than \"best-practices\" guidance.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- **Timely and important.** Evaluation design choices meaningfully shape this subfield; focusing on protocol is valuable in its own right.\n\nAll the three findings seem correct and fatal:\n- Ablations show over-represented large models inflate results (Fig. 2).\n- The static-ranker baseline dominates (Table 1) and should be fairly easy to reproduce.\n- The ∆score vs ∆accuracy study addresses an often-ignored need and shows that score gaps frequently fail to predict accuracy gains (Fig. 4; App. B).\n\nIt's correct and likely impactful on the field of source-independent transferability estimation.", "weaknesses": "**[Critical] Could the paper implement the fixes recommended instead of best practices?**\n\nThe critiques are actionable, yet the paper stops short of fixing it. Best-practices section seems unsuitable as it reads as advice masquerading as work that is not done.\n\n- If the pool over-indexes ResNet/DenseNet, please propose and release a diversified model space (include ViTs and other modern families). Alternatively consider a two-stage aggregation: Compute SITE within each architecture family, then aggregate fairly across families. But a new benchmark, if proposed, instead of simply this critique could be widely adapted.\n\n- If a static ranker “solves” current datasets, expand the dataset suite (e.g., include harder, real-world transfer targets such as those discussed in *Does Progress on ImageNet Transfer to Real-World Datasets?* by Fang et al.). Leaderboard suites from *Benchmark suites instead of leaderboards for evaluating AI fairness* by Wang et al. might also be valuable.\n\n- For fidelity, propose an actionable fidelity metric grounded in decision quality (∆score vs ∆accuracy study already seemed promising). To provide some straws in the wind -- social-choice theory offers rank-aggregation tools with known axioms and trade-offs; some classical criteria may fit SITE’s needs.\n\nIf fixed, the resulting benchmark could be potentially quite valueable for the next generation of SITE algorithms.\n\n**[Major] Central findings #1 and #2 already seems well-known.**\n\n- The strong performance of dataset-agnostic rankings (by size or ImageNet top-1) echoes established findings -- e.g., *Do Better ImageNet Models Transfer Better?* (Kornblith et al., CVPR’19) and *Do ImageNet Classifiers Generalize to ImageNet?* (Recht et al., ICML’19). Could the work clarify what is new beyond confirming known transfer trends.\n- Lots of similar work: *Back to the Basics on Predicting Transfer Performance*, *Scalable Diverse Model Selection for Accessible Transfer Learning* (relevant to finding #1) are not cited or differentiated from. Could the authors detail the core differences to past work?\n\n**[Minor] Reporting and metrics.**\n\n- How large are the uncertainty bands (confidence intervals and significance tests) for τ_w to indicate whether observed differences are meaningful.\n- The work seemed to need more polishing to concisely state the message. Please confirm whether my understanding and critiques are correct, I had to read this multiple times to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reACr9xAyH", "reviewer": "Reviewer_GpHM", "summary": "This paper provides an empirically-backed critique of existing evaluations for SITE (source-independent transferability estimation). This problem concerns how to estimate which of a pool of models will perform best on a target domain after finetuning. The paper makes several well-received points: existing evaluations tend to use overly simple pools of models and datasets. As a result, naive baselines like a hand-crafted static ranking of models can outperform highly sophisticated SITE methods. Popular ranking metrics also tend to understate how poorly SITE scores reflect actual differences in finetuned model accuracies (on an absolute scale, not just in terms of rank ordering). The paper offers several constructive recommendations for how to improve evaluation in the space. I believe the strongest point is that evaluations must capture a meaningful degree of variance in finetuned model performance across different domains, i.e. rank orderings of finetuned models on target datasets should show a meaningful degree of variance that SITE metrics should then explain.", "questions": "- please feel free to comment on the noted minor weaknesses/concerns above\n- typo: as two scorings with induce the same order can differ in their evaluation merely due to calibration", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Very important: The paper is well written and very clear.\n- Very important: The paper’s experiments get straight to the point and effectively undermine the validity an existing, popular approach to SITE benchmarking.\n- Very important: The recommendations are fairly actionable and should push work on this problem in the right direction.", "weaknesses": "- Of some importance: I am not deeply familiar with work on SITE and so a small concern of mine is that this paper is strawmanning what is a small and not overly influential line of research. Yes, I agree that these papers are being published in top venues. But is this really the mainstream way that people think about capturing low-dimensional properties of AI’s adaptability to downstream tasks? Forgive me for comparing this problem area to LLM evaluations, but I think what people tend to put stock in these days is a small set of benchmark scores for an AI system. A set of 3-4 benchmark scores is taken as gospel for which model is most “intelligent”, and therefore going to perform best when tailored to a specific downstream task. Sure, there is clever ML work going on to understand properties of vision model representations and how they relate to downstream performance. But this is not the main thing I have in mind when I think of how people assess broad transferability of AI systems.\n- Of minor importance: I had a small quibble with this point: “practitioners are not interested in knowing whether they should use a ”bigger vs. smaller” ResNet, but which architecture performs best under a fixed size budget.” I think this point is stated a little more elegantly in the recommendations section: “To isolate architectural inductive bias, models should be matched on computational budgets (e.g., parameter counts or FLOPs).” In reality, practitioners care about a lot of things, including size, training speed, inference speed, ease of access to models, interoperability with common training and deployment libraries, etc. I think the claim in the recommendations section helps point out what kind of model pool would lend support to scientific conclusions about predicting downstream adaptability of different model architectures. Still more consideration might go into constructing a model pool that reflected all the various axes of variance that practitioners care about (which is more than just performance vs. size).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "scJWYE8gBs", "reviewer": "Reviewer_fF5c", "summary": "The authors propose a deeper look into the problems in the assumptions made for evaluating transferability estimation. They show how the the status quo benchmark framework (datasets, models, evaluation protocols) is faulty will 3 concrete emperical critiques and make concrete suggestions for good practises that future works can focus on.", "questions": "1. How does the fidelity affect top models? Practitioners generally care about the top performing models, so I would be curious to see how $\\Delta$Acc - $\\Delta$T correlation for model pairs where at least one of the models is in the top-k or the bottom-k. This would tell us if fidelity fails in the current evaluation setup for important models.\n\n2. Re-iterating from W4, does the static leaderboard still outperform SITE metrics for granular sample-level datapoints?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-motivated problem statement: shifting from SDTE to SITE is timely and important as we know less and less about the true distribution of training data. Also, with scale, distribution matching will only get more expensive.\n\n2. Probing into the correlation between the difference in magnitude of accuracy and tranferability is a critical analysis the paper did quite well. With more fine-grained studies, it can be quite insightful to practioners.", "weaknesses": "My main issues with this paper revolve around the lack of extensive experimentation: to make the strong claim that a form of benchmarking is \"fundamentally flawed\" requires significantly more analysis than currently provided. \n\n1. Even though the paper prescribes including ViT, ConvNeXt, MLP-Mixer etc, the core experiments to determine the core claim does not extend beyond the standard set of models (ResNet, DenseNet, etc). Though intuitive, we need emperical evidence that the three critiques will hold true with a diverse pool of models.\n\n2. Related works such as [1] have identified similar problems, such as \"bias toward high-capacity models\" and the unrealistic model search space, in addition to other relevant problems this work has not addressed such as the lack of fully labelled target datasets in real-world scenarios. They propose solutions such as adding self-supervised models and evaluating transferability estimation on model weights before and after optimisation using the Wasserstein Distance(even though their analysis focusses on SDTE, both these solutions are applicable to SITE). Existing literature already extends the scope of this work by providing concrete solutions to fix the evaluation setup, calling the novelty of this work into question.\n\n3. Finetuning needs to be more vigorous: Small changes in seed, optimizer, number of epochs, etc can produce statistically significant variance in model performance. The paper does not but should report independent fine-tuning sweeps and checkpoints and report mean &plusmn; standard deviation to check for this sensitivity. Also, the same seeds should be applied to all models and model families.\n\n4. I disagree with the claim that larger models naturally outperform their smaller counterpart. Recently, [2] has shown that sample-level benchmarking showcases interesting scenarios where older versions of the same model or smaller models of the same family outperform their newer iteration/larger size counterpart for specific subsets of the evaluation datapool. This brings into question whether the static ranking based on model size is robust to more specific data subsets.\n\n5. The NLP results feel like a footnote in the appendix. It should be as rigorous as the image classification experiments. In general, for a paper claiming transferability estimation metrics are flawed and that the insights are applicable to other domains and tasks, it is in the scope of the work to demonstrate that. Related works such as [3] do show analysis on medical image classification.\n\n\nMinor points:\n1. L137: using the same notation n for (T_n) and data (x_n, y_n).\n\n[1] Kazemi et al, Benchmarking Transferability: A Framework for Fair and Robust Evaluation, 2025\n\n[2] Ghosh et al, ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities, ACL 2025\n\n[3] Chaves et al, Back to the Basics on Predicting Transfer Performance, 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7R9cRV1ov", "reviewer": "Reviewer_ULr8", "summary": "This paper discusses the evaluation of metrics for Source Independent Transferability Estimation. These metrics attempt to predict pretrained models' downstream performance after finetuning on a given dataset. The paper raises three points of criticism aimed at current evaluation practices based on a small set of models and datasets: First, the set of included models is often unrealistic and dominated by differently sized versions of a small set of architectures. Evaluations results are heavily dependent on the set of included models. Second, a heuristic static model ranking outperforms all evaluated methods. Third, current evaluation metrics focus on rank correlations, partially masking failures to predict the magnitude of performance differences.", "questions": "How exactly was the static ranking chosen? The paper says \"based on the model size and an alternation of ResNet and DenseNet model families\", but there seem to be two densenets followed by two resnets at ranks 4-7.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The \"static ranking\" experiment relatively convincingly demonstrates that the current evaluation setup for SITE metrics is flawed", "weaknesses": "- The paper could be strenghtened by producing an actual setup that mitigates the discussed issues.\n- The writing could be improved in various places. \n   - For example, the discussion mentions a bunch of datasets without citations \n   - Some claims in the paper are also unsupported (by either citations or arguments). Example: \"In addition, SDTE metrics are not reliable when the discrepancy between the source and target datasets is very high\"\n    - As a minor nitpick, the paper could also benefit some additional proofreading in terms of grammar/sentence structure. For example: \"For purpose of this study we examine the following metrics on standard benchmark\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)