Implicit Modeling for Transferability Estimation of Vision Foundation Models
Authors: Yaoyan Zheng
Huiqun Wang
Nan Zhou
Di Huang
Venue: NeurIPS 2025 poster ()
Keywords: Transferability Estimation
Model Selection
Primary area: deep_learning

Abstract: Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning. This capability facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model’s intrinsic transferability, coupled with a Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables generalization across a broader range of models and downstream tasks. Extensive experiments on a comprehensive benchmark—spanning extensive training regimes and a wider variety of model types—demonstrate that ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency.

Implicit Modeling for Transferability Estimation of Vision Foundation Models
Yaoyan Zheng
Huiqun Wang Nan Zhou
Di Huang"
State Key Laboratory of Complex and Critical Software Environment,
Beihang University, Beijing. China
School of Computer Science and Engineering.
Beihang University, Beijing, China
fyaoyanzheng, hqwangscse, zhounan0 0431, dhuang] @buaa edu on
Abstract
Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning, This capabil- ity facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model's intrinsic transferability, coupled with a Divide-and- Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables gen- eralization across a broader range of models and downstream tasks. Extensive experiments on a comprchensive benchmark spanniing extensive training regimes and wider variety of model types demonstrate that ITM consistently outper- forms existing methods in terms stability, effectiveness, and efficiency. Code is available htttss//gggabbbcomm//U/AAHguuu
Introduction
With the success of the pre-training and fine-tuning paradigm, substantial number of pre-trained models are publicly available. However, recent studies [1] reveal that their performance varies significantly across architectures, datasets, and pre-training strategies, posing increasing challenges identifying the most suitable models for downstream tasks.
Transferability Estimation (TE), which predicts the performance ranking of pre-trained models on downstream tasks with minimal computational cost, attracts considerable research attention as an efficient solution to model selection. Unlike the computationally prohibitive brute-force approach of individually fine-tuning each model, TE methods aim develop efficient metrics quantify model suitability. Early approaches estimate the compatibility between pre-trained models and downstream tasks by measuring divergences in logit spaces 2 or embedding spaces
5), while more recent methods improve upon static estimations by simulating the dynamic evolution of embedding spaces during fine-tuning to enhance predictive accuracy 7
However, most existing TE techniques are primarily evaluated on models with similar architectures and training paradigms (e.g., supervised pre-trained CNNs) and fail to generalize models trained with advanced pre-training strategies or novel architectural designs. When applied to ViT [8) models developed using techniques such Instance Discrimination (ID) 9 10) and Masked Image Modeling further complicate unified
(MIM) I| 12), the distinct convergence behaviors exhibited
Cooresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Batched Data
E UEJ
B = UE
Embedding
Approximate Torget Space E
Pre-trained Model
Pre-trained Downstream
Space E
Batch-wise Division
Divide and Conque
Model
Data
Evolution
Simulation
Transferability: Z
(a (E(
(a) Previous Dynamic TE Methods
(b) ITM (Owrs)
Figure 1: Comparison between previous dynamic TE methods and the proposed ITM. (a) Previ- ous methods simulate the entire embedding space by hand-crafted rules and estimate transferability based on the approximated target space. (b) ITM implicitly models transferability z and simulates subspace evolution using a divide-and-conquer strategy for more accurate estimation.
estimation, leading to notable declines in accuracy and unreliable model selection. Mcanwhile, these methods are predominantly designed for image classification tasks and lack adaptation mechanisms for downstream task heads, thereby limiting their applicability to more complex scenarios such as semantic segmentation.
The performance of a pre-trained model on downstream tasks depends on two key aspects: (1) its intrinsic properties, such as architecture, pre-training data sources, and strategies; and (2) the characteristics the downstream tasks, which impose varying demands on model-specific capabilities. The interplay between these factors produces unique adaptation dynamics for each model, resulting in different levels of transferability across scenarios. A Although previous studies attempt to model these dynamics by directly simulating the evolution of embedding spaces, they fail to comprchensively capture both aspects, thereby limiting their generalization capabilities.
To address these limitations, we aim to incorporate an implicit modeling of a model's transferability into the simulation of embedding space evolution by using small number of learnable parameters, enabling a more comprehensive assessment of both model properties and task adaptability. However, achieving this goal presents two major challenges: (1) implicit modeling requires knowledge of the final embedding states after fine-tuning, which is impractical for direct estimation; and (2) modeling the evolution of candidate models across downstream tasks is computationally intensive, as adaptation dynamics vary significantly across models and tasks.
In response these challenges, we propose the Implicit Transferability Modeling (ITM) paradigm. As illustrated in Fig. ITM decouples the transfcrability of pre-trained models and encodes them through an implicit latent representation, enabling the approximation of embedding space evolution without requiring explicit simulation. To achieve feasibility and computational efficiency, we introduce Divide-and-Co and-Conquer Variational Approximation (DVA) that partitions the embedding space into subspaces and streamlines their evolution modeling. By integrating these components, ITM enables more precise and scalable transferability estimation across diverse architectures and pre-training strategies. We benchmark I'TM on recent models following more complete training regimes across ten widely used downstream tasks and conduct comprehensive comparisons against state-of-the-ae TE methods. Experimental results show that ITTM consistently outperforms existing approaches, achieving stable and substantial gains in estimation accuracy and generalization ability.
Our contributions are summarized as follows:
We introduce the Implicit Transferability Modeling (ITM) paradigm for accurate and gener- alized transferability estimation across diverse architectures and pre-training strategies.
2. We propose Divide-nnd-Conuee Variational Approximation (DVA) strategy efficiently model embedding evolution while minimizing computational costs.
3. We achieve state-of-the thet performance in transferability estimation, demonstrating substan- tial improvements across ten downstream tasks and ten recent models trained with various pre-training methods.

---

2 Related work
2.1 Transferability estimation
Distinct from conventional model selection strategies based on traditional fine-tuning and its variants, such pruning [1, task-head tuning [3], or model encoding [14], Transferability Estimation has emerged as a promising research direction, distinguished from prior approaches by its computational efficiency and simple, plug-and-plan design. Based the type of estimation method, recent TE 16
approaches [2 5 15
22 23 (24] can be categorized into two primary realms: static statistical methods and dynamic evolution- based methods.
6 17
18 19
20 21
Static statistical methods assess transferability by analyzing the statistical properties of the em- bedding spaces of pre-traaee pre-trained models. NCE 2 and LEEP (3] estimate logit discrepancies between model outputs and downstream task annotations using conditional probability or Bayesian metrics. NLEEP [25] improves upon LEEP by replacing the output layer with Gaussian Mixture Model (GMM) to enhance efficiency and calibration. Log.EE [4) leverages the logarithm of maximum evi- dence to provide stable predictions at lower computational cost. ETran [5] combines multiple metrics, introducing an energy-basee based measure to improve estimation accuracy, while GBC [13] employs the Bhattacharyya coefficient to evaluate class separability the feature space. These static approaches climinate the need for complex parameter updates, thereby achieving greater computational efficiency. However, by neglecting the embedding evolution that occurs during fine-tuning, they fail to capture full adaptation dynamics, ultimately limiting their prediction accuracy.
Dynamic evolution-based methods simulate aspects of the fine-tuning process through mapping functions or learning frameworks to achieve more accurate transferability estimation. PED 7 introduces potential energy-based update model predict the evolved state. LEAD 6 employs ordinary differential equations and downstream objectives to better capture the evolution of logits during adaptation. SA [22) perturbs the feature space through spread and attracts operations to approximate the robustness exhibited during fine-tuning. Although these methods represent significant progress, the recent proliferation of pre-trained models with diverse architectures introduces greater discrepancies in initial states and convergence behaviors. Existing dynamic approaches overlook the intrinsic properties of pre-trained models and focus solely on simulating embedding space updates, thereby limiting their ability to provide comprehensive and accurate transferability estimation.
2.2 Pre-trained models
The pre-training and fine-tuning paradigm leverages large-scale dataset knowledge for better perfor- mance on downstream tasks. Recent work further expands it by exploring diverse architectures and pre-training strategies, resulting in substantial progress and increasingly varied model capabilities.
Model architecture. Early applications demonstrate the paradigm's 's effectiveness on traditional convolutional neural networks (CNNs) such as ResNet [26] and DenseNet [27]. Compared to training from scratch, these CNNs learn rich prior knowledge from large-scale datasets and achieve superior downstream performance. More recently. Vision Transformers (VITs) 8 and Swin Transformers (28 introduce transformer architectures [29] into computer vision. Unlike CNNs, ViTs better capture rongerange dependencies and global context, exhibit distinct convergence behaviors, and achieve improved performance on complex downstream tasks.
Pre-training strategy. Early pre-training efforts adopt a fully supervised approach, with Ima- geNet [30] as the dominant dataset to boost model performance. However, reliance on dense human annotations limits scalability and broader applicability. More recently, contrastive learning strategies such as SimCLR [1 32 and MoCo B3
34] leverage large-scale unlabeled data, enabling models to acquire richer prior knowledge and achieving strong transfer learning results. In parallel, masked image modeling approaches like MAE [1] and SimMIM 12 reconstruct masked input regions, delivering tfate-of-the-ar the performance across variety of downstream tasks.
Despite these advances, the growing diversity of pre-training strategies and model architectures leads to pre-trained models with significantly differing characteristics 0. posing challenges for existing transferability estimation methods. This highlights the need for more generalizable and robust TE approaches capable of handling broader range of models.
3

---

3 Methods
3.1 Problem formulation
Consider a given model collection + 1 where the models @; have different architectures and are pre-trained using distinct strategies. For ground-truth model ranking, every model is fine-tumed on the downstream training set Dr and evaluated on the test set Dg. The goal of transferability estimation is to predict the performance ranking of the collection of models by assigning metric score 8; to each model di, based on the extracted features E, from d, and the labels Y of the downstream dataset, thereby quantifying each model's suitability for downstream tasks.
To evaluate the ranking correlation between the predicted scores S {s and the true perfor- mance R (r:) M we follow prior work 6 7 4. adopting weighted Kendall's Tue the metrie: Tu
Wij sign[(G; C,) P3)]. where G,, P. E [1, M] denote the ranks of the
i-th element in R and S, respectively, and way = G, assigns weight based on the importance of the model pair. The value of weighted Kendall's Tw quantifies the consistency between the predicted and true rankings. Higher values indicate stronger alignment and better transferability estimation performance, thereby enabling more reliable model ranking in real-world application scenarios.
3.2 Implicit modeling of transferability
When adapting pre-trained model Φ; to a downstream task, its final performance depends on both the intrinsic properties of Φ; and the characteristics of the downstream task D. The fine-tuning process can be viewed as adapting model from prior knowledge K. learned from the pre-training data, to task-specific knowledge K. on downstream data (35), governed by an intrinsic attribute (KC, K.) that determines transferability. Recent dynamic TE approaches primarily predict performance by modeling mapping [(bb Dr) E - E, where E is the original embedding space of d on Dr, and É is its approximate state after fine-tuning. However, methods relying on handcrafted rules or idealized assumptions only partially capture the knowledge transfer process and struggle to generalize across the growing diversity of model architectures and pre-training strategies.
To address this, we propose implicitly modeling of the transferability variable for each model-task pair, rather than directly modeling -):-)) as in previous work. The post-fine-tuning embedding space posterior distribution q( [E|E, 2), transforming the complex mapping into
É is represented probabilistic estimation framework. Building on this formulation, we introduce a Divide-and- Variational Approximation (DVA) to efficiently approximate the final embedding after full fine-tuning, enabling more accurate and scalable transferability estimation across diverse pre-trained models.
3.3 Divide-and-conquer variational approximation
To enable effective transferability estimation with implicit modeling, we propose Divide-and Conquer Variational Approximation (DVA) to approximate the evolution of the embedding space.
Batch-wise division. Given the initial embedding space E, a parameterized module ta aims to approximate the ideal mapping I'(s--) by maximizing the posterior distribution go E, =), where 2 represents the latent variable capturing the global characteristics of the pre-trained model 6 on the downstream dataset D. Based on the principle of batch-shuffled independence assumptions [36] 37 [3], we treat the evolution of each subspace independently. Benefiting from this paradigm, we mitigate the entanglement of the global embedding space, enabling a more flexible and architecture- agnostic modeling process. This stands in contrast to prior TE methods that attempt to model the evolution of the entire embedding space, which is often highly entangled and architecture-dependee To this end, we partition the embedding space into a collection A comprising K subspaces, and approximate the overall mapping by independently modeling each subspace with & as follows:
qu E 2) = (
Ex (x))(( (E,, Ex),2)
(1)
qv E E),2). j=1

---

()) Full F Finetune Pipeline
(b) Framework of ITM
(c) Training Process within Batch
Pre-trained Models
D
Dg
P
By R
Head
(CE:W.))
Fine--tuning
B E Je RxC
(JL,
Approsemation
Ej
DA
creudo-clusteer Center
Head'
Head
Fine-tuned Models
Validation
Lye
MMGG
(5g Sg.
Tosk Heod
Testing
Loog
(F's FM
Correlation
Logits *....
Y 1
Pre-trained
Fine-tuned
Initialized
Parameter-free
Frozen
Learnable
Figure 2: Illustration of the proposed Implicit Transferability Modeling (ITM) paradigm. (a) Ground-truth model ranking. (b) Overview of ITM, which approximates embedding space evolution via Divide-and- mini-batch: the transferability is integrated into E, to form the posterior condition θ via W.. which is jointly optimized with the downstream task head through the downstream objective Lobj-
Variational Approximation (DVA). (c) Detailed view of DVA on a single
Thus, the posterior go [E|EE z), originally defined over the full embedding space E, can factorized into a series of posterions gu (E,l E, corresponding to subdivisions of E, where E, E A. This reformulation enables the modeling of the entire embedding space evolution E > E as the evolution of set of subspaces (E, - E, K In practice, define each subspace Ey the embedding representation of a mini-batch DB p- In subsequent evaluations, we find that different batch sampling strategies do significantly affect model convergence under mainstream protocols.
Since explicitly modeling = requires solving an intractable inverse problem in a high-dimensional space, and z is coupled with E, as part of the posterior condition in qu E, 2), we introduce conditioning mapping, f(; W:) with learnable parameters W, to embed the latent representation 2 into each batch of pre-trained embeddings. Based on this, the posterior is transformed into
By EE ))),, where θ [EE; Wz), enabling end-to-end optimization of the integration of through the evolution of each subspace.
Pseudo-cluster center generation. Given the posterior conditioning θ, the final state Ey is required to learn the evolution mapping 1 O - É,,. However, obtaining the true E, impractical, as requires full fine-tuning of the model on the downstream task, which is computationally unaffordable. To address this limitation, we leverage findings that modern pre-trained models exhibit strong convergence capabilities on training data [36] (39. resulting in well-separated distributions in the embedding space after convergence. Specifically, representations of different classes tend to form distinct clusters, reflecting the static structure of the final embedding space.
Based on these inferences. we adopt a pseudo-cluster center generation strategy mimic the static distribution separability of the evolved embedding space. The pseudo-cluster centers {c are designed to ensure well-separated structure in the final embedding space. They can be generated using one-hot vectors, random vectors sampled from high-dimensional spaces, eigenvectors obtained through Principal Component Analysis (PCA) or Laplacian- bassdd methods to promote sparser distributions. Additionally. shifting strategy based on the statistical properties (μ, a) of the initial feature distribution E is applied to further accelerate model convergence. Finally, we utilize the pseudo-cluster centers as the final state È {c,, y E Y} for subsequent estimation. detailed comparison of clustering generation strategies provided in the appendix.
5

---

Deparametric approximation. After dividing the embedding space and setting the final state, our goal is to learn the mapping v ; Ej. between the synthesized condition 3 and the final state E, estimate the embedding space after fine-tuning. straightforward approach would involve optimizing W separately within each mini-batch through iterative updates using the function g ;: W g). However, performing such per-batch optimization across all subspaces during training and validation would incur prohibitively high computational costs, and is also difficult to implement within existing frameworks.
To address this, we propose a deparametric approximation of the embedding space evolution, lever- aging dynamic equation-base bassd formulation eliminate the need for explicit optimization of W,. This allows efficient transferability estimation while preserving the underlying adaptation dynamics.
Specifically, we consider mapping layer parameterized by W.. g. which is used to model the posterior distribution of E by transforming the synthesized condition O into its final state Ej. This transformation at iteration is expressed as E θ w where denotes the n-th update step of W g. As representative example, considering the MSE loss as Lpc. the loss function over mini-batch of size B can be formulated as:
Cpc (W()) = 2B
E
1
Tr((O, w( -, E2)).
B
Thus, the gradient descent update rule for W, becomes:
aCpc 0 O w (1) - ,, EE]
aw!n] B W W(n)
dc
B
aw 1)
where 77 denotes the learning rate. The corresponding evolution of the subspace E)")
follows:
" O Oc pC
E, (x+1)) w(n)
g
ow!#)
(4)
(I 0 0 )
e w(n) (n)
B
B
where define the constant matrix C n 0,, which depends only the initial input condition θj. Thus, the update can be rewritten as:
E, (++1) = CCj (x) CE
(5)
This deparametric approximation removes the dependency on the learnable parameters W and enables transferability estimation without iterative updates for each subspace, significantly reducing computational overhead while preserving estimation accuracy.
3.4 Framework
Building on the proposed Divide-and aandCCoquee Variational Approximation (DVA), the full ITM
framework is constructed as illustrated Fig. 2 During estimation, the latent transferability is implicitly embedded into each subset of the pre-trained embeddings E, to form θ, via ʃ(; W.). Then, the deparametric approximation update performed based on the synthesized posterior condition O and the pseudo-cluster for the specific downstream task.
To maintain compatibility with diverse downstream tasks, we incorporate the downstream objective Loby into ITM's ss optimization, as illustrated Fig. ()). This ensures the learned embedding evolution to align with task-specific requirements. The objective Lotj typically takes the form of cross-entrops loss for standard classification tasks, but alternative formulations can applied as needed for other scenarios, making ITT' broadly applicable beyond classical classification and enhancing its generalization potential. During evaluation, we assess the accuracy of the evolved embedding spaces on the evaluation set De as the estimated score. The training procedure is detailed in the appendix.

---

Experiments
4.1 Benchmark
6 we construct a benchmark based on 10 classic single-lahel image classification datasets. However, with the rise of foundation visual models and improved training protocols, existing benchmarks may not fully capture model capacities. For comprehensive comparison between ITM and state-of-the-art methods, we select 10 recent pre-trained vision models and fine-tune them using integrated and enhanced protocols. The downstream datasets, pre-trained model zoo, and fine-tuning procedures are detailed below. By integrating the models from prior benchmarks, we develop an extended benchmark of 20 models, which is detailed in the appendix.
Following prior work 17
7
Downstream datasets. We use 10 widely adopted single-label image classification datasets for transfer learning. primarily sourced through the official PyTorch [40] (c.g.. prrchiision.datasettt CIFAR-10, CIFAR- 1000 44), FGVC Aireraft
42
Caltech- 101
43 DTD 44 Oxford-IIII Pets (45).
Stanford Cars [46). SUN-397 (47,, Food-101
48 and Oxford 102 Flowers [49] These datasets span
diverse scenarios, with varying class counts and dataset sizes, offering comprehensive evaluation of the generalization ability of vision foundation models.
Pre-trained model zoo. Prior TE methods largely focus on models with similar architectures and supervised pre-training, limiting generalization to newer models adopting diverse architectures and self-supervised strategies. To cover common downstream
cases, select 10 representative models across supervised, contrastive, and masked image modeling (MIM) pre- are-trainin..
Specifically, we include ResNet-18 [26]. MobileNetV2 [50], EfficientNet-BB -OO [5T], and DenseNet- 121 27 for supervised learning; DINO-S8 TO], DINO-B16 [O]. and MoCov3-B16 [9] for con- trastive learning; and MAE-B16 I1. MAE-LI6 [1. and SimMIM-B16 16 [12] for MIM. Here, S/B/L denote small, base, and large ViT
g models, while 8/16 refer to the patch size.
Fine-tuning protocols. Fine-tuned performance is critical for accurate TE ranking. However, settings used in prior work 117 4 often underexploit modern models, introducing evaluation bias. To address this, we follow official implementations [10 51 26 [2] and adopt Adam W [52) to jointly fine-tune backbones and classification heads for 100 epochs, Learning rates are grid-searched searched over 10-5
10 5 2 x 10-5,5 5 x 10-5 and weight decays over 10 10 Evaluation is performed every epoch, and the best checkpoint is used for ground- ranking. All experiments use single NVIDIA V100 GPU (32 GB), with batch sizes of 64 for classification and 8 for segmentation. Further fine-tuning details and performance comparisons are provided in the appendix.
4.2 Implementation details
To balance accuracy and efficiency, we set the ITM training iterations to 500. The transferability score is evaluated every 100 iterations, with the highest score recorded the final estimation for each model. During training, 4/5 of the official training split from each benchmark is randomly sampled for optimizing the ITM framework, while the remaining 1/5 is reserved for score calculation. The step size r in DVA is fixed at 0.01, and the iteration count n determined adaptively (details provided in the appendix). Across all experiments, the learning rate is adopted as the optimizer. All comparisons are conducted under a consistent environment with 8-core CPUs to ensure fairness. Pre-trained models, benchmarks, and code will be publicly released to facilitate reproduction and future research. In the efficiency comparison, we exclude the basic time of feature extraction, following previous works 7 6), to provide a clean comparison of the running time of the TE process.
5 x 10-3 and AdamW [52)
4.3 Comparison with previous approaches
Benchmark comparison. We evaluate the proposed ITM against state-of-the-art methods on the
benchmark, with results summarized in Tab. [
As shown in Tab. I ITM significantly outperforms all counterparts in weighted Kendall's Tu- achieving a substantial performance margin. While existing methods struggle to generalize to newer architectures due to the growing diversity of model collections and increasingly complex convergence behaviors, ITM leverages implicit transferability modeling to deliver more accurate estimation and stronger generalization capabilities. Specifically, ITM achieves the highest average
7

---

Table 1: Comparison of weighted Kendall's and wall-clock time (in seconds) across different methods on the benchmark datasets. Each wall-clock time does not include the average feature extraction time of 738 seconds (on GPU), but only corresponds to the running time of the TE method (on CPU).
Methods
Cal101
Cars
CIFAR100
CIFARIO
DID
Aireraft
Flowers
Food
Pets
SUN
Avg.
Weighted Kendall's Tw
NTLepp
25
0.47
0.04
0.32
0.48
0.57
0.13
0.62
0.24
0.30
0.01
0.32
LogME
0.61
PARC
18
0.08
0.00
-0.07
0.25
0.42
0.12
0.62
0.19
0.10
0.01
0.17
SFDA 17
0.59
0.07
0.48
0.79
0.13
0.18
-0.39
0.33
0.25
ETran
[
0.13
-0.06
-0.14
0.21
0.36
0.27
0,08
0.23
0.38
-0.06
0.14
PED ?
0.32
-0.01
0.51
0.77
0.06
-0.20
0.16
0.60
-0.20
0.07
0.21
SA (LDA)
22
0.31
-0.11
0.34
0.22
0.16
ITM (Ours)
0.56
0.61
0.59
0.69
0.77
0.44
0.65
0.44
0.73
0.62
0.61
Wall-clock
Time (s)
NLcep
25.52
44.91
862.49
208.25
4.60
17.65
3.42
1387.65
47.33
266.61
LogME
0.85
1.32
4,50
2.62
0.50
0.86
0.54
6.29
0,53
1.31
1.93
PARC 18
14.42
19.82
116.65
118.19
0.74
13.14
106.19
41.27
SFDA
17
4.02
5.43
20.11
18.56
1.70
3.92
1.91
28.86
2.20
5.43
9.21
ETran
1.71
2.30
7.58
6,88
0.77
1.64
0.98
10.63
0.96
2.31
3.58
PED 7
6.99
8.31
34.32
46.80
2.33
5.95
2,80
41.72
3.94
8.29
16.14
SA (LDA) 22
4.65
6.08
10.29
9.04
2.92
4,48
4.11
13.82
3.16
6.08
6.46
TTM (Ours)
100
1/6
1.0
ITM (0urs)
ITM (Ourz)
ITW (Ours)
ITM (Ours)
ITM (Ours)
ITK (Oura)
-0.9
-000
NLEEP
LogME
PARC
ETran
SFDA
PCD
ITM (Ours))
(Ours)
IT (Ours)
ITM
2
BLEEP
LogME
-0
Flowers
Food
Aug.
Figure 3: Comparison of stability between ITM and state-of-the-art methods. The axes show the weighted Kendall's Tw scores of each method. Points appearing above the dashed line indicate cases where ITM achieves higher estimation accuracy than its counterparts.
weighted Kendall's Tw (0.61 vs. 0.45), demonstrating clear improvements over previous state-of-the. art methods and validating its effectiveness. We further rerun ITM with five different random seeds, resulting score of 0.60 + 0.01, reflecting its stability and robustness across varying initializations.
In terms of running time, ITM remains highly competitive. It requires only 8.42 seconds on average to evaluate across the benchmark, which is negligible compared to the 738 seconds needed compute the initial embedding spaces shared by all TE methods. Although slightly slower than LogME and ETran, ITM outperforms both by large margin in accuracy (0.61 vs. 0.45 and 0.14), achieving better balance between accuracy and efficiency.
Stability evaluation. To further evaluate ITM's stability, conduct additional experiments following the methodology of [2I]. We include four classical CNN models, as in prior studies 77 17, to expand the benchmark and better assess generalization across architectures. In each trial, 10 models are randomly sampled from a pool of 14, and weighted Kendall's тu is computed to evaluate the performance of each TE method. This results in 1001 combinations (C12), providing robust measure stability. Fig. compares ITM against existing approaches.
As shown in Fig. ITM consistently outperforms other TE methods in wide range of experiments, demonstrating superior stability in the selection random subsets of models. These results underscore the strong generalizability and robustness of ITM.

---

0.60
120
Weightdd Bu
Running Time [6]
100
Weight ted Lau
Weighted
0.45
bs-4
0.40
bs-18
beeM
0.35
b5-
20-
100
200
500
1000
16
64
256
3024
4006
Training Iterations
Batch Size
(a) Weighted Kendall's Tee
(b) Running Time vs
Figure 4: Ablation study on batch size. (a) Weighted Kendall's Tu across different combinations of batch size and iteration count. (b) Impact of batch size on weighted Kendall's Tw and running time.
Table 2: Quantitative ablation study evaluating different candidate loss functions for Le Losses include Mean Absolute Error (MAE), Cross-Entropy (CE), and Mean Squared Error (MSE).
Lg Cal101
Cars
CIFARI00
CIFARIO
DTD
Aircraft
Flowers
Food
Pets
SUN
Ta
CE
0.717
0.483
0.573
0,689
0.602
0.323
0.582
0.437
0.619
0.519
0.554
MAE
0.717
0.458
0.590
0.689
0.629
0.377
0.575
0.437
0.714
0,470
0.566
MSE
0.560
0.608
0.590
0.689
0.768
0.428
0.651
0.437
0.734
0.616
0.608
4.4 Ablation study
Batch division. Batch-wise division controls the approximation granularity in DVA's posterior decomposition and critical to ITM's performance. We study the impact of batch size and iteration count on weighted Kendall's Tw and rumning time, shown in Fig. 4
From Fig. we observe that batch size increases from to 1024, accuracy improves consistently, since larger batches better estimate transferability. However, further increasing batch size to 4096 slightly degrades accuracy while significantly increasing running time, likely because an overly large batch overwhelms DVA and hampers convergence. Balancing efficiency and accuracy, we set batch size to 256 and iteration count to 500 the final configuration.
Losses in DA. During the deparametric approximation, we utilize Lue for pseudd--llustering-bass updating. To fully assess the choice of loss functions, we compare classical losses including MSE, MAE, and CE. The results are shown in Tab. 2
As observed in Tab. 2 MSE achieves the highest average correlation (Tw. 0.608), outperforming MAE (Tw = 0.566) and CE (T... = 0.554). Although results vary slightly across individual datasets, the overall trend suggests that MSE provides the most robust estimation of transferabilitt. This may be attributed to the smoothing properties of MSE, which lead to more stable pseudo-cluster updates during the approximation process. Based this, we adopt MSE the default loss function for Epe-
Task generalization. Since ITM eliminates dependence on single-label supervision during estima- tion, it offers practical potential to generalize beyond image classification tasks. Unlike previous counterparts that are constrained by task-specific supervision and architectural coupling, ITM can be readily extended to dense prediction tasks such semantic segmentation, which require additional task heads and fine-grained feature modeling. To validate this capability, we construct a proxy benchmark for segmentation tasks, including five widely used foundation models evaluated on the Cam Vid 133 and Cityscapes [54] datasets. The results are presented in Tab.
As shown in Tab. 3 ITM delivers stable and accurate transferability predictions even for segmentation tasks, successfully selecting the most suitable models for downstream segmentation applications. This demonstrates the strong generalization ability and practical applicability of ITM. In contrast, existing methods such as PED, LogME, and SFDA designedd around single image-level classification labels strugll to generalize effectively to new tasks.

---

Table 3: Quantitative ablation study on semantic segmentation. We conduct experiments on two
dense prediction datasets, CamVic Vid [33] and Cityscapes [54] using five pre-trained ViT models.
Models
Datasets
Metrics
Tw
MoCov3-B16
DINO-BI6
MAE-B16
SimMIM-B16
MAE-LI6
mloU
58.11
60.05
63,99
64.52
68.25
Cam Vid
0.61
ITM
85.87
86.41
88.58
83.85
89.03
mioU
40.06
41.45
44,21
43.72
47.33
Cityscapes
0.72
ITM score
79.77
79.14
83.11
78.03
83.86
Conclusion
We present Implicit Transferability Modeling (ITM), a framework for transferability estimation that models embedding space evolution during fine-tuning as a posterior distribution, with latent integration capturing transferability. To enhance scalability, we introduce Divide-and Conquur Variational Approximation (DVA), enabling efficient embedding evolution without explicit fine-tuning. Experiments on an enhanced benchmark across diverse architectures and pre-training strategies show that ITM consistently outperforms state-of-the-art methods in effectiveness and generalization.
Limitations. While ITM demonstrates stronger generalization than prior methods, it remains con- strained by its reliance on embedding space discrimination and cannot yet directly handle complex supervision scenarios such as detection or vision-language tasks. common limitation of current TE methods is their dependence on final output features E o o(X), often overlooking the intrinsic prop- erties the model á. Although ITM partially addresses this via latent integration, more fine-grained modeling such leveraging intermediate representations or enriched output embeddings may be necessary to further improve transferability estimation. From another perspective, most existing TE methods focus primarily performance under full fine-tuning. However, the properties of base mod- els within the rapidly evolving PEFT [55 56 57 58 paradigm may not align with such assumptions. It remains an open challenge to extend transferability estimation frameworks to accurately predict performance under parameter-efffcient adaptation schemes.
Acknowledgment
This work is partly supported by the National Key Research and Development Plan (2044FB33003302), National Natural Science Foundation of China (82441024), the Beijing Natural Science Foundation (1.251073), the Research Program of State Key Laboratory of Complex and Critical Software Environment, and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter, and Josif Grabocka. Quick-tune: Quickly learning which pretrained model finetune and how, In International Conference on Learning Representations, 2024.
[2] Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In IEEE/CVF International Conference on Computer Vision, pages 1395- 1405, 2019.
[3] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure evaluate transferability of learned representations. International Conference on Machine Learning, pages 7294-7380 PMLR, 2020.
[4] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: pre-trained models for transfer learning. In International Conference on Machine Learning. pages
Practical assessment of
PMLR, 2021.
[5] Mohsen Gholami, Mohammad Akbari, Xinglu Wang, Behnam Kamranian, and Yong Zhang. Etran: Energy-based transferability estimation. In IEEE/CVF International Conference on Computer Vision, pages 18613-18622, 2023.
[6] Zixuan Hu, Xiaotong Li, Shixiang Tang, Jun Liu, Yichun Hu, and Ling- Yu Duan. Lead: Exploring logit space evolution for model selection. In /EEF/CVF Conference on Computer Vision and Pattern Recognition, pages 28664-28673, 2024.
[7] Xiaotong Li, Zixuan Hu, Yixiao Ge, Ying Shan, and Ling-Yu Duan. Exploring model trans- ferability through the lens of potential energy. In IEEE/CVF International Conference on Computer Vision, pages 5429-5438, 2023.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2020.
[9] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In /EEE/CVF International Conference Computer Vision, pages 8640-9649. 2021.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[12] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653 9663, 2022.
[13] Joris Guerin, Shray Bansal, Amirreza Shaban, Paulo Mann, and Harshvardhan Gazula. Vibes- vision backbone efficient selection. In British Machine Vision Conference, 2025.
[14] Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta- learning. In /EEF/CVF International Conference on Computer Vision, pages 6430-6439, 2019.
[15] Michal Pándy, Andrea Agostinelli, Jasper Uijlings, Vittorio Ferrari, and Thomas Mensink. Transferability estimation using bhattacharyya class separability. In IEEF/CVE Conference on Computer Vision and Patter Recognition, pages 9172-9182, 2022.
11

---

[16] Fanqing Meng, Wenqi Shao, Zhanglin Peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Foundation model is efficient multimodal multitask model selector. In 37th International Conference Neural Information Processing Systems, pages 33065-33099 2023.
[17] Wenqi Shao, Xun Zhao, Yixiao Ge, Zhaoyang Zhang, Lei Yang, Xiaogang Wang. Ying Shan, and Ping Luo. Not all models are equal: Predicting model transferability in a self-challenging fisher space. In European Conference on Computer Vision, pages 286-302. Springer, 2022.
[18] Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning. Advances in Neural Information Processing Systems, 34:19301- 19312, 2021.
[19] Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac- bayesian metrics for estimating the transferability of pretrained models to classification tasks. In European Conference on Computer Vision, pages 252-268. Springer, 2022.
[20] Kshitij Dwivedi and Gemma Roig. Representation similarity analysis for efficient task taxonomy transfer learning. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12387-12396. 2019.
[21] Andrea Agostinelli, Michal Pándy, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. How stable are transferability metrics evaluations? In European Conference on Computer Vision, pages 303-321. Springer, 2022.
[22] Prafful Kumar Khoba, Zijian Wang, Chetan Arora, and Mahsa Baktashmotlagh. Feature space perturbation: A panacea to enhanced transferability estimation. In /EEE/CVF Winter Conference on Applications of Computer Vision, pages 1299-1308. IEEE, 2025.
[23] Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre- trained models are from neural collapse on the target dataset informs their transferability. In IEEE/CVF International Conference on Computer Vision, pages 3549-5558 2023.
[24] Tengxue Zhang. Yang Shu, Xinyang Chen, Yifei Long, Chenjuan Guo, and Bin Yang. Assessing pre-trained models for transfer learning through distribution of spectral components. In AAAI Conference Artificial Intelligence, volume 39, pages 22560-22568, 2025.
[25] Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2663-2673, 2021.
[26] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[27] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4700-4708 4708, 2017.
[28) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In /EFF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[30] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 248-255. leee, 2009.
[31] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
12

---

[32] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong sumi-supervised learners. Advances in Neural Information Processing Systems, 33:22243- 2020.
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In IEEF/CVE Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
[34] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04290 2020.
[35] Yi Xin, Siqi Luo, Haodi Zhou, Junlong Du, Xiaohong Liu, Yue Fan, Qing Li, and Yuntao Du. Parameter-efficient fine-tuning for pre-trained vision models: A survey. CoRR, abs/2402.02242. 2024.
[36] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
[37] Léon Bottou. Lcrge-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 19th International Conference Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pages 177-186. Springer, 2010.
[38] Qi Meng, Wei Chen, Yue Wang, Zhi-Ming Ma, and Yan Liu. Convergence analysis of distributed stochastic gradient descent with shuffling. Neurocomputing, 337:46-57, 2019. 2019.
[39] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look memorization in deep networks. In International Conference Machine Learning, pages 233-242. PMLR, 2017.
[40] A Paszke. Pytorch: An imperative style, tighpperformance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[42] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint arXiv:1306.5751, 2013.
[43] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 178-178. IEEE, 2004.
[44] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[45] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3498-3448 3505. IEEE, 2012.
[46] Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013.
[47] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1495-3492. IEFE, 2010.
[48] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. 1ood-100-mining discriminative components with random forests. In European Conference on Computer Vision, pages 446-461. Springer, 2014.
13

---

[49] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphies & Image Processing. pages 722-729. IEEE, 2008.
[50] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.
[51] Mingxing Tan, Q Efficientmet Le, al. Rethinking model scaling for convolutional neural networks. In International conference on machine learning, Long Beach, CA, USA, volume 15, 2019.
[52] Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[53] Gabriel J Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. Segmentation and recognition using structure from motion point clouds. In European Conference on Computer Vision, pages 44-57. Springer, 2008
[54] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for seman- tic urban scene understanding. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3213-3223, 2016.
[55] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[56) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, volume 1, page 3, 2022.
[57] Nan Zhou, Jiaxin Chen, and Di Huang. ivpt: Improving task-relevant information sharing in visual prompt tuning by cross-layer dynamic connection. arXiv preprint arXiv:2404.05207, 2024.
[58] Nan Zhou, Huiqun Wang, Yaoyan Zheng, and Di Huang. Progressive parameter efficient transfer learning for semantic segmentation. In International Conference on Learning Representations, 2025.
[59] Charles Spearman. The proof and measurement of association between two things. The American journal of psychology, 100(3/4):441 1987.
[60] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
[61] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818-28888888 28--226,220 2016.
[62] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.
[63] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[64] Guolei Sun, Yun Liu, Henghui Ding, Thomas Probst, and Luc Van Gool. Coarse-to-fine feature mining for video semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3126-3137, 2022.
[65] Yaoyan Zheng, Hongyu Yang, and Di Huang. Deep common feature mining for efficient video semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
14

---

A Technical Appendices and Supplementary Material
A.1 Radar Chart
MAE-L16
MobileNetv2
ResNet-18
PARC
7Trm
(Ours)
Ground Truth
MAE 016
CINTTED CaiendES
OMATUD Crhandes
(IMA200
CIMAC:00
Callec D00
DINO-816
C113
ITTD
Ancon
Dm
DINO-S8
1M22 Pat
OPARLE
(b)
Dac
Dack
Nownster
MANH
helt::
SANI
SAN07
Hawcot
SFDA
PED
ITM (Ouis)
Ground
Figure 5: Comparison between ITM, recent TE methods, and ground-truth performance across benchmarks. Scores are normalized [0.3, 1] for clearer visualization.
(a) Evaluation on MAE and CNN pre-trained models. Static TE methods (e.g., PARC [18] and ETran (5) fail to generalize to MAE [ due to weak discriminative power initial embeddings. (b) Evaluation on DINO and MAE pre-trained models. Dynamic TE methods (e.g.. SFDA [17] and PED (7) overestimate MAE's performance. In contrast, ITM generalizes well across diverse pre-trained models and provides accurate estimations.
A.2 Pseudo-code
Algorithm Training process DVA without deparametric approximation
Require: Embedding of training dataset extracted by model Φ, target cluster center E, target label Y. linear layers f.g. h, number of training iterations T', learning rate a, number of subspace iterations M and learning rate of subspace 7]
Ensure: Optimized parameters W 2, W g, WA. of linear layers f.g. h.
1: Randomly initialize W., W
2: for 1 to T do
3: Sample one batch of data: (E,, E. Y.) + sample(E, (E É, Y,t)
4: Compute O, θ + [EE;; W.)
5: Randomly initialize W
6: for m to MM do
7:
Compute gradient: grad, <- Vw, MSE(s(O,), E.))
8:
Update parameters of g: Wg + W,, 7) grad, end for
9:
10:
Compute logits: logits +- h(g(O.))
11: Compute gradient: grad, +- Vw. CE(logits, Y.), gradn +- Vw, CE(logits, Y.)
12: Update parameters f. h: W, + W. a grady. W, + W a grad,
13:
end for
14: return W., W,, Wh.
As shown Algorithm T DVA requires multiple training steps for the parameter W, during each macro iteration, followed by updates to the parameters W, and W.. This "training within training" approach is not only challenging to implement but also inefficient. However, clearly demonstrates the core idea of DVA: training the batch-specific parameter W, exclusively for each data batch.
15

---

Algorithm 2 Training process of DVA
Require:
Embedding of training dataset E extracted by model d, target cluster center E, target label Y. linear layers [ ht number of training iterations T, learning rate a, number of subspace
iterations and learning rate of subspace
Ensure: Optimized parameters W Wv, of linear layers f,
1: Randomly initialize W., Wh.
2: for = to T do
3: Sample one batch of data: (E,, E,, Y,) sample E, E, Y,
4: Compute ; θ + W.E,
deparametric approximation: E, (2) f (1 7 O Eş) + Ej (Eq. 5
5:
6: Compute logits: logits t- ((E()))
7: Compute gradient: grad, s- V w. CE(logits, Y.), grad,, + Vw, CE(logits, Y.)
8: Update parameters f, h: W. + W. ---a a gradd, W. + W a egradd
9: end for
10:
return W W
As shown in Algorithm 2 we introduced the deparametric approximation method, which elimi- nates the need for training the linear layer g in lines 5 to of Algorithm T Instead, the feature output E."" " after several iterations is directly obtained using and 1). Additionally, we employ the Pseudo-clustering Optimization method generate pseudo-cluster centers that guide the convergence direction of E Finally, I and h are trained directly using CrossEntropy loss function.
A.3 Role of Latent Variable
Table 4: Cosine similarity matrix for the learned mapping parameters W., The background colors highlight the main patterns of consistency. Red cells show high intra-task similarity. Yellow cells show medium intra-model similarity.
D-A
M-A
C-A
D-F
M-F
C-F
D-R
M-R
C-R
D-A
1.00
0.197
0.181
0.133
0.133
0.133
0.138
0.134
0.133
M-A
0.197
1.00
0.215
0.133
0.142
0.134
0.131
0.158
0.134
C-A
0.181
0.215
1.00
0.131
0.132
0.135
0.132
0.132
0.142
D-F
0.133
0.133
0.131
1.00
0.340
0.318
0.137
0.133
0.134
M-F
0.133
0.142
0.132
0.340
1.00
0.349
0.132
0.152
0.136
C-F
0.133
0.134
0.135
0.318
0.349
1.00
0.132
0.133
0.139
D-R 0.138
0.131
0.132
0.137
0.132
0.132
1.00
0.301
0.283
M-R 0.134
0.158
0.132
0.133
0.152
0.133
0.301
1.00
0.309
C-R 0.133
0.134
0.142
0.134
0.136
0.139
0.283
0.309
1.00
ITM is designed to estimate transferability at the level of model-task interaction, rather than treating the model or task independently. The latent variable introduced to capture this interaction between a model's 's representational capacity and the semantic demands of downstream task. This task conditioned nature is essential, as the effectiveness on its own features but also on their alignment with the target task.
pretrained model depends not only
Since explicitly modeling is intractable due to the high-dimensional and non-linear nature of embedding evolution (and the absence of supervision for transferability). ITM adopts an implicit modeling strategy. For each model-task pair, a mapping function (- parameterized by W.. is learned. This function is shared across all subspaces (batches) for the pair and operates on conditioned embeddings that encode both the static properties of the pretrained model da and the distributional characteristics of the downstream task data. Combined with the update trajectory, this function simulates how representations adapt to new task, without requiring end-to-end fine-tuning.
16

---

To further validate and interpret the learned latent variable 2, we analyze the mapping parameters W across different model-tast pairs. We perform Singular Value Decomposition (SVD) on each W and compare the left singular vectors using cosine similarity. These vectors represent the dominant transformation directions the input space and provide a principled basis for comparing the learned functions. We study three pretrained Models-DINO-BE (C)-on three tasks: Aircraft (A), Flowers (F), and Cars (R). The resulting similarity matrix is shown in Tab. From this analysis, observe three salient patterns:
(D), MAE-B16 (M), and MoCov3-B16
Task Consistency:
Within each task (e.g., Aircraft, Flowers, Cars), the intra-task similarity among different models (i.e., the 3 diagonal blocks) remains consistently high. This suggests that ITM learns task-specific transformation functions that are stable across diverse backbones, highlighting its ability to capture task-dependent adaptation behavior.
Model Consistency: For each model (e.g., DINO-B16), the inter-task similarity-i.e similarity between the same model across different tasks is noticeably higher than that of unrelated model-task pairs. This implies that ITM effectively encodes model-specific inductive biases into the learned mappings.
Task Dominance: Overall, task-driven consistency is stronger than model-driven consis- tency, indicating the downstream task has greater influence than the model choice. This aligns with Table 4, where performance variance is more pronounced across tasks than across models.
A.4 Benchmark
Table 5: Accuracy of different models across datasets under optimized fine-tuning settings.
Models
Cal101
Cars
CIFARIOD
CIFARTO
DTD
Aireraft
Flowers
Food
Pets
SUN
VOC
DenseNet- 121
97.23
88.39
85.67
97.38
69.47
83.86
90.41
85.00
91.77
72.49
82.75
DenseNet-16 161
98.21
89.84
97.79
72.61
24.24
92.00
74.10
82.27
DenseNet- 169
97.29
88.83
86.24
97.75
70.90
83,80
91.58
85.58
92.91
73.76
81.85
DenseNet-201
97.47
89.55
86.39
97.65
72.98
84.04
91.07
86. 14
93.08
74.13
80.86
EfficientNet-Be
97.87
87.25
87.01
68.88
81.61
89.71
85.82
90.68
73.87
81.10
GoogleNet
96.54
86.54
83.39
96.97
69.73
83 29
88.01
81.36
90.60
69.67
80.45
InceptionV3
96.95
84.73
83.42
96.92
66.75
79.60
87.14
81.79
92.86
68.07
79.50
MnasNet
95.56
81.12
83.40
96.78
60.96
75.00
76.48
82.87
90.65
70.02
78.78
MobileNetv2
96.03
85.65
82.15
96.48
67.34
76.36
90.03
83.69
89.12
71.16
82.45
ResNet-18
95.74
83.58
82.69
96.57
65.59
77.56
88.52
79.92
88.50
68.71
76.74
ResNet-34
96.54
86.11
85.31
97.30
67.50
77.53
90.03
24
92,70
70.75
81.56
ResNet-50
97.58
88.87
84.60
97.70
70.85
84.49
91.38
87.08
93.65
74.85
84.19
ResNet-101
71.17
90.91
77.21
ResNet-152
97.64
89.07
88.58
98.01
71.44
81.64
89.25
88.35
94.79
76.18
84.73
DINO-B16
98.16
88.42
90.53
98.70
74.31
78.40
93.19
87.88
92.97
77.02
84.51
DINO-S8
97.47
89.33
90.24
98.65
71.97
80.02
90.84
90.84
93.10
77.53
85.58
MoCov3-B16
91.70
88.53
90.74
98.68
72.07
75.70
93.61
87.15
89.75
75.92
80.37
MAE-B16
97.52
88.16
87.55
98.42
69.04
72.16
85.80
87.30
89.81
75.29
82.86
MAE-LI6
97.98
91.21
91.28
98.55
74.26
85,30
90.73
90.82
79.18
88.01
SimMIM-B16
96.20
86.02
88.80
98.63
66.17
68.32
83.87
87.88
87.16
74.55
79.62
As shown in Tab. 5 and Tab, 6 we present the performance twenty difffrrnt pre-trained models on eleven downstream single-label classification datasets. Unlike previous works such as PED [ and SFDA we adopt a more practical training setup by using the superior Adam W optimizer and training for 100 epochs. This results in consistently better model performance in Tab. compared to prior methods, as shown in Fig. 6 The improvement is particularly evident on larger datasets such as CIFAR-[10 10, 100], SUN397, and Food101, where previous approaches like PED, which only train models for 5k iterations, which insufficient to fully capture the true performance of the models.
As shown in Tab. 7 we adopt consistent training split configurations across all datasets. Standard accuracy serves as the metric for ground-truth ranking computation. All datasets are sourced from the official sicchiision.datsseee module of PoTorch [40].
A.5 Other evaluation protocols
We evaluated the transferability estimation methods based other rank correlation metrics, including Spearman's p (59] and Kendall's 66)] As shown Tab. ITM also achieved the best performance on these two metrics.
17

---

Table 6: Comparison of benchmark attributes in terms of model diversity and fine-tuning protocols. "Mix" indicates the inclusion of models from multiple categories for evaluating TE methods.
Model
methods
Full settings
TE methods
CNN
WT
Mix
Supervised
Contrastive
learning
Masked image
modeling
Mix
Ratch size
Optimizer
Iterations
SFDA, SA. PED, LEAD
SGD
(Arg. 16.2 epochs)
5000 lierations
IIM (Ours)
64
Adam
100 spochs
Ours
SFDA, PED, al.
Rebk WI
per weweeee
Auptoret
19C-4
(N) CIRAR-1D
(30 CIBUR-10
(d Food- 101
(d) SUN-317
Figure 6: Performance comparison of model zoo entries across benchmarks. Under identical dataset settings across four benchmarks, the benchmark adopted in ITM better realizes the potential of each model and consistently delivers stronger performance than those used in prior studies, providing a more realistic evaluation of real-world model selection scenarios.
Table 7: Comparison of dataset settings across benchmarks. "Trainval" denotes joint use of training and validation sets. "mCA" indicates mean per-class accuracy, while "Acc" refers overall accuracy. ITM adopts standard splits and a unified evaluation metric for fair benchmarking.
settings
Methods
Cal101
Cars
CIFARI00
CIFARIO
DTD
Aireraft
Flowers
Food
Pets
SUN
VOC
Training dataset
SFDA, FED,
train
trainval
trainval
trainval
trala
train
train
tranvaal
ITM (Ouns)
train
train
train
trin
train
trann
train
train
train
Testing metric
SFDA, FED, et ol.
ITM (Ouns)
mCA
Ace
Acc
Ace
Acc
Ace
Acc
mCA
Acc
mCA
Acc
Age
Acc
DCC
Acc
Ace
Acc
mLA
Table 8: Comparison of Spearman's
59 and Kendall's T [60) for different methods on various
datasets.
Methods
Cal101
Cars
CIFARI00
CIFARIO
DTD
Aireraft
Flowers
Food
Pets
SUN
Avg.
Spearman's p
PED
0.61
0.05
0.81
0.92
0.19
-0.37
-0.07
0.84
20.18
0.25
0.31
LogME
0.81
0.61
0.62
0.61
0.77
0.45
0.90
0.20
0.37
0.52
0.59
WLeepp
0.37
0.24
0.25
0.42
0.53
0.44
0.79
0.09
0.41
0.01
0.35
PARC
-0.01
0,10
-0.19
0.05
0.49
0.43
0.79
0.03
0.26
-0.05
0.19
SFDA
0.71
0.16
0.68
0.70
0.09
0.38
-0.58
0.43
0.52
0.13
0.32
ETran
-0.01
-0.08
0.09
0.54
0.13
0.05
0.39
-0.13
ITM (Ours)
0.75
0.84
0.71
0.70
0.87
0.54
0.78
0.45
0.84
0.75
0.72
Kendall's
PED
0.556
0.067
0.644
0.778
0.111
40.333
-0.061
0.689
-0.156
0.111
0.241
LogME
0.600
0.422
0.422
0.467
0.600
0.333
0.778
0.111
0.244
0.378
0.436
N/Leep
0.289
0.156
0.244
0.289
01.222
0.289
0.600
0.156
0.333
0.067
0.284
PARC
0.022
0.111
-0.156
0.022
0.378
0,289
0.600
0.067
0.200
0.067
0.160
SFDA
0.511
0.156
0.467
0.600
0.067
0.244
-0.479
0.244
0.422
0.111
0.234
ETran
-0.022
.01122
.1.156
-0.022
0.111
(1.422
0.111
0.156
0.378
-0.066
0.089
ITM (Ours)
0.629
0.644
0.556
0.600
0.719
0.422
0.600
0.378
0.719
0.600
0.587
A.6
Benchmark of CNN models
Following the setup of SFDA [17], we also establish benchmark on supervised pre-trained CNN
models, including [enseNet-[121,]]] [12,,_16111111111111 1222,,11,,19121111 169,201] (77 ResNet-[344.0,.1 101, 150] (26). Inception V3 [6]] GoogleNet [62]. MobileNetV2 [50]. and MnasNet [63] As shown in Tab. our method continues to achieve the best performance on traditional benchmarks. Additionally, when ranking supervised pre-trained CNN models, most previous methods still perform better even as the number of models increases to 12, compared to their performance when handling a more diverse set of models (MIM, ID). This further highlights the limitations of existing TE methods in the effective handling of diverse pre-trained models.
18

---

Table 9: Comparison of weighted Kendall's 60 for different methods on CNN benchmark of SFDA I7
Methods
Cal101
Cars
CIFAR100
CIFARIO
DTD
Flowers
Food
Pets
SUN
VOC
Avg.
PED
0.401
0.230
0.559
0.625
-0.133
0.190
0.511
01755
40.101
0.446
0.350
LogME
0,358
0.402
0.750
0.761
0.340
0.141
0.780
0.664
0.817
0.758
0.597
eTeep
PARC
0.093
0.352
-0.456
-0.553
0.033
-0.134
-0,595
0,516
0.676
-0.119
-0.019
SFDA
0.501
0.485
0.791
0.761
-0.236
0.232
0.312
0.799
0.099
0.619
0.436
ETran
-0.301
-0.223
-0,414
0.043
-0.114
-0.159
-0.363
40.560
-0,458
-0.227
-0,278
SA (+LogME)
0.703
0.251
0.738
0.761
0.470
0.074
0.872
0.676
0,570
0.811
0.593
(Ours)
0.781
A.7 Adaptive
232
15c
Effomnteree
Hostokund
Ratherk
Decos
Helest
Hacsol
Devseliit 22
pfrissseeee
Holss
Hicicc
|Al|
(b) ||0|
Figure 7: Visualization of the divergence initial embedding distributions across different pre-trained models on Caltech-101. The mean and variance are computed to highlight differences the static embedding states.
The goal of DVA is to map the embedding spaces of the pre-trained backbone before and after full fine-tuning However, shown in Fig, Z different pre-trained models have initial embedding spaces with different means and variances, which means that using the same y) in equation 5 may be unfair models with initially poor feature distributions. Since equation is similar recurrence, we computed the average Fuclidean distance all initial features in initial embedding space E = (fi)² of model Φ, after standardization, to the cluster centers of their respective classes: dis, = レ= II[ f" ll where f the mean of the features for class y/
linear
4.F represents the standardized features, and I denotes
Since the linear recurrence coefficient n/C relation when batch size B = and 1) 1/o:
1 20,0] is complex, consider the recurrence
(n)) (1 - 170 θ (1 (1- m7) E E E/m=) (1 n7o) 1 O - (1 nu) )
(6)
(7)
This represents two initial spaces O and θ being updated towards the target E, and É, using different numbers of iterations n, and nj. By setting equation:
|E("] E =||B|-) E,l12 we obtain the
[I no) θ + (1 (1 No ) E E:llz [[1 Ton )))' 9 +
na) ) Egg12
= (1 mn) )m | E,l12 = (1 - 70) | E,
This implies that (1 - ) is inversely proportional to the Euclidean distance between the features and the target point. Therefore, we compute an adaptive learning rate 77 based on the model's 's dis
diss 1 + n4
n; fddss,, [og dis
In Eq. (9) we set ng = 20, 1Ju = 0.01 as an anchor for the model & when dis, = 1.0.
In practice, for batch sizes B > 1, the effectiveness of the aforementioned derivation is primarily restricted to the component of + parallel to the pseudo-cluster center E. Combined with the effective
19

---

1/B scaling of the learning rate nu, this causes the adaptive n mechanism to gradually degenerate towards fixed nú with increasing B. As result, the ITM approach can still use adaptive n with large batch sizes while preserving stability (as shown in Fig. 4 though its capacity to balance the convergence speeds across different models will be reduced. Furthermore, as the batch size increases, the convergence of ITM itself also accelerates. This diminishes the necessity for adaptive n to provide nuanced adjustments to update step lengths (or to fine-tune the update progression), making it possible to achieve comparable performance even with a fixed 72.
A.8 Pseudo-cluster centers
0.63
Mean
0.62
Range
Weightd Kendallss
0.58
0.57
Onc-hat
Random
PCA
Laplacian
Figure 8: Performance different methods for generating orthogonal pseudo-cluster centers. Mean and variance are computed over 10 independent runs to assess robustness and consistency.
As shown in Fig. we explored different methods for generating standard orthogonal pseudo-cluster centers on the unit sphere as targets for feature alignment. Specifically, "PCA" and "Laplacian" refer orthogonalization techniques applied after randomly generating centers using Principal Component Analysis (PCA) and the Laplacian matrix, respectively. Through repeated experiments, we observed that the performance across these methods exhibited minimal variation. This can be attributed the fact that the classification head h applies an additional linear transformation to the embedded features, making the choice of pseudo-cluster centers robust. Furthermore, can be mathematically proven that two sets of standard orthogonal bases P and Q can be transformed by an orthogonal matrix W, such that P WQ. In this scenario, the distance distribution of any feature in the space relative to these basis pseudo-clusto centers remains invariant before and after the transformation. Thus, even a simple linear classification head is robust to different standard orthogonal pseudo-cluster centers. To address the issue of random instability during practical use, we adopted one-hot encoding of the classes as the cluster centers.
A.9 Comparison with mutual information
While ITM does not explicitly compute mutual information (MI), its estimated scores can be viewed as empirical proxies for model-task alignment, which relates to-but goes beyond-MI. Conventional MI captures statistical overlap between datasets, but fails account for representational alignment or task-specific dynamics. ITM instead estimates how well model's features can adapt to a target task evolution capturing both semantic and structural compatibility. Thus, ITM offers a more fine- ine-grained and dynamic perspective than traditional MI-based metrics.
via latent, model-condiiionn
A.10
Task generalization
The successful application to semantic segmentation highlights ITM''s broader potential. In principle, ITM is task-agnostic its core DVA framework models the evolution of the embedding space, which is not inherently tied to a specific output type. The framework's 's use of a general downstream objective, Lobj. allows it to be adapted to any discriminative task where a task-specific head can be applied to the evolved embeddings. This makes ITMM valuable tool for a wide range of applications beyond classification and segmentation. For instance, complex video analysis tasks such as video semantuc segmentation 664 [5], critical first step is often the selection of a powerful pre-trained image model to act as frame-level feature extractor or backbone. ITM provides a practical and efficient solution for this crucial model selection process, enabling researchers to identify the most suitable backbone without extensive experimentation for use subsequent, more complex or specialized downstream tasks.
20

---

A.11
Embedding space mapping
Pretrained
(10ites)
IMM (Diters
(100
ITM (500iters)
Fineturd
MAE-BI6
0.49
0.71
0.79
0.86
O. 984
DINO-B16 ResNot-18
0.79
0.83
0.
0.92
0.93
0.
Figure 9: Illustration of the embedding evolution process. T-SNE visualizations of the embedding distributions for ResNer-18, DINO-B16, and MAE-BIG 16 on the CIFAR-10 test set, showing both initial and final states, along with intermediate states E (n) during ITM evolution. The number in the lower-left corner of each plot indicates the model's 's test accuracy at that stage.
As shown in Fig. 9 the embedding distributions of pre- pre-trained models E from different pre-training methods quite disparate, but after fine-tuning, their embedding space E converges much better distribution. Previous methods either solely rely on pre-trained model features or use crude methods simulate the fine-tuning process, which explains why these methods perform poorly such complex benchmarks.
A.12 Licenses
The CNN models and datasets used in this work leverage the PyTorch framework and the official torchvision datasets, both licensed under the 3-Clause BSD License. We gratefully acknowledge the contributions of the PyTorch development team. The licensing information for other models is shown
the Tab. 10
Table 10: Licenses of ViT models.
MoCov3
DINO
MAE
SimMIM
License
CC BY-NC 4.0
Apache-2.0
CC BY-NCCCA 4.0
MIT
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly articulate the core problem effllienffficiet and accurate transferability estimation (TE) amid the growing diversity of pre-trained models-and present Implicit Transferability Modeling (ITM), along with the Divide-and- Conquer Variational Approximation (DVA), the proposed solution. These claims are substantiated in the main body through the formal definition and formulation of ITM, implementation details of DVA, and extensive experiments a broad benchmark, with comparative results demonstrating improvements accuracy, efficiency, and stability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We explicitly discuss the limitations of our method in Sec. Please refer to that section for details.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
22

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides theoretical results in the form of the deparametric ap- proximation for embedding space evolution, detailed in Section 3.3 (Equation 5). It also
leverages established lemmas on mini-batch independence [36] 37 38 as theoretical support for the formulation in Equation 1. In the ablation study, we further evaluate variants of the pseudo-clustering loss Lpc- Since the corresponding deparameterized updates for these variants follow similar derivations as Equations (2)- we omit the full proofs for brevity.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper fully discloses all necessary information to reproduce its main experimental results. Section 4 details the benchmark setup, including dataset selection, pre-trained model zoo, and enhanced fine-tuning protocols. Hyperparameters- batch size, training iterations, learning rates, and optimizer settings (e.g., AdamW) explicitly reported in Section 4.2. Evaluation metrics like weighted Kendall's Tu are clearly defined, and implementation components including the DVA configuration and pseudo- clustering strategies are thoroughly discussed in Section 3 and further validated through ablation studies in Section 4. Together, these disclosures provide sufficient information for reproducibility. Additionally, we commit to open- sourcin all benchmarks, code, and pre-trained models upon acceptance facilitate reproducibility and future research.
as
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
23

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: All datasets used for training and evaluation are publicly available bench- marks. We provide the necessary codebase in the supplementary material, and all datasets, code, and pre-trained models will be open- open-soureed upon acceptance. The main paper thor- oughly documents the experimental setup and implementation details (Sections 3 and 4), including training protocols, model selection, and evaluation metries, ensuring the work reproducible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
publ ic /ug squides, CCoeeSuomi ss onPolicy cy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https)
/ //nips http:///pwlcccciicioooeeoccc http://guu.ddeeoooouuuummb.mma Coo/eeodeSSSSma as onP onPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all necessary training and testing details to support understanding and reproducibility, including data splits, hyperparameters, optimizer configurations, and fine-tuning protocols. Ablation studies on key hyperparameters such as batch size and training iterations thoroughly presented in Section 4.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: We provide appropriate information to assess the statistical reliability of its experimental results. Accuracy. Following prior work [2I], we assess TE stability across 1001 benchmarks (Fig. 2) and perform pairwise comparisons with state-of-the-art methods to highlight ITM's superiority. This provides strong empirical basis for analyzing variance across different model selections and offers insights into ITM's consistency and generalization.
Pseudo-cluster center generation. We explore alternative generation strategies in Ap- pendix A.8, confirming the robustness of our design. Evaluation metrics. Appendix A.4 reports additional rank correlation metrics to further validate the reliability of TE results.
Benchmarks. addition to our main benchmark spanning diverse pretraining paradigms (e.g., supervised, MAE, DINO, SimMIM, MoCov3) and architectures (CNNs, ViTs), we also evaluate ITM on a benchmark of 12 supervised CNNs (Appendix A.5).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
We provide sufficient information regarding the computational resources re- quired to reproduce the experiments, including the use of single NVIDIA V100 GPU with 32GB memory. All TE methods are executed under a consistent 8-core CPU environ- ment. Additionally, we report and compare the execution times of different TE methods. These details collectively ensure transparency and reproducibility in terms of computational requirements.
Justification:
25

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://eerrippscccppbbbi/////es
Answer: [Yes]
Justification:
We conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We briefly discuss the societal impact of TE and our proposed ITM framework in Sections and 2 the main paper. The primary motivation behind TE- and by extension, ITM- reduce the substantial computational cost and associated environmental footprint incurred by brute-force fine-tuning of large vision foundation models for model selection. By enabling efficient and accurate transferability estimation, ITM contributes more sustainable machine learning practices and makes high-performance AI more accessible for deployment in resource-constraing settings.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
26

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no high-risk misuse concerns requiring safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See A.9 in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperss thoode / datt sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
27

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing experiments or new research conducted directly with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing experiments research conducted directly with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM Victppss//eeriissccc////eeeeeeee
LLM) for what should should not be described.
28


Reviews:
[{"review_id": "hZW81mAtkk", "reviewer": "Reviewer_U85n", "summary": "The authors works on the transferability of vision foundation models, specifically:\n1. Proposed a novel method to evaluate transferability through modeling pretraining embedding $\\mathbf{E}$ and latent variable $\\mathbf{z}$.\n2. Reduce compute complexity of proposed method through variational approximation, batch-wise reduction and deparametric approximation.\n3. Conducted comprehensive experiments, using including models of difference sizes and beat their other baselines on VTAB.", "questions": "Based on what I specified in \"weaknesses\"section, I would like the authors to convince me that this work is not overly complicated and to clarify its value in real-world scenarios. For example, with all these approximation approaches, what is their actual upper bound? How does each level of approximation affect the final performance. Then I can decide better on whether or not it is really applicable in real world.", "limitations": "yes. They left comments in their \"conclusion\" part", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is novel in the context of transferability estimation, and its presentation is thorough and detailed.\n2. The experiments conducted are comprehensive and well-detailed on their testbed.\n\nWeaknesses:\nMy main concern: seems the method are overly complicated. Which may prevent its real applicability in real world scenarios.", "was_revised": true, "final_justification": "Thank the authors for their efforts in the rebuttal. I'm convinced this approach is applicable in real appliactions. I will raise my score to 4.", "rebuttal": "We thank the reviewer for  your recognition of our method's novelty and the comprehensive experiments and raising this important concern regarding the complexity and real-world applicability of our proposed method. Below, we address each of your points in detail:\n\n**[W1-Complexity and Applicability ]** \nITM introduces a new paradigm for transferability estimation by modeling fine-grained model–task interactions through implicit representation dynamics and a divide-and-conquer adaptation strategy. While the overall framework is more expressive than prior work, each of its components has been deliberately designed to ensure computational efficiency and scalability. In contrast to approaches that rely on heuristic simulation or end-to-end fine-tuning, ITM leverages both static and dynamic cues from pretrained embeddings, avoiding expensive optimization through a lightweight latent modeling mechanism guided by pseudo-cluster-based evolution.\n\nThe proposed pipeline is grounded in a principled and modular approximation strategy. Specifically, it decomposes the embedding space into semantically coherent, batch-wise subspaces to enable localized conditional modeling. The Pseudo-Clustering Generation (PCG) module introduces flexible, discriminative anchors, while the Deparametric Approximation (DA) efficiently models subspace evolution over time. Together, these components enable dynamic transferability estimation without incurring prohibitive computational cost.\n\nEmpirical results corroborate the efficiency of our framework. As reported in Table 1, ITM adds only a marginal computational overhead—averaging 8.42 seconds per evaluation—which is negligible compared to the total inference time (738 seconds), and remains on par with state-of-the-art methods. More importantly, ITM significantly enhances estimation accuracy, improving rank correlation by up to 35% relative to strong baselines (e.g., from 0.45 for LogME to 0.61 for ITM). Furthermore, ITM demonstrates broad generalization across a variety of pretrained models and tasks. In addition to classical classification benchmarks, we evaluate ITM on emerging foundation models (e.g., CNNs, ViTs, contrastive and masked pretraining paradigms) and, for the first time in the transferability estimation literature, dense prediction tasks such as semantic segmentation (Section 4). These results highlight the practical relevance and robustness of ITM across diverse scenarios.\n\nWe will revise the final manuscript to further clarify this trade-off between expressiveness and efficiency, and to emphasize how the modular structure of ITM enables scalable, accurate, and general-purpose transferability estimation with minimal overhead.\n\n**[Q1-Value in Real-World Scenarios]** \nUnlike prior transferability estimation (TE) approaches that rely on static heuristics or simplistic separability proxies, ITM introduces a lightweight yet expressive framework that models transferability via implicit representation evolution. Rather than simulating the full embedding trajectory, ITM adopts a divide-and-conquer strategy that models evolution locally in embedding subspaces, thereby enabling efficiency and robustness.\n\nITM is, to our knowledge, the first TE method to:\n\n- Generalize across a wide range of backbone architectures (e.g., CNNs, Vision Transformers) and training paradigms (e.g., supervised learning, contrastive learning, masked autoencoding);\n- Extend beyond image classification to dense prediction tasks such as semantic segmentation, while maintaining strong correlation with downstream performance despite the introduction of task-specific heads.\n\nThese capabilities make ITM well suited for real-world applications, including AutoML, model selection under resource constraints, and federated or distributed learning settings. Its modular formulation supports composability and scalability. Our extensive experiments (see Table 1, Figure 3, Tables 3, 4, and 7) consistently demonstrate strong cross-domain performance. To facilitate adoption and ensure reproducibility, we will release the full codebase, benchmark pipelines, and pretrained models.\n\n**[Q2 – Approximation Strategy and Upper Bound Justification]**\n- *Theoretical Rationale*: The ITM approximation pipeline—comprising batch-wise decomposition, pseudo-cluster generation (PCG), and deparametric approximation (DA)—is grounded in a principled decomposition of the transferability estimation process. Modeling global embedding dynamics across heterogeneous model-task pairs is inherently ill-posed and computationally expensive. ITM sidesteps this by focusing on localized embedding evolution within subspaces, a strategy supported by theoretical insights that local semantic structures tend to be more stable and transferable.\n\n- *Empirical Validation*: The ITM score, derived from the DVA, provides a tractable and quantifiable estimate of transferability. Across 11 classification benchmarks involving over 20 pretrained models with diverse architectures and pre-training strategies, ITM consistently outperforms strong baselines in predictive performance (Table 1, Figure 3, Tables 3, 4, 7, and 8). We further demonstrate robustness to random seeds and clustering initializations (Section 4.4), with minimal sensitivity in estimation results.\n\n- *Alignment with Fine-Tuned Embeddings*: While establishing a formal theoretical upper bound remains challenging due to the inherent noise and non-convexity of fine-tuning dynamics, we evaluate the semantic alignment between the approximated embeddings and those obtained from fully fine-tuned models. As shown in the t-SNE visualizations (Figure 9), ITM’s predicted embedding trajectories closely resemble the structure of post-finetuning distributions, indicating that our approximation is both semantically coherent and practically effective.\n\n  In summary, the ITM approximation strategy is not only theoretically grounded and empirically robust, but also essential for achieving scalable, generalizable, and high-fidelity transferability estimation across diverse model–task scenarios. We will further refine the manuscript to better articulate the rationale and effectiveness of this approach.\n\nWe greatly appreciate your thoughtful feedback and are happy to provide additional clarification as needed."}, {"review_id": "Hwgg9wZAVf", "reviewer": "Reviewer_ZajB", "summary": "This paper tackles the problem of estimating which pre-trained image foundation model will be best-suited for a given downstream task by proposing a method that combines an estimate of out-of-the-box model performance with an approximation of how the model's embedding space might evolve under additional fine-tuning. This is an improvement from current attempts to estimate transferability, which tend to only do static evaluations (i.e. statistical estimates without fine-tuning) or dynamic evaluations (i.e. simulating the fine-tuning process), but not both at once. This method achieves strong performance in estimating transfer on a wide variety of image foundation models on unseen image classification datasets as well as segmentation tasks.", "questions": "- How should I understand differences between the weighted Kendall's tau_w intuitively? For instance, in Table 1, how significant is the difference between methods yielding a tau_w of 0.61 vs. 0.45, especially considering that LogME is more than 4x faster? \n- What if your primary evaluation criterion is ability to generalize to a wide variety of tasks? Would you recommend repeating this study on a variety of downstream tasks, or is there a better strategy?\n- To what extent does this method generalize more broadly to other downstream tasks such as regression or even multi-modal transfer? \n- Is this method potentially useful in terms of estimating the mutual information between the training dataset and the downstream task dataset?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Novel methodology that combines two distinct types of existing approaches in the literature in a computationally-efficient manner\n- Reports SOTA accuracy averaged over a large number of open-source datasets and fine-tuned FMs \n- In an effort to avoid evaluation bias, the authors use official implementation settings to fine-tune existing FMs\n- They are planning to make their code, benchmarks, and pre-trained models publicly available\n- Detailed description of theoretical grounding and implementation in practice\n\nWeaknesses:\n- The paper is concerned almost entirely with the task of single-label image classification, with a small results table for transfer into segmentation tasks at the end, which potentially limits its impact\n- Method is not SOTA in wall-clock time (though it is among the fastest of the methods evaluated)\n- Though the method is evaluated across a number of different benchmarks, none of the results reported in the tables have error bars, making it difficult to fully compare the methods.", "was_revised": true, "final_justification": "I appreciate the authors' thorough responses, especially around the interpretation of their figure of merit, cross-domain generalization results, and specifying ITM's inherent variance. I agree that showing results on e.g. regression tasks or other modalities might require significant development beyond what is presented here. Without the addition of significant new results on new tasks or modalities, I think it's fair to keep my score as-is (\"accept\"). I believe the work is notable and would be of interest to many attendees at NeurIPS.", "rebuttal": "Thank you for your thoughtful and constructive review. We sincerely appreciate your recognition of the novelty of our methodology, our state-of-the-art performance across diverse benchmarks, and the clarity of our theoretical and practical contributions. Below, we respond to each of your questions and concerns in detail.\n\n**[W1-Evaluation Scope]** \nPrior TE methods have predominantly focused on single-label image classification tasks. On one hand, their frameworks—whether based on static heuristics or dynamic feature simulations—are closely tied to classification-specific assumptions and thus struggle to generalize to dense prediction tasks such as segmentation. On the other hand, classification accuracy is commonly adopted as a proxy to evaluate model capacity and alignment with downstream tasks.\n\nTo ensure a fair comparison with prior work, we adopt this setting and evaluate ITM across a broad suite of standard classification datasets, following established TE protocols (see Table 1, 4, 8, and Fig. 3).\n\nCrucially, ITM goes beyond existing TE approaches by supporting dense prediction tasks—demonstrating, to our knowledge, the first TE framework applicable to semantic segmentation. Leveraging its DVA formulation and modular design, ITM supports task-specific heads and models adaptation dynamics even in spatially structured output spaces. As shown in Table 3, ITM achieves accurate and consistent performance on CamVid and Cityscapes.\n\nTo evaluate cross-domain generalization, we assess ITM on the OIA-ZIB medical segmentation dataset (~80K MRI slices, 6 classes). ITM achieves a weighted Kendall’s tau of 0.57, showing strong correlation with downstream mIoU despite domain and task shift:\n\n|           | MoCov3-B16 | DINO-B16 | MAE-B16 | SimMIM-B16 | MAE-L16 | Weighted Kendall |\n| --------- | ---------- | -------- | ------- | ---------- | ------- | ---------------- |\n| mIoU      | 0.7306     | 0.7319   | 0.7338  | 0.7349     | 0.7359  | -                |\n| ITM Score | 0.9346     | 0.9394   | 0.9540  | 0.9138     | 0.9600  | 0.57             |\n\nThese results underscore ITM’s robustness on dense, high-variability tasks without reliance on handcrafted heuristics.\n\nDue to time constraints, further segmentation benchmarks could not be included during the rebuttal. However, we are actively expanding our segmentation evaluation suite and will release all code, evaluation toolkits, pretrained weights, and baselines to support reproducibility and broader adoption.\n\n**[W2 – Wall-clock Runtime Not SOTA]** \nIn line with common TE practice (e.g., LogME, PED, SFDA, ETran), we report only the time required for transferability estimation (TE), excluding embedding extraction time, which is shared across all methods and orthogonal to the estimator’s design.\n\nWhen the shared embedding inference time (~738 seconds) is included—see Line 266 for details—the overall wall-clock difference becomes marginal. Specifically, the additional cost of ITM relative to LogME is (8.4 s – 1.9 s) / (738 s + 1.9 s) ≈ 0.9%, which is negligible in practical scenarios where embedding computation dominates total runtime.\n\nCrucially, this modest overhead yields substantial benefits. ITM improves weighted Kendall’s tau from 0.45 to 0.61 (a 35% gain), while also generalizing across diverse pretraining schemes, architectures, and task types (including segmentation), which prior methods fail to support.\n\nWe will clarify this runtime trade-off in the final version to avoid ambiguity.\n\n**[W3-Lack of error bars]** \nFollowing your suggestion, we reran our main experiments across five additional random seeds. ITM achieves a mean weighted Kendall’s tau of 0.602 ± 0.006, demonstrating stable, low-variance performance.\n\nThis consistency further supports ITM’s reliability, complementing the improvements over baselines shown in Fig. 3. We will include standard deviation and error bars in the final version.\n\n**[Q1-Interpretation of Weighted Kendall’s Tau]**  \nWeighted Kendall’s Tau provides a rank correlation measure that emphasizes the relative importance of higher-ranked items, making it particularly relevant for scenarios where selecting top-performing models is critical (e.g., AutoML or low-resource deployment). It prioritizes getting the top of the ranking correct, which aligns closely with the practical use case of model selection.\n\nIntuitively, a weighted τ = 0.61 implies an 80.5% probability of correctly ranking any randomly weighted sampled model pair (vs. 72.5% for LogME with τ = 0.45). This 8% absolute improvement in ranking accuracy is substantial and has concrete impact: selecting the wrong model from the top candidates can lead to significantly degraded downstream performance. \n\nIn our experiments, this improvement in τ consistently translated into selecting models with materially higher accuracy on downstream tasks, especially under resource constraints where only the top-ranked models are deployed. The other metrics (as shown in Tab. 7) in the appendix also demonstrate the effectiveness of ITM.\n\nWe will add this explanation in the final version to clarify the practical value of higher τ scores.\n\n**[Q2-Primary Evaluation on a Wide Variety of Tasks]** \nWe fully agree that evaluating generalization across a broader spectrum of downstream tasks is both practically important and a key direction for transferability estimation (TE) methods.\n\nWhile single-label image classification remains the de facto evaluation standard in TE, it offers a consistent and interpretable metric (e.g., Kendall’s Tau) and enables direct, fair comparisons with prior work such as LogME, SFDA, and PED. Aligning with this standard allows us to benchmark ITM under established assumptions.\n\nHowever, extending evaluation to diverse downstream tasks requires establishing ground-truth rankings via full fine-tuning of all models—an effort that scales exponentially with the number of tasks and architectures and is beyond the feasible scope of this work and the rebuttal period.\n\nTo balance rigor and practicality, we follow standard classification protocols while taking a significant step forward: ITM is the first TE method to generalize to dense prediction tasks, such as semantic segmentation. As shown in Table 3 and the OIA-ZIB benchmark, ITM maintains strong predictive performance even under domain shift, demonstrating its ability to model transferability beyond traditional classification.\n\nImportantly, ITM is built upon a modular and task-agnostic formulation of adaptation dynamics. This makes it inherently extensible to a broader range of tasks, including multi-label classification, object detection, and vision-language scenarios. As part of future work, we plan to explore a more generalized formulation in which a single latent variable $z_{\\text{general}}$ is learned for each pretrained model and optimized to minimize the average prediction error across diverse tasks. Such a formulation could offer a unified representation of a model’s intrinsic transferability and enable more effective model selection in large-scale, heterogeneous settings.\n\nIn summary, while repeating this study across a broader range of tasks is a natural and promising next step, we believe ITM already provides a practical and extensible foundation for doing so. We will clarify this roadmap, and the associated trade-offs, in the final version to better reflect both current contributions and future directions.\n\n**[Q3-Generalization to Regression or Multi-modal Transfer]**\nThank you for raising this important point. ITM is task-agnostic in principle—its core latent modeling and approximation strategies are not inherently tied to any specific output type (e.g., classification labels), making it theoretically applicable to a broad range of tasks, including regression and multi-modal transfer.\n\nHowever, in its current implementation, ITM relies on the Pseudo-Clustering Generation (PCG) module, which utilizes semantic label information to approximate embedding evolution. As a result, it is naturally suited to semantically structured tasks such as classification, segmentation, and multimodal alignment—where discrete or textual supervision is available. Extending PCG to regression tasks is non-trivial, since these tasks involve continuous outputs rather than categorical labels, making clustering and transition modeling more challenging.\n\nTo adapt ITM for regression tasks (e.g., object detection or keypoint estimation), the framework would need to support continuous target spaces and account for task-specific prediction heads. This would require designing a regression-compatible variant of PCG and modifying the Deparametric Approximation (DA) module accordingly. We recognize this as a meaningful extension, and as discussed in our limitations section, we plan to explore this direction in future work.\n\nWe believe this extension is feasible given the modular nature of ITM and look forward to advancing its broader applicability in future iterations.\n\n**[Q4-Relation to Mutual Information between Datasets]** Thank you for this thoughtful point. While ITM does not explicitly compute mutual information (MI), its estimated scores can be viewed as empirical proxies for model–task alignment, which relates to—but goes beyond—MI.\n\nConventional MI captures statistical overlap between datasets, but fails to account for representational alignment or task-specific dynamics. ITM instead estimates how well a model’s features can adapt to a target task via latent, model-conditioned evolution—capturing both semantic and structural compatibility. Thus, ITM offers a more fine-grained and dynamic perspective than traditional MI-based metrics.\n\nWe appreciate your insightful framing and will clarify this relation in the final version.\n\nWe greatly appreciate your thoughtful feedback and are happy to provide additional clarification as needed."}, {"review_id": "FHDudxQqYn", "reviewer": "Reviewer_Ly76", "summary": "This paper presents an approach to build a transferability score between a pretrained model and a target dataset.\n\nThis is done by extracting latent features by running the model on the dataset, and then learning a mapping between the latent space and the final fine-tuning performance. The innovation consist of splitting the dataset into batches and to process sub-region of the latent space corresponding to these batches independently.\n\nI should emphasize that I thought the paper very hard to follow and I am not sure of my understanding, despite being used to the literature about transfer learning.", "questions": "See questions in the weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- The results obtained by the proposed method appear to be good on the tested datasets and pretrained models. They often outperform the approaches used for comparison.\n\n\n**Weaknesses**:\n- I thought the paper was really hard to follow and I struggled to understand what is being done, and what motivates the choices made by the authors. Many concepts, ideas and notations are introduced without clear explanations and justifications. For example:\n    - What is the rationale behind splitting the space into batches? It is central to the proposed approach, and I could not find a clear justification for this methodological choice.\n    - What is the scope of the learnable function to map embeddings to transferability? Is there a different function for each pretrained model? For each subspace of each pretrained model? Is it the same function (shared weights) for all pretrained models?\n\n- Experiments are not sufficient:\n    - Datasets: only classification datasets, why not experimenting with more complex datasets (segmentation, detection)? The claims talks about downstream tasks in the broad sense, but the experiments only present classification. This limited scope should be clearly stated in the title, abstract, intro. Later on, we discover that the approach was tested on segmentation, why was it not mentioned in the section about datasets description? Why not comparing the different approaches on the semantic segmentation datasets? \n    - Models: 10 pretrained models is really not a lot. Model zoos have way over 1000 models and the real challenge today is how we select a good model among this large pool of models. \n    - Wall-clock time appears particularly fast. I find it hard to believe that it takes less than 10 seconds to assess the transferability of 10 models on CIFAR100, knowing that we need at least one forward pass of each model on each of the 50k images. More details are needed to understand what exactly is presented here.\n    - Comparsion to the literature: Missing references to the compared approaches and details about why they were selected.\n    - “In contrast, existing methods such as PED, LogME, and SFDA—designed around single image-level classification labels—struggle to generalize effectively to new tasks.” —> I don’t see what supports this claim in Table 3.\n\n- In the introduction, I found the limitations of the existing literature very abstract. The result is that I couldn’t grasp the gaps that this paper attempts to fill. For example, “existing TE techniques are primarily evaluated on models with similar architectures and training paradigms […] and fail to generalize to models trained with advanced pre-training strategies or novel architectural designs” —> I don’t see why the approaches presented just above are architecture-dependent or training paradigm-dependent. More explanations are required. Another example: “previous studies attempt to model these dynamics by directly simulating the evolution of embedding spaces, they fail to comprehensively capture both aspects, thereby limiting their generalization capabilities.” —> This is very abstract, what aspects? Why did previous attempts fail? How did they fail? When I reached line 40, it was still not clear what limitations the paper is trying to overcome.\n\n\n\nNote: My concerns about this paper are mostly due to the fact that I believe the paper needs heavy rewriting to improve clarity. I might be wrong in my assessment and if other reviewers feel that the paper is understandable, feel free to favor their review over mine.", "was_revised": true, "final_justification": "After rebuttal", "rebuttal": "Thank you for your thoughtful and detailed feedback. We sincerely appreciate your effort in reviewing the paper. Below, we provide point-by-point responses and clarifications.\n\n**[Summary: Clarity and Readability]** \nWe appreciate your comments and regret that parts of the manuscript may have been unclear. While other reviewers found the paper well-organized, we acknowledge that the introduction may lack sufficient conceptual grounding for readers unfamiliar with *transferability estimation* (TE), an emerging but distinct field from traditional transfer learning.\n\nTE focuses on predicting how well a pretrained model will perform on a new task *without fine-tuning*, avoiding costly trial-and-error across large model pools. Unlike transfer learning, which adapts model weights, TE produces a *score* for guiding efficient model selection.\n\nWhereas existing TE approaches rely on *static heuristics* or *rule-based dynamics*, our method introduces a principled framework that models *latent representation evolution*. This enables more accurate, generalizable TE across diverse architectures, pretraining paradigms, and—for the first time—complex tasks like semantic segmentation.\n\nWe will revise the introduction and related work sections to clarify these goals and contributions, helping make the paper more accessible without compromising rigor.\n\n**[W1-1 Motivation for Batch-Wise Space Division]** \nPrior TE methods typically model the global embedding space, which is high-dimensional, entangled, and often dependent on specific architectures—limiting generalization.\n\nIn contrast, ITM adopts a divide-and-conquer approach by modeling local representation dynamics in batch-wise partitions of the embedding space. This allows ITM to capture consistent and fine-grained adaptation patterns while maintaining scalability.\n\nWe will revise the manuscript to highlight this motivation more clearly and explain its theoretical and practical benefits.\n\n**[W1-2 Scope of the Learnable Mapping Function]** \nTo assist your understanding, here is a consensus in the existing field of transferability estimation: Given a model zoo $\\{\\phi_i\\}$ and a series of downstream datasets $\\{D_i\\}$, all TE methods require that for each $<\\phi_p, D_q>$ pair, the features of all samples are extracted to form an initial Embedding $E_{pq}$. A prediction score is then given by a TE function, $TE(E_{pq}, Y_q)$, where $Y_q$ represents the labels for task $D_q$. In other words, for $M$ models and $N$ tasks, the TE method must run $M \\cdot N$ inferences for feature extraction and $M \\cdot N$ TE function calls for calculating the TE score. Therefore, for our ITM, all the method descriptions are based on the $E$ and $Y$ obtained from a single pair $<\\phi, D>$.\n\nIn the transferability estimation (TE) literature, the core objective of the learnable mapping function is to predict how well a given pretrained model will adapt to a target task—typically by approximating how its feature representations would evolve post fine-tuning. Depending on the method design, this mapping may be shared across models or conditioned on each model-task pair.\n\nFor example, static TE methods often adopt a shared function to directly map from initial embeddings to separability scores, while dynamic methods may attempt to simulate adaptation trajectories in a model- or task-specific manner (see Sec. 2 for a taxonomy).\n\nITM follows this general paradigm but introduces a more discriminative and adaptive formulation. Specifically, ITM learns a unique mapping function for each model–task pair, parameterized by a latent variable $\\mathbf{W}_z$, which governs a lightweight evolution function operating at the batch level. This function takes a local batch of embeddings as input and outputs their evolved representations, effectively approximating the dynamics of fine-tuning without requiring access to gradient-based training.\n\nThis localized and modular design enables ITM to efficiently capture model–task-specific adaptation signals while remaining computationally lightweight and broadly applicable across architectures and tasks. Importantly, it avoids brute-force simulation or end-to-end optimization, distinguishing ITM from prior approaches.\n\nWe will revise Section 3 in the final version to clarify the scope and benefits of this mapping function design.\n\n**[W2-1 Dataset Scope and Presentation]** \nWe fully agree that broader evaluation across diverse and complex downstream tasks is crucial for advancing transferability estimation (TE). In fact, one of the key contributions of our work is the explicit extension of TE beyond standard classification to more complex settings such as semantic segmentation.\n\nWhile prior TE methods—including NLEEP, LogME, PARC, SFDA, ETran, PED, and LEAD—have focused almost entirely on single-label image classification and CNN-based backbones, our approach is the first to support dense prediction tasks. Not only did we construct a more authoritative benchmark with 20 models and 11 image classification datasets (as shown in Figure 6), but we also tested ITM's ability to generalize to the semantic segmentation task on pre-trained ViT models. To our knowledge, this is the most diverse benchmark to date in the TE literature, covering both CNN and ViT backbones, and a range of pretraining strategies such as supervised learning, contrastive learning (e.g., MoCo, DINO), and masked image modeling (e.g., MAE, SimMIM).\n\nSpecifically, we include two standard semantic segmentation datasets—CamVid and Cityscapes—as well as an out-of-distribution medical segmentation benchmark (as shown in the reply to Reviewer ZajB). Our results show that ITM maintains strong correlation with downstream performance on these dense tasks, including under domain shift, demonstrating the method’s robustness and generalizability. We will add more descriptions about the Dataset as suggested.\n\n**[W2-2 Model Coverage]**\nWe agree that selecting from large model zoos (1,000+ models) is a key challenge. However, evaluating at this scale would require 10,000+ fine-tuning runs and 100,000+ TE evaluations, amounting to millions of GPU hours—beyond our current computational resources.\n\nInstead, we curated 20 representative models covering diverse architectures, training paradigms, and scales. To simulate large-scale selection, we conducted *random subset evaluation* (Fig. 3), sampling 6 models from 10 to form 1,001 subsets. ITM consistently outperformed baselines with low variance, validating its robustness in large-model-pool scenarios.\n\nWe will emphasize this more prominently in the final version and release the full codebase to support future large-scale validation.\n\n**[W2-3 Wall-Clock Time Clarification]**\nFollowing standard TE practices (e.g., PED, ETran, LEAD), we report only TE runtime, excluding the shared embedding inference (~738s), as stated in Line 266. ITM’s added runtime over LogME is just 0.9% = (8.4 s – 1.9 s) / (738 s + 1.9 s) when considering total time—negligible in practice.\n\nWe will clarify this runtime breakdown more explicitly in the final version.\n\n**[W2-4 Comparisons to the literature]** \nWe compare against open-sourced, officially implemented TE baselines (e.g., LogME, PED, SFDA), which ensures fair evaluation on modern model zoos. We will make these references clearer in tables and describe baseline selection criteria in more detail.\n\n**[W2-5 Claims and Clarification on Generalization]** \nExisting TE methods are designed for classification and assume fixed-head architectures. They cannot generalize to tasks like segmentation, which require additional heads and structure-aware adaptation. These methods do not model how representations evolve under such conditions, making them unsuitable for dense prediction tasks.\n\nIn contrast, ITM explicitly captures these dynamics via DVA, enabling transferability estimation for architectures involving flexible task-specific heads. This is why only ITM is evaluated on segmentation (Table 3). We will clarify this in Section 2 and the Table 3 caption.\n\n**[W3 Literature Gaps and Limitations Clarification]** \nCurrent transferability estimation (TE) methods generally fall into two categories: (1) static statistical methods, which rely on frozen embedding distributions (e.g., LogME, PARC), and (2) hand-crafted dynamic approximations, which simulate fine-tuning effects using heuristic metrics (e.g., SFDA, PED). While these approaches have shown promise on conventional CNNs and supervised training paradigms, they struggle to adapt to modern visual foundation models—such as ViTs or self-supervised models—which exhibit more diverse and complex adaptation behaviors, as demonstrated in Figures 5 and 6 of our paper. These observations suggest that static modeling or rigid dynamics are insufficient to accurately capture transferability across increasingly heterogeneous model families and task requirements. In response, our method—ITM—introduces a fundamentally different framework. Instead of relying on frozen statistics or heuristics, ITM implicitly models the latent evolution of embedding spaces via a shared transferability function for each model-task pair, and is trained using our DVA strategy. This allows ITM to adaptively account for both the intrinsic properties of pretrained models and the demands of downstream tasks, supporting greater generalization across domains, architectures, and even dense prediction tasks. We will revise the introduction and methodology sections to more clearly articulate these limitations in prior works and emphasize the unique motivation and advantages of our approach.\n\nWe hope these responses address your concerns. Please don’t hesitate to reach out if further clarification is needed. We are committed to improving the clarity and impact of the final version."}, {"review_id": "0kNSFJFl0s", "reviewer": "Reviewer_zA6C", "summary": "This paper proposes Implicit Transferability Modeling (ITM), a framework for estimating how well a pretrained vision model will perform on a new downstream task, without full fine-tuning. Unlike prior methods that explicitly simulate embedding space evolution during adaptation, ITM introduces a latent variable representing model transferability and a Divide-and-Conquer Variational Approximation (DVA) to efficiently simulate embedding adaptation in subspaces. Experiments are conducted on on a range of downstream tasks and models, showing performance improvement in both accuracy and efficiency.", "questions": "Please refer to the weaknesses section above", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes a novel idea: learn a latent representation of transferability instead of hardcoding how models evolve. This makes the method more flexible and scalable.\n- The method clearly improves over past work on standard datasets. It runs faster than many dynamic baselines while also being more accurate.\n\n**Weaknesses**\n\n- The DVA, may be an oversimplification of the actual fine-tuning process. The deparametric approximation is derived from a simple MSE loss objective, resulting in a linear iterative update toward fixed pseudo-cluster centers. Real fine-tuning is a complex, non-linear process, typically driven by cross-entropy loss and sophisticated optimizers like AdamW. It is questionable whether the proposed linear evolution simulation can realistically capture these dynamics. The framework may not be modeling fine-tuning so much as it is measuring how well a model's initial embeddings can be linearly mapped to separated clusters—a useful proxy, perhaps, but a proxy nonetheless.\n\n- The central claim of \"implicitly modeling transferability\" using `z` is ambiguous. The framework learns a single set of weights $W_z$ on a subset of the downstream task data to create a conditioned embedding $Θ_j$. This seems to capture properties of the task rather than the intrinsic, pre-trained properties of a specific model. The framework doesn't appear to explicitly model anything unique about $\\phi_i$ and $\\phi_j$ other than their initial feature distributions $E$. \n\n- The latent variable z is introduced as a representation of “intrinsic transferability,” but it is neither grounded nor validated. \nThere is no effort to interpret it, probe its consistency across tasks, or show whether it captures architectural, data-driven, or task-alignment priors.\n\n- The framework's reported indifference to the geometry of the target pseudo-cluster centers is concerning. This suggests the method is capturing a very coarse signal about class separability, which is only useful upto classification and not fine-grained/dense tasks. \n\n- How is the independence of the K subspaces ensured? It is unclear how realistic and well enforced this assumption is\n\n - The method assumes that the final fine-tuned embedding space is cluster-separable, and approximates this via pseudo-centers. But this assumption is quite strong and fragile. It breaks down on tasks where class boundaries are not well-separated in the embedding space (e.g., fine-grained visual categorization, heavy class imbalance, or noisy labels). Nearly all considered tasks are clean image classification datasets.\n\n- Figure 2 is extremely dense and hard to parse. \n\n- Overall, the paper is well written and clear. However, it is hard to follow at a few places. For example, Section 3 is quite dense and a lot of details (such as Deparametric approximation) are only briefly explained. A more detailed explanation would really help enhance clarity.", "was_revised": true, "final_justification": "I thank the authors for their detailed response. Their rebuttal has addressed most of my concerns, so I will increase my score to 4.", "rebuttal": "We thank the reviewer for your recognition of our method's novelty and the supieror performance. Below, we address each point in detail and clarify key motivations and design decisions in ITM.\n\n**[W1:Simplicity of DVA and Linear Updates]** While full fine-tuning is a complex, non-linear process involving sophisticated losses and optimizers, ITM is not designed to replicate this process in its entirety. Rather, its goal is to approximate the overall trajectory of representational evolution in a lightweight and tractable manner—sufficient to estimate relative model-task compatibility without requiring expensive fine-tuning.\n\nThe proposed DVA framework serves as a practical surrogate. It approximates embedding evolution through batch-wise updates toward pseudo-cluster centers, but crucially, this process is guided by a learnable function $f(\\cdot \\mid W_z)$, optimized across the entire embedding space. This enables ITM to capture task-conditioned adaptation patterns unique to each model-task pair, without resorting to static or hand-crafted transformations.\n\nAs a result, ITM’s latent evolution is significantly more expressive than static projection-based baselines or separability heuristics in prior work. We will clarify this motivation and modeling strategy more explicitly in the final version.\n\n**[W2:Role of Latent Variable z in Modeling Transferability]** ITM is designed to estimate transferability at the level of model-task interaction, rather than treating the model or task independently. The latent variable $z$ is introduced to capture this interaction between a model’s representational capacity and the semantic demands of a downstream task. This task-conditioned nature of $z$ is essential, as the effectiveness of a pretrained model depends not only on its own features but also on their alignment with the target task.\n\nSince explicitly modeling $z$ is intractable due to the high-dimensional and non-linear nature of embedding evolution (and the absence of supervision for transferability), ITM adopts an implicit modeling strategy. For each model-task pair, a mapping function $f(\\cdot \\mid W_z)$, parameterized by $W_z$, is learned. This function is shared across all subspaces (batches) for the pair and operates on conditioned embeddings that encode both the static properties of the pretrained model $\\phi_i$ and the distributional characteristics of the downstream task data.\n\nCombined with the pseudo-cluster-based update trajectory, this function simulates how representations adapt to a new task, without requiring end-to-end fine-tuning. We will make the role of $z$ and the modeling mechanism more intuitive in the final version.\n\n**[W3: Additional Evaluation on Latent z]** We thank the reviewer for the insightful suggestion. To further validate and interpret the learned latent variable $z$, we analyze the mapping parameters $W_z$ across different model-task pairs.\n\nWe perform Singular Value Decomposition (SVD) on each $W_z$ and compare the left singular vectors using cosine similarity. These vectors represent the dominant transformation directions in the input space and provide a principled basis for comparing the learned functions.\n\nWe study three pretrained models—DINO-B16 (D), MAE-B16 (M), and MoCov3-B16 (C)—on three tasks: Aircraft (A), Flowers (F), and Cars (R). The resulting similarity matrix is:\n\n| |D-A|M-A|C-A|D-F|M-F|C-F|D-R|M-R|C-R|\n|-|-|-|-|-|-|-|-|-|-|\n|**D-A**|**1.00**|**0.197**|**0.181**|0.133|0.133|0.133|`0.138`|0.134|0.133|\n|**M-A**|**0.197**|**1.00**|**0.215**|0.133|`0.142`|0.134|0.131|`0.158`|0.134|\n|**C-A**|**0.181**|**0.215**|**1.00**|0.131|0.132|`0.135`|0.132|0.132|`0.142`|\n|**D-F**|0.133|0.133|0.131|**1.00**|**0.340**|**0.318**|`0.137`|0.133|0.134|\n|**M-F**|0.133|`0.142`|0.132|**0.340**|**1.00**|**0.349**|0.132|`0.152`|`0.136`|\n|**C-F**|0.133|0.134|`0.135`|**0.318**|**0.349**|**1.00**|0.132|0.133|`0.139`|\n|**D-R**|`0.138`|0.131|0.132|`0.137`|0.132|0.132|**1.00**|**0.301**|**0.283**|\n|**M-R**|0.134|`0.158`|0.132|0.133|`0.152`|0.133|**0.301**|**1.00**|**0.309**|\n|**C-R**|0.133|0.134|`0.142`|0.134|`0.136`|`0.139`|**0.283**|**0.309**|**1.00**|\n\nFrom this analysis, we observe three salient patterns:\n- Task Consistency: Within each task (e.g., Aircraft, Flowers, Cars), the intra-task similarity among different models (i.e., the 3×3 diagonal blocks) remains consistently high. This suggests that ITM learns task-specific transformation functions that are stable across diverse backbones, highlighting its ability to capture task-dependent adaptation behavior.\n- Model Consistency: For each model (e.g., DINO-B16), the inter-task similarity—i.e., similarity between the same model across different tasks—is noticeably higher than that of unrelated model–task pairs. This implies that ITM effectively encodes model-specific inductive biases into the learned mappings.\n- Task Dominance: Overall, task-driven consistency is stronger than model-driven consistency, indicating the downstream task has a greater influence than the model choice. This aligns with Table 4, where performance variance is more pronounced across tasks than across models.\n\n**[W4:Geometry of Pseudo-Cluster Centers]** We appreciate the reviewer’s concern. ITM does not rely on the precise geometry of the pseudo-cluster centers. These centers serve as flexible soft anchors to guide the batch-wise representational updates—not as rigid or semantically grounded class prototypes. Their purpose is to shape the direction of representation evolution without requiring assumptions such as spherical clusters or fixed inter-class margins.\n\nMoreover, the downstream classifier $h$ is independently learned and can adapt to the evolved embedding distribution, even when class boundaries are non-linear or overlapping. This allows ITM to generalize well to tasks that go beyond simple class separability.\n\nWe demonstrate this empirically in Table 3 and Appendix A.7, including segmentation tasks where dense spatial predictions are required. The consistent transferability estimation across both classification and dense prediction tasks supports that ITM is not limited by coarse geometric assumptions.\n\n**[W5: Assumption of Independent K Subspaces]** We would like to clarify that ITM does not strictly enforce subspace independence. Instead, this is a practical approximation adopted to enable scalable and localized modeling. Within the DVA framework, each batch is treated as a locally stable manifold—a heuristic motivated by prior work in contrastive and manifold learning, which has shown that local neighborhoods in the embedding space tend to exhibit semantic coherence and reduced entanglement.\n\nTo evaluate the robustness of this design choice, we compare three batch sampling strategies (random division, uniform sampling, and max-variance sampling) on CIFAR-10. The table below summarizes the maximum variation (Δ) in ITM scores across strategies at different iteration steps:\n\n|Iteration Step|MAE-B16|ResNet-18|DINO-B16|\n|-|-|-|-|\n|200|0.014|0.004|0.001|\n|600|0.006|0.011|0.004|\n|1000|0.005|0.003|0.002|\n\nThese consistently low variations demonstrate that ITM's estimates are highly stable and robust across different subspace configurations, confirming that the subspace assumption does not impact estimation reliability.\n\nWe will include this analysis in the final version and revise Section 3 to more clearly describe the rationale and implications of the subspace decomposition.\n\n**[W6:Cluster-Separability Assumption]** As noted earlier, ITM does not assume hard or perfect separability of the fine-tuned embedding space. The pseudo-cluster centers in our framework are not rigid class prototypes, but soft attractors that guide representation evolution. We make no assumptions about spherical clusters, uniform inter-class distances, or clean label boundaries—this is by design to ensure flexibility and robustness in diverse, imperfect real-world scenarios.\n\nTo support this, we evaluate ITM on 13 downstream tasks that include both classification and segmentation, covering a wide range of complexity levels, label noise, and class distributions. These datasets include fine-grained recognition tasks, imbalanced categories, and dense prediction benchmarks, which present substantial spatial and structural variability. Across these challenging scenarios, ITM consistently demonstrates strong correlation with actual transfer performance (see Table 1, Table 3, and reply to ZajB-W1), suggesting that it does not rely on strict cluster separability to be effective.\n\nIt is also important to distinguish between the pseudo-centers used during optimization and the actual structure of the evolved embeddings. The pseudo-centers are heuristically initialized targets that serve as anchors for the iterative DVA process. However, the final embedding organization is not defined by these initial centers, but rather shaped dynamically through interaction between the model and task during training. As shown in Figure 9, the resulting embedding clusters reflect the underlying adaptation process rather than enforcing a pre-specified geometry.\n\nWe will revise the final version to clarify this conceptual distinction and emphasize that ITM does not rely on strong separability assumptions.\n\n**[W7: Figure2 and Section3- Clarity and Detail ]** Thank you for the suggestion. We will revise Figure 2 by separating the three main stages into clearer sequential panels with better visual flow. Additionally, we will expand Section 3.3 with a more detailed explanation of the deparametric approximation, including its motivation and design choices, to improve clarity and reader understanding.\n\nWe are committed to enhancing the clarity and accessibility of the final version. Please feel free to reach out with any remaining questions—we are happy to provide further clarification."}]

Author rebuttal:

Meta-review: The paper introduces Implicit Transferability Modeling (ITM), a framework that learns implicit, model-specific transferability representations and uses a Divide-and-Conquer Variational Approximation (DVA) to efficiently predict downstream performance across diverse pre-trained architectures without full fine-tuning. 

Reviewers initially raised concerns about algorithmic complexity and significance, evaluation scope and time efficiency, and clarity of presentation. The authors’ rebuttal provided detailed complexity analysis, expanded baselines, timing breakdowns, and clarified notation, fully satisfying all reviewers, who subsequently upgraded their scores and endorsed the work’s novelty and community value. After carefully reviewing the revised discussion, I agree with the reviewers’ final assessment. Therefore, I recommend Accept.
Decision: Accept (poster)