Title: Implicit Modeling for Transferability Estimation of Vision Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transferability Estimation

Model Selection
Abstract: Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning. This capability facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model’s intrinsic transferability, coupled with a Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables generalization across a broader range of models and downstream tasks. Extensive experiments on a comprehensive benchmark—spanning extensive training regimes and a wider variety of model types—demonstrate that ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Implicit Transferability Modeling (ITM), a framework for estimating how well a pretrained vision model will perform on a new downstream task, without full fine-tuning. Unlike prior methods that explicitly simulate embedding space evolution during adaptation, ITM introduces a latent variable representing model transferability and a Divide-and-Conquer Variational Approximation (DVA) to efficiently simulate embedding adaptation in subspaces. Experiments are conducted on on a range of downstream tasks and models, showing performance improvement in both accuracy and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper proposes a novel idea: learn a latent representation of transferability instead of hardcoding how models evolve. This makes the method more flexible and scalable.\n- The method clearly improves over past work on standard datasets. It runs faster than many dynamic baselines while also being more accurate.\n\n**Weaknesses**\n\n- The DVA, may be an oversimplification of the actual fine-tuning process. The deparametric approximation is derived from a simple MSE loss objective, resulting in a linear iterative update toward fixed pseudo-cluster centers. Real fine-tuning is a complex, non-linear process, typically driven by cross-entropy loss and sophisticated optimizers like AdamW. It is questionable whether the proposed linear evolution simulation can realistically capture these dynamics. The framework may not be modeling fine-tuning so much as it is measuring how well a model's initial embeddings can be linearly mapped to separated clusters—a useful proxy, perhaps, but a proxy nonetheless.\n\n- The central claim of \"implicitly modeling transferability\" using `z` is ambiguous. The framework learns a single set of weights $W_z$ on a subset of the downstream task data to create a conditioned embedding $Θ_j$. This seems to capture properties of the task rather than the intrinsic, pre-trained properties of a specific model. The framework doesn't appear to explicitly model anything unique about $\\phi_i$ and $\\phi_j$ other than their initial feature distributions $E$. \n\n- The latent variable z is introduced as a representation of “intrinsic transferability,” but it is neither grounded nor validated. \nThere is no effort to interpret it, probe its consistency across tasks, or show whether it captures architectural, data-driven, or task-alignment priors.\n\n- The framework's reported indifference to the geometry of the target pseudo-cluster centers is concerning. This suggests the method is capturing a very coarse signal about class separability, which is only useful upto classification and not fine-grained/dense tasks. \n\n- How is the independence of the K subspaces ensured? It is unclear how realistic and well enforced this assumption is\n\n - The method assumes that the final fine-tuned embedding space is cluster-separable, and approximates this via pseudo-centers. But this assumption is quite strong and fragile. It breaks down on tasks where class boundaries are not well-separated in the embedding space (e.g., fine-grained visual categorization, heavy class imbalance, or noisy labels). Nearly all considered tasks are clean image classification datasets.\n\n- Figure 2 is extremely dense and hard to parse. \n\n- Overall, the paper is well written and clear. However, it is hard to follow at a few places. For example, Section 3 is quite dense and a lot of details (such as Deparametric approximation) are only briefly explained. A more detailed explanation would really help enhance clarity.\n\nQUESTIONS:\nPlease refer to the weaknesses section above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach to build a transferability score between a pretrained model and a target dataset.\n\nThis is done by extracting latent features by running the model on the dataset, and then learning a mapping between the latent space and the final fine-tuning performance. The innovation consist of splitting the dataset into batches and to process sub-region of the latent space corresponding to these batches independently.\n\nI should emphasize that I thought the paper very hard to follow and I am not sure of my understanding, despite being used to the literature about transfer learning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- The results obtained by the proposed method appear to be good on the tested datasets and pretrained models. They often outperform the approaches used for comparison.\n\n\n**Weaknesses**:\n- I thought the paper was really hard to follow and I struggled to understand what is being done, and what motivates the choices made by the authors. Many concepts, ideas and notations are introduced without clear explanations and justifications. For example:\n    - What is the rationale behind splitting the space into batches? It is central to the proposed approach, and I could not find a clear justification for this methodological choice.\n    - What is the scope of the learnable function to map embeddings to transferability? Is there a different function for each pretrained model? For each subspace of each pretrained model? Is it the same function (shared weights) for all pretrained models?\n\n- Experiments are not sufficient:\n    - Datasets: only classification datasets, why not experimenting with more complex datasets (segmentation, detection)? The claims talks about downstream tasks in the broad sense, but the experiments only present classification. This limited scope should be clearly stated in the title, abstract, intro. Later on, we discover that the approach was tested on segmentation, why was it not mentioned in the section about datasets description? Why not comparing the different approaches on the semantic segmentation datasets? \n    - Models: 10 pretrained models is really not a lot. Model zoos have way over 1000 models and the real challenge today is how we select a good model among this large pool of models. \n    - Wall-clock time appears particularly fast. I find it hard to believe that it takes less than 10 seconds to assess the transferability of 10 models on CIFAR100, knowing that we need at least one forward pass of each model on each of the 50k images. More details are needed to understand what exactly is presented here.\n    - Comparsion to the literature: Missing references to the compared approaches and details about why they were selected.\n    - “In contrast, existing methods such as PED, LogME, and SFDA—designed around single image-level classification labels—struggle to generalize effectively to new tasks.” —> I don’t see what supports this claim in Table 3.\n\n- In the introduction, I found the limitations of the existing literature very abstract. The result is that I couldn’t grasp the gaps that this paper attempts to fill. For example, “existing TE techniques are primarily evaluated on models with similar architectures and training paradigms […] and fail to generalize to models trained with advanced pre-training strategies or novel architectural designs” —> I don’t see why the approaches presented just above are architecture-dependent or training paradigm-dependent. More explanations are required. Another example: “previous studies attempt to model these dynamics by directly simulating the evolution of embedding spaces, they fail to comprehensively capture both aspects, thereby limiting their generalization capabilities.” —> This is very abstract, what aspects? Why did previous attempts fail? How did they fail? When I reached line 40, it was still not clear what limitations the paper is trying to overcome.\n\n\n\nNote: My concerns about this paper are mostly due to the fact that I believe the paper needs heavy rewriting to improve clarity. I might be wrong in my assessment and if other reviewers feel that the paper is understandable, feel free to favor their review over mine.\n\nQUESTIONS:\nSee questions in the weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of estimating which pre-trained image foundation model will be best-suited for a given downstream task by proposing a method that combines an estimate of out-of-the-box model performance with an approximation of how the model's embedding space might evolve under additional fine-tuning. This is an improvement from current attempts to estimate transferability, which tend to only do static evaluations (i.e. statistical estimates without fine-tuning) or dynamic evaluations (i.e. simulating the fine-tuning process), but not both at once. This method achieves strong performance in estimating transfer on a wide variety of image foundation models on unseen image classification datasets as well as segmentation tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Novel methodology that combines two distinct types of existing approaches in the literature in a computationally-efficient manner\n- Reports SOTA accuracy averaged over a large number of open-source datasets and fine-tuned FMs \n- In an effort to avoid evaluation bias, the authors use official implementation settings to fine-tune existing FMs\n- They are planning to make their code, benchmarks, and pre-trained models publicly available\n- Detailed description of theoretical grounding and implementation in practice\n\nWeaknesses:\n- The paper is concerned almost entirely with the task of single-label image classification, with a small results table for transfer into segmentation tasks at the end, which potentially limits its impact\n- Method is not SOTA in wall-clock time (though it is among the fastest of the methods evaluated)\n- Though the method is evaluated across a number of different benchmarks, none of the results reported in the tables have error bars, making it difficult to fully compare the methods.\n\nQUESTIONS:\n- How should I understand differences between the weighted Kendall's tau_w intuitively? For instance, in Table 1, how significant is the difference between methods yielding a tau_w of 0.61 vs. 0.45, especially considering that LogME is more than 4x faster? \n- What if your primary evaluation criterion is ability to generalize to a wide variety of tasks? Would you recommend repeating this study on a variety of downstream tasks, or is there a better strategy?\n- To what extent does this method generalize more broadly to other downstream tasks such as regression or even multi-modal transfer? \n- Is this method potentially useful in terms of estimating the mutual information between the training dataset and the downstream task dataset?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The authors works on the transferability of vision foundation models, specifically:
1. Proposed a novel method to evaluate transferability through modeling pretraining embedding $\mathbf{E}$ and latent variable $\mathbf{z}$.
2. Reduce compute complexity of proposed method through variational approximation, batch-wise reduction and deparametric approximation.
3. Conducted comprehensive experiments, using including models of difference sizes and beat their other baselines on VTAB.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposed method is novel in the context of transferability estimation, and its presentation is thorough and detailed.
2. The experiments conducted are comprehensive and well-detailed on their testbed.

Weaknesses:
My main concern: seems the method are overly complicated. Which may prevent its real applicability in real world scenarios.

QUESTIONS:
Based on what I specified in "weaknesses"section, I would like the authors to convince me that this work is not overly complicated and to clarify its value in real-world scenarios. For example, with all these approximation approaches, what is their actual upper bound? How does each level of approximation affect the final performance. Then I can decide better on whether or not it is really applicable in real world.

LIMITATIONS:
yes. They left comments in their "conclusion" part
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02