Title: Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transfer learning

Feature extraction

Adaptation models

Representation learning
Abstract: Foundation models (FMs) trained with different objectives and data learn diverse representations, making some more effective than others for specific downstream tasks. Existing adaptation strategies, such as parameter-efficient fine-tuning, focus on individual models and do not exploit the complementary strengths across models. Probing methods offer a promising alternative by extracting information from frozen models, but current techniques do not scale well with large feature sets and often rely on dataset-specific hyperparameter tuning. We propose Combined backBones (ComBo), a simple and scalable probing-based adapter that effectively integrates features from multiple models and layers. ComBo compresses activations from layers of one or more FMs into compact token-wise representations and processes them with a lightweight transformer for task-specific prediction. Crucially, ComBo does not require dataset-specific tuning or backpropagation through the backbone models. However, not all models are equally relevant for all tasks. To address this, we introduce a mechanism that leverages ComBo’s joint multi-backbone probing to efficiently evaluate each backbone’s task-relevance, enabling both practical model comparison and improved performance through selective adaptation. On the 19 tasks of the VTAB-1k benchmark, ComBo outperforms previous probing methods, matches or surpasses more expensive alternatives, such as distillation-based model merging, and enables efficient probing of tuned models. Our results demonstrate that ComBo offers a practical and general-purpose framework for combining diverse representations from multiple FMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript builds on an empirical observation that contemporary foundation models learn different representations (Fig. 2). Consequently, the manuscript proposes a probing method which combines features from multiple frozen foundation models into a single representation. The method begins by taking activations from all layers across all models.  These activations are first processed to equalise the token count for all models. This is done by reshaping tokens from each layer into a 2D grid and applying bilinear interpolation. The tokens are then standardised and concatenated with their counterparts from other models. The resulting matrix S has a dimensionality of (token count * layer count) x (sum of all model dimensions). The matrix S is then affinely projected with learnable parameters (W, b) into a lower-dimensional space. The resulting matrix is then concatenated with a cls token and fed to a shallow transformer F that generates the final representation. Learnable parameters are the task head, transformer F, cls token, and the projection (W, b). Interestingly, the resulting procedure keeps foundation models frozen, which greatly reduces the computational requirements. Experimental evaluation on VTAB-1k indicates that the proposed method on average beats relevant previous works.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts: \n\nS1. The manuscript addresses an interesting problem.\n\nS2. The proposed method is intuitive and easy to follow. \n\nS3. The presented ablations are insightful.\n\nS4. The proposed method performs well on different tasks with a constant hyperparameter setting.\n\nWeaknesses:\n\nW1. Table 1 shows that ComBo improves over SMP and Head2Toe when probing a single foundation model. However, a similar comparison is missing in Table 2 (merging features from multiple foundation models). It is hard to conclude on the effectiveness of the proposed method w.r.t related work without these crucial results.\n\nW2. The method focuses solely on transformer architectures and does not comment on the integration of other architectures (e.g. convolutional models). \n\n\nTypos:\n- L144. Linear projection -> you mean affine projection (linear projection W + translation b)\n\nQUESTIONS:\nQ1. It seems that the manuscript misses the most important experiment. Please answer W1 in detail.\n\nQ2. What if one or more foundation models are irrelevant for some downstream tasks. Could ComBo detect this and exclude such a foundation model during the inference? The benefit would be in inference speedup.\n\nQ3. The VTAB benchmark has a relatively low class count across the tasks (max <400). How does ComBo behave in setups with large class count (over 1k classes) and the VTAB full setting (Table 7)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method to combine the features of multiple pretrained backbones to improve classification accuracy on the VTAB dataset. In particular, the method is a probing based adapter, called ComBo, that utilizes a transformer to compress image features generated from patches at multiple layers from multiple pretrained backbones to a lower dimensional space. ComBo surpasses the performance of competitive probing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well written and easy to read.\n- The experiments are thorough and demonstrate the merits of the approach.\n- The idea of using a transformer to amalgamate the features from multiple pretrained backbones both spatially and layer-wise is novel.\n\n**Weaknesses**\n- The biggest weakness in the approach is the increase in inference time resources (memory, computation) that is proportional to the number of pretrained models $K$. The relatively small accuracy gains (78.2 with ComBo using 4 models from Table 2 versus 77.6 using Adapter+ with a single model) would be difficult to justify in practice given the 4 times increase in inference cost.\n- As a new probing method when utilizing just one pretrained backbone, ComBo falls short of PEFT methods (see Table 1).\n- The paper would be improved by expanding the evaluation to include few-shot datasets such as Meta-Dataset (Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.)\n- A couple of early methods that present various approaches for combining the features of multiple backbones were missed in the references and related work:\n\n    - Chowdhury, A., Jiang, M., Chaudhuri, S., & Jermaine, C. (2021). Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9445-9454).\n\n    - Liu, L., Hamilton, W., Long, G., Jiang, J., & Larochelle, H. (2020). A universal representation transformer layer for few-shot image classification. arXiv preprint arXiv:2006.11702.\n\nQUESTIONS:\n- Have you considered using something other than a linear classifier to predict class probabilities? It has been shown that Linear Discriminant Analysis (LDA) outperforms a linear classifier on VTAB-1K [1]:\n\n[1] Shysheya, A., Bronskill, J., Patacchiola, M., Nowozin, S., & Turner, R. E. (2022). Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. ICLR 2023.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

comments: SUMMARY:
Authors propose ComBo, a method to combine features from multiple foundation models. Specifically a two-stage approach is adopted: features from multiple layers and models are concatenated and projected, which is then passed into a lightweight transformer for further processing. Experiment results show that ComBo is better than other baselines like Head2Toe and SMP.

STRENGTHS AND WEAKNESSES:
Strength:
1. Overall the paper is easy to follow.
2. The proposed method ComBo does not require extensive hyperparameter tuning.

Weakness:
1. The method proposed is not practical. When combining features from multiple models, it requires forwarding multiple models and obtaining their features. In Table 6, if one puts the distillation baseline in it, it should be far more efficient than ComBo since it does not require multiple models’ inference. 
2. When compared with LoRA, the proposed method has worse performance while being less efficient during inference time. Admittedly, ComBo does not require back propagation through the foundation model. However I doubt if the real world has cases where trading inference’s speed for training efficiency is needed. I encourage authors to justify the real-world use case here.
3. In spite of the limited practicality, the ideas raised in future work sections are potentially interesting and it would add in this paper’s value if some of the analysis could be done: e.g. analyzing the parameters to help with model selection/pruning. This may bring insight on understanding different FMs, adding to the value of this work.

QUESTIONS:
see weaknesses

LIMITATIONS:
n/a
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFoundation models learn diverse features based on their training data and objectives. For downstream tasks, different models or layers within these models may provide useful representations. Existing strategies such as fine-tuning and distillation are unable to exploit these complementary strengths or are too expensive. In this paper, the authors present ComBo, a lightweight, scalable adapter based on the probing framework that (i) extracts feature representations from selected layers of multiple models, (ii) normalizes and aligns them spatially via interpolation, (iii) projects them into low-dimensional embeddings using a shared linear projection, and (iv) processes the compressed embeddings with a small transformer. The method operates on frozen models and does away with the expensive backprop into FMs. It also avoids dataset-specific tuning. The approach was tested on 19 vision tasks across natural, specialized, and structured domains. ComBo beats previous probing methods (Head2Toe, SMP), and matches or surpasses full fine-tuning in many structured tasks. It also beats distilled models like RADIOv2.5.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- doesn't require backprob through foundation models making the approach computationally efficient\n- handles activations from multiple models and layers without task-specific tuning unless previous probing methods\n- outperforms existing probing approaches and matches/surpasses full fine-tuning in structured tasks \n- uses fixed hyperparameters making it robust and easier to apply\n- works with both pretrained or tned models and integrates new models without any retraining\n- lower parameter overhead compared to multi-backbone fine-tuning and distillation\n\nWeaknesses:\n\n- The paper assumes that spatial features from different foundation models can be aligned via interpolation to a shared spatial resolution. This may work well for ViT-style models, but how generalizable or robust this assumption is, particularly for architectures with fundamentally different tokenization strategies.\n- novelty of the paper is more system integration than in algorithmic design\n\nQUESTIONS:\nThe paper assumes that spatial features from different foundation models can be aligned via interpolation to a shared spatial resolution. This may work well for ViT-style models, but how generalizable or robust this assumption is, particularly for architectures with fundamentally different tokenization strategies.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}