Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
Authors: Benjamin Ramtoula
Pierre-Yves Lajoie
Paul Newman
Daniele De Martini
Venue: NeurIPS 2025 poster ()
Keywords: Transfer learning
Feature extraction
Adaptation models
Representation learning
Primary area: deep_learning
TL;DR: We introduce a new probing architecture capable of efficiently combining features from multiple pre-trained vision foundation models.

Abstract: Foundation models (FMs) trained with different objectives and data learn diverse representations, making some more effective than others for specific downstream tasks. Existing adaptation strategies, such as parameter-efficient fine-tuning, focus on individual models and do not exploit the complementary strengths across models. Probing methods offer a promising alternative by extracting information from frozen models, but current techniques do not scale well with large feature sets and often rely on dataset-specific hyperparameter tuning. We propose Combined backBones (ComBo), a simple and scalable probing-based adapter that effectively integrates features from multiple models and layers. ComBo compresses activations from layers of one or more FMs into compact token-wise representations and processes them with a lightweight transformer for task-specific prediction. Crucially, ComBo does not require dataset-specific tuning or backpropagation through the backbone models. However, not all models are equally relevant for all tasks. To address this, we introduce a mechanism that leverages ComBo’s joint multi-backbone probing to efficiently evaluate each backbone’s task-relevance, enabling both practical model comparison and improved performance through selective adaptation. On the 19 tasks of the VTAB-1k benchmark, ComBo outperforms previous probing methods, matches or surpasses more expensive alternatives, such as distillation-based model merging, and enables efficient probing of tuned models. Our results demonstrate that ComBo offers a practical and general-purpose framework for combining diverse representations from multiple FMs.

Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
Benjamin Ramtoula'
Pierre-Yves Lajoie Paul Newman'
Daniele De Martini
UUniversity of Oxford 2Polytechnqque Montréal {benjamin, pnewman, damiele)@robots ox ac ac pierre-yyves.laoieeooll.... lawoieepollytt..cc..
Abstract
Foundation models (FMs) trained with different objectives and data learn diverse representations, making some more effective than others for specific downstream tasks. Existing adaptation strategies, such as parameter-efficient fine-tuning, fo- cus on individual models and do not exploit the complementary strengths across models. Probing methods offer a promising alternative by extracting information from frozen models, but current techniques do not scale well with large feature sets and often rely on dataset-specific hyperparameter tuning. We propose Combined backBones (ComBo), simple and scalable probing-basse based adapter that effectively integrates features from multiple models and layers. ComBo compresses activa- tions from layers of and processes them with a lightweight transformer for task-specific prediction. Crucially, ComBo does not require dataset-specific tuning backpropagation through the backbone models. However, not all models are equally relevant for all tasks. To address this, we introduce a mechanism that leverages ComBo's joint multi-backbone probing to efficiently evaluate each backbone's 's task-relevance, enabling both practical model comparison and improved performance through selective adaptation. On the 19 tasks of the VTAB-Ik benchmark, ComBo outper- forms previous probing methods, matches or surpasses more expensive alternatives, such as distillation-basee models. Our results demonstrate that ComBo offers practical and general-purpose framework for combining diverse representations from multiple FMs.
more FMs into compact token- wise representations
model merging, and enables efficient probing of tuned
Introduction
Transfer learning of foundation models (FMs) has become the predominant approach solve new downstream tasks in computer vision. Through large-scale pre-training, these models develop general representations, useful across diverse tasks. Practitioners now have access to multiple such models. each trained with specific data and supervision: CLIP [T] to align text and image embeddings, MAE [2 to reconstruct missing image patches, or SAM segment regions images. These differences in training objectives and data distribution influence the learned representations, making specific models more suitable than others for downstream tasks (Fig. Tap.
Ideally, the complementary representations these models should be harnessed when solving a new task, but it is unclear how combine these different models. Whilst effective, tuning techniques usually designed to be applied to individual models and often require backpropagating through the backbone. Trial-and-error of tuning techniques find the best model would mean that users need to
Project page: bramtoula, github. cttchub..o/cmbbl co/combo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.7
CLPP
DINOy2
0.4
SAM
Random
0.1
Layer
CLIP
5VAN
CLIP
DINOV2
SAM
DIND22
Imagehet-22K
Validaaion
0.4
MAE
ComBo
Random
9
10
11
Layer
Task Head
(a) Validation accuracy linear probing on activations from different layers of vision transformers (ViTs) trained from CLIP (1, DINOv2 [ ImageNet-21K 77]. MAE 22. SAM [3] as well as randomly initialised
(b) Combined backBones (ComBo) adapter overview. Our adapter uses ac- tivations from multiple layers of multi- ple models to solve downstream tasks.
ViT. The experiments of the VTAB-IK IK benchmark [10].
the Clevr/distance 8 and SVHN p
tasks
Figure I: (left) Different foundation models learn diverse representations. and given different downstream tasks, the most capable pre-tra pre-trained model might not be the same. Not only can the model vary, but the location of the layer containing the most directly relevant features can also change, with intermediate layers occasionally leading to the highest probing accuracy. (right) We propose an adapter that can take advantage of these diverse representations by probing layer outputs from multiple frozen models. This allows us to efficiently adapt to diverse downstream tasks without needing to backpropagate through potentially large foundation models.
backpropagate through every model to select the best one, which can quickly become computationally expensive as models become larger and more models of interest are released. Other approaches studied in recent works 4 [ distil multiple FMs into single student model. However, this is an expensive process during which can be challenging to preserve all useful features from all teachers.
Probing-based adapters 11 leverage activations from multiple layers of frozen pre-trained models, making them well-suited for combining representations from multiple models without backpropagation. Unfortunately, probing-based adapters still underperform compared tuning methods and require careful regularisation. Moreover, current probing approaches do not scale well to activations from dense feature maps and typically resort average-pooling neighbouring activations to reduce the number of inputs, with pooling region sizes specifically tuned for each dataset.
We propose Combined backBones (ComBo) adapter, a novel probing-basee -bsed adapter that not only overcomes the limitations of prior methods but also enables the combination of activations from multiple models (Fig. lb). We design our adapter to process raw feature maps from pre-trained models without arbitrary pooling, relying only on learnt parameters compress information. ComBo works by combining embeddings from different layers into compact descriptor for each original image patch and training a small transformer to process these descriptors. ComBo scales multiple input models and consistently outperforms previous probing techniques, without requiring extensive task-specific hyperparameter or regularisation tuning. all while maintaining strong performance and efficiency when applied to individual models. While different models can complement each other, specific subsets of models will likely be more relevant for each task. As such, we also introduce method using ComBo to efficiently measure each model's 's task-relevance, providing a practical way for practitioners to select the most relevant backbones for task. We find that using the most relevant combination of backbones can also improve ComBo's efficiency and performance.
Our contributions are follows: 1) An analysis of differences between representations learned by foundation models (FMs), 2) A new probing adapter for pre-trained FMs that demonstrates greater stability and performance than previous probing approaches. 3) Practical demonstration of multi-model integration capabilities: probing multiple large FMs together without backpropagation through any of them, making large model combinations accessible to practitioners with limited computational resources, and 4) method using ComBo compare the relevance of backbones for given downstream tasks, enabling efficient model selection for ComBo and other adaptation methods.

---

DENOV2
MABS
Layer
Layer
Layee 12
Figure 2: Images that maximise activations of different neurons of different models 33 6 technique from Ghiasi et al. [3]. All models rely VIT-B architecture, but each was trained with different data and supervision. Although all models usually pick up low-level patterns in early layers, distinct patterns appear as we get to intermediate and later layers, highlighting di differences in learned representations. Models trained with ImageNet-21K, MAF, or SAM appear more sensitive to specific textures in later layers, whereas late CLIP neurons appear to react to patterns of semantic entities, and DINOv2 late neurons appear more sensitive to complex abstract shapes. These observations align with the idea that diverse features can be found by using multiple models, but also by using multiple layer outputs. Images are generated from randomly sampled neurons.
7 using the
2 Related works
Representations learned by FMs FMs are models trained on large amounts of diverse data and that can effectively be adapted to diverse downstream tasks [4]. Examples include DINOv2 [5), CLIP [ and SAM [3], all based on the ViT architecture [16]. Although there are indications that large- scale models trained with different modalities and supervisions are converging towards a "platonic representation" [17]. current generations of models still display variations in their representations, and consequently on their suitability for given downstream tasks. In fact, multiple previous studies have observed variations in learned representations based on the type of supervision of model 13 18-20). We further validate these studies by comparing the performance of modern FMs by linear probing outputs of different layers of different models, shown in Fig. Ia and by applying the visualisation technique from Ghiasi et al. [3] to produce results shown in Fig. 2
Combining multiple FMs Given the complementary strengths of different FMs, several works combine features from multiple pre-trained models. In meta-learning, Chowdhury et al. [21 show that using a "library" of multiple pre-trained diverse feature extractors significantly outperforms single extractors. Liu et al. [22) propose a "Universal Representation Transformer" that uses attention to dynamically combine features from multiple models, achieving strong few-shot learning performance. However. both only use final layer outputs, ignoring intermediate layers that provide complementary information but require adapter architectures capable of handling the increased feature dimensionality, Other works focus on model merging 23), combining task-specift models without additional training by merging weights 24|26 However, the constraint of not using any additional data is not always present, and improvements can be gained by efficiently using training to combine models. This setting is explored with agglomerative models such as SAM-CLIP 4 and RADIO (27]. which use knowledge distillation
(28) to train student model retaining multiple FMs strengths. However, distillation is expensive and must be repeated for each additional FM, and can struggle to preserve all models fairly, with issues such "mode switch" where the distilled model aligns more with specific teachers at certain resolutions [5)
Tuning vision models Regardless of whether multiple FMs combined used separately, adapting pre-trained models to new downstream tasks requires careful consideration of adaptation techniques, which we categorise into two groups: tuning- and probing- methods. Tuning- based approaches involve updating model weights to modify their behaviour in the forward pass. While full fine-tuning can be expensive, Parameter-efffcient fine-tuning (PEFT) techniques, such as adapters (29]. LoRA [30, and prompt tuning [31), provide techniques to update models with few parameters and often outperform full fine-tuning. In particular, Adapter+ B22 has found that adapter modules (29 in specific configurations and appropriate initialisations can outperform other more complex adaptation techniques without the need for extensive task-specific tuning. However, these tuning methods are typically designed to adapt single model rather than leverage complementary representations from multiple models simultaneously, and they usually rely on backpropagating through the adapted model, which can become prohibitive with multiple large models.
3

---

M
M2
Shared weights A
1110111 100100
0100111
B 1
F
TIII IU]
S
Transformer encoder
CLS 1
Figure 3: The ComBo adapter. Given intermediate feature maps from multiple models, first learn small projection A of their combined layers' embeddings which we apply to all their tokens. These tokens are then passed to small transformer model which outputs a cls token on which we place our classification head.
Probing vision models Alternatively, Sabbing-based approaches rely on extracted activations from frozen models. This approach is particularly appealing when combining multiple FMs, as climinates the need to backpropagate through each model. Hcad2Toe [[T] observed the potential of probing features from multiple layers of a model with a linear layer. Building on this, Structured Model Probing (SMP) [12] noted that different tasks might require more complex adapters than single linear layer. They improved over Head2Toe by introducing new regularisation loss based on structures within the model; they also incorporated an MLP into the adapter architecture, which would naturally be regularised have minimal effect on simpler tasks. While these probing approaches could theoretically be extended to incorporate input features from multiple models, they face practical limitations. They require careful regularisation and hyperparameter tuning dependent on the specific task, and they do not scale efficiently with an increasing number of input features. Both methods typically resort to average-pooling neighbouring activations to reduce computational demands, with pooling region sizes manually tuned for different datasets. In contrast, probing paradigm to efficiently leverage features from multiple FMs without the scaling limitations of previous methods, while eliminating the need for dataset-speciffe hyperparameter tuning.
approach extends the
3 Method
Prior approaches like Head2Toe and SMP [2] have demonstrated the value of probing activations from multiple layers of pre- pre-trained models. They do by stacking activations from different layers and feature map locations and passing them through a linear layer. However, naive stacking can lead very high-dimensional inputs: for ViT-B with 12 blocks, each processing 197 768-dimensional tokens; stacking them and applying them as input of a 100-class linear classifier would require more than 181M M parameters. Head2Toe and SMP manage this by reducing feature map resolution through musk-dependent average pooling, process that we argue can discard important spatial information.
To address this issue, we introduce ComBo, novel probing adapter to efficiently leverage rich representations from multiple FMs while preserving spatial information. We build ComBo around transformer architecture 333 for its proven ability dynamically attend relevant information through its self-attention mechanism, making it ideal for focusing on task-relevant features from and pre-trained models. Rather than processing
pixel values, ComBo's 's transformer processes features ex extracted from pre-trained model activations. This design allows our method to identify and focus the most task-relevant features across different models and spatial locations, while benefiting from the extensive research and optimisations developed for transformer architectures in recent years.
Directly feeding concatenated features from multiple models and layers into transformer would lead to prohibitive input size, especially as we include more models. To address this challenge, we use two-stage approach that separates layer-wise feature selection from spatial feature integration. A first stage introduces an affine projection layer before the transformer to compress high- nigh-dimensional concatenated features from multiple layers spatial locations into wwver-dimensional representation; this approach preserves spatial information by maintaining the token structure while reducing the embedding dimension. The compressed tokens are then processed by ComBo's transformer.
We now formalise our approach in the context of probing features from transformer backbones, as these are the dominant architectures in recent vision FMs. Fig. 3 depicts system.

---

be a set of K pre-trained models, each with N₂ layers L2 (Lr,1. L........ ..... N.,N, J. Given an input image x, each model operates according to its architecture, initially partitioning the image into patches and subsequently processing it as sequence of tokens. At the output of each layer in model M₁. we probe the feature maps FR.J E R grxxDe where Tp is the number of processed tokens and D₂ is the embedding dimension of model Mg. Since different models might produce different numbers of tokens, we interpolate the feature maps to a consistent spatial resolution: we reshape each feature map to its 2D spatial layout and apply bilinear interpolation to obtain a uniform token count across all models. We finally normalise each interpolated feature map to ensure consistent scale across models and layers through mean and standard deviation computed across both token and embedding dimensions. We denote the final processed feature map at layer of model & with the symbol F.J..
Feature map extractions Let M = {M1.
Layer embeddings compression Across the layers, each token index E 21, T) corre- sponds to spatial position in the input image. We can then stack the normalised features from all selected layers of all models at the specific spatial positions by:
S; = =Fi,1 :]: F1,2 :J:...;; ; FK.NN l. RDD
where - -EKK Enn Dz is the total dimension of the stacked features.
To learn a selection of the most relevant features among the layers and models, we learn an affine projection = {W. b) compress the stacked features dimension D' < D. This projection is shared across all token positions, allowing a first selection of the most relevant features from all layers and models considered while preserving the spatial layout of feature maps.
Transformer processing The compressed token representations t = {t,, t,,..., tr} are then processed by transformer encoder F after prepending a learnable class token tak:
0 F((tats t1; ;
For classification tasks, we use the output class token Oats and apply a linear classifier to predict the class probabilities. When training the ComBo adapter, jointly optimise the affine projection parameters (W, b), the learnable class token tck. and the transformer model F.
Our approach is simple but powerful. Separating the feature selection through first affine projection over embeddings of different layers, and then a transformer to combine spatially distributed features enables approach scale well to more inputs without discarding any original feature map activations. Moreover, our transformer leverages strong features from pre-trained models, enabling efficient low-dimensional embeddings while still capturing essential complex patterns.
Evaluating task-relevance of models While combining multiple FMs can improve performance, not all models are equally relevant for every task. Including only the most task-relevant backbones can reduce computational costs while maintaining or improving performance. To identify relevant backbones, we introduce a mechanism inspired by the feature selection Head2Toe [Л], but adapted to select entire backbones rather than individual features. ComBo's 's joint processing of multiple backbones enables us to do so without evaluating each model individually.
To evaluate models' task-relevance we train ComBo with an additional regularisation term applied to the projection weights W in A. For each backbone Mç. we compute an importance score sk (measuring relk-relevnnce) as the &g norm of the projection weights associated with all its layers (i.e., the columns of W corresponding to that model's representations). We add these scores to our training loss via a regularisation term that encourages sparse model selection by penalizing the sum of scores: Liotal
k=1 Lusk + X Ak Sk. where Lusk is the task-specific loss (c.g., cross-entropy) and À is a regularisation coefficient. After training with regularisation, the final importance scores {si, . 5 } directly reflect each model's task-relevance. When selecting model subsets for ComBo, we first train with all models and regularisation to compute importance scores, then select the most relevant models and retrain without regularisation using only this subset.
Applying ComBo to other architectures While our paper focuses on ViT- architectures, which currently represent the dominant paradigm in modern vision FMs, the core ComBo method- ology could likely be extended to other backbone architectures. The general principle remains: (1) extract features from multiple layers of multiple models, (2) align them spatially an architecture- dependent manner, (3) compress them via learned projection, and (4) process the compressed
5

---

features with transformer. For architectures with different tokenisation strategies spatial layouts, such as CNNs, feature maps could be handled through different spatial alignment and interpolation strategies, similar to how we manage different patch numbers in ViTs.
Experiments
Dataset One of the key motivations of our approach is to have access general features that might be useful for diverse tasks. To validate this, we perform experiments on the Visual Task Adaptation Benchmark (VTAB) [O]. VTAB consists of 19 tasks covering widely different image domains, making it ideal for validating the performance of our approach. The tasks are separated into three categories: Natural tasks contain classical computer vision problems (e.g., semantic classification), Specialised tasks focus on images from specialised equipment (e-g., satellite or medical imagery), and Structured tasks require understanding of the structure of the scene (c.g., object counting). All tasks are framed as classification problems, allowing us maintain a similar architecture and metrics for assessing performance in these diverse domains. To validate the performance in low-data settings, we focus on the VTAB-Ik setting with 1000 labelled images per task to train our adapter. We use the provided splits of 800 training and 200 validation images, and the full test sets for final results.
Training We apply ComBo to ViT-B [6] pre-trained frozen backbones and consider feature maps at the output of each block (12 feature maps per model). For each VTAB-1K 1 task, we process the feature maps through the ComBo adapter (See. 3). The cls token Ous at the output of our transformer model F is then fed projection and our transformer (FF using a cross-entropy loss.
linear classifier head that train jointly with parameters of our affine
Where Head2Toe and SMP optimiser settings, and regularisation strength, we find that ComBo performs well on all tasks with constant settings. As such, we follow the settings used in Adapter+ [32]: for all datasets, we a batch size of 64 and an Adam V optimiser [34] with a learning rate of 0.001 and a weight decay of 0.0001. We train model for 100 epochs with linear warmup over 10 epochs and cosine scheduler. When evaluating the task-relevance of models, we use a regularisation coefficient À of 0.01. We report the mean accuracy value over seeds. Following the original VTAB settings and Adapter+ [10 32), transformer F, we resize images to 224 x 224 pixels and do not use any augmentations. For standard implementation from the timm library 331 with depth of 6 blocks, an embedding dimension of 128, and heads (1.7M parameters). When combining multiple models with different tokenisation settings (e.g., DINOv2 ViT-B/14 (6) and SigLIP ViT-B/16 (36), we interpolate feature maps to the smallest shape. For ComBo adaptations, we discard any backbone cls and register tokens and only use tokens corresponding to initial image patches.
require hyperparameter tuning of feature pooling region size,
4.1 Probing a single backbone
In Tab. we first validate the performance of ComBo when probing single model, specifically ViT-B/16 trained on ImageNet-2IK (7],,, commonly used to evaluate adaptation methods.
Parameter-efficient tuning methods consistently outperform probing methods in our evaluation, with Adapter+ achieving the best overall performance despite using fixed hyperparameters across tasks. This is unsurprising as these methods preserve the pre- pre-trained model's 's structure while efficiently updating the forward pass, making them effective even with limited training data. Probing methods typically less performant since they face the challenge of learning optimal feature combinations from scratch, particularly in low-data regimes. Despite this difficulty, our proposed ComBo adapter outperforms both full fine-tuning and previous probing approaches.
We observe that while ComBo, Head2Toe, and SMP perform comparably on natural and specialised tasks, ComBo demonstrates substantial improvements structured tasks. This performance gap likely stems from fundamental differences in how spatial information is processed. Previous ap- proaches employ average pooling across feature maps, which reduces spatial resolution and potentially discards critical information for structured tasks. In contrast, ComBo initially preserves the complete original feature maps, learning which information preserve.
key practical advantage of ComBo its effectiveness with fixed hyperparameters across all datasets. Unlike Head2Toe [|]] and SMP [2], which require dataset-specific hyperparameter tuning, our

---

Table 1: Comparing tuning and probing techniques for VIT-B/16 trained ImageNet-21K 37 the VTAB-Ik Ik benchmark. We report accuracy (%) on the test set. indicates methods requiring task-specific hyperparameter optimisation. Our proposed ComBo adapter outperforms other probing methods and full line-tuning while maintaining fixed hyperparameters across all datasets, demonstrating particular strength on structured tasks where spatial information preservation is critical. PEFT methods (e.g.. Adapter+) achieve the highest performance overall. highlight the best result for each adapter category in bold.
Natural
Specialised
Structured
Caltech10 38
CIFAR 100
Howers 102
Sun397 33
SVHN 44
Camelyon F5
BrmSAT 46
Relineqalty 48
x
Cleve Count
Clev-Dat g
dSpe- SgeeLLcc 50
dSpe-On 50
S1
KITTI Dis
SNORD- Azim-iim
WORB--E Ele
Global Average
DTD A0
Pets 42
Avernge
Resise45
Average
DMLab
Average
Tuning methods, require backpropogeting through
adupted model
Fall
92.6
73.2 70.4 97.9 86.2
19.6
90.6
78.6 87.1 96.6
87.5
74.0
86.3
66.6
61.0
49.8
82.6
51.9
79.7
33.5
37.0
57.8
74.2
LaRA
91.7
90.9 56.7
95.7
77.7
62.3
49.0
31.0
47.0
VPT-Deop
31
93.0 83.0
71.2
99.0
91.3
56.0
84.1
82.5
84.9 96.6
82.5
74.5
84.6
77.5
58.7
49.7
86.2
56.1
79.6
37.9
50.7
62.1
76.4
Adapter+
94.2 83.7
90.6
55.8
88.2
83.3
87.5 97.0 87.4
72.9
82,9
60.9
53.7 88.4
55.2
80.8
37.3
46.9
63.3
Probing methods, do not require buckpropuggatnng through the edopted modell estendoble profe mabiigte modelss
Linear probing 88.1 78.1 69.0 90.0
56.9 36.0 73.9
90.7
79.5 32.4 30.5
35.9 11.2
14.3
24.5
29.6
HeadEToe & [1]] 90.9 75.3 75.0 99,5 86.1 50.9 K3.5 80.2 84.4 95.7
84.4
74.3
84.7 51.7 59.4 44.0
47.1 40.3
65.6
32.5
41.1
47.7 70.9
SMP [22 90.9 79.3
90.4
55.3
75.0
77.5 58.0
40.8
33.0
49.0
55.4
ComBo tours)
91.3 76.7 69.8
99.3
90.0
49.3
81.4
79.7 84.9 96.6 84.6
71.8
84,5
81.7
60.0
46.5
89.0
47.6
77.9
28.4
45.1
59.5
74.6
Table 2: Probing different foundation models with ComBo on VTAB-IK. We report accuracy (%) on the test set. We compare probing individual foundation models with ComBo, probing all models simultaneously. probing the top two models selected based on our task-relevance estimation method, and fine-tuning a distilled model (RADIOv2.5) with Adapter+. Our multi-model ComBo approach achieves the highest global average, demonstrating that direct probing of multiple FMs can outperform approaches requiring expensive distillation, even with strong parameter efficient tuning. The best result for cach task is highlighted in bold and the second- best is underlined.
Nutural
Specialised
Structured
Global Average
#NORB-A Azim
Coltech101
CIFAR 100
Flowers 102
Sun397
SVIN
Avrrage
Camelyon
BeoSAT
Resisc45
Resimopathy
Awrage
Clew Count
Cleve-Dis
DMLab
dSpe-Loc
aSpe-Oni
KITTI-Dis
SNORD Elev
AMrage
DTD
Pots
Probing individuul foundlation models with ComBo (owrs)
53
70.8 77.2
87.9 44.4 83.1 79.3
96.7 89.1
23.3 85.8
87.2
60.7
51.6
81.9
79.6
75.6
DINOv2 6
94.7 79.4 78.4 99.7 91.1 49.3 85.9 82.6
86.0
96.2
88.3
73.2
85.9
92.6
60.4
51.8
88.2
53.3
82.0
36.4
55.6
65.0
77.9
SAM 3
76.7 25.6 52.4 80.0 55.6 18.3 77.6
55.2
78.8
94.9
75.0
68.8
79.4
85.6
58.5
43.3
84.2
42.8
78.7
23.3
39.0
56.9 63.8
SigL.P
94.0 66.6 79.2 98.7 92.4 53.1 79.2
80.5
83.0
96.0
89.6
73.2
85.4
88
60.0
54.5
85.0
48.9
81.6
32.8
48.9
62.6
76.2
Probing multiple foundation models with ComBo
All models (ours) 79.2 79.1 99.6 92.6 50.4 83.3
97.0
90.5
86.5
91.5
54.1
89.7
82.3
34.1
64.8 78.2
Top two models (ours) 95.1 80.5 78.9 99.7 93.3 53.4 87.3 84.0
85.3 96.9
89.9
73.3 86.3 92.0 60.9
55.0
90.6
52.7
82.2
33.9
55.3
65.3
78.6
Distifling four foundation models above and tuning resulting model with Adapter+ [2] RADIOv2.5
S
91.2 81.8 71.5 97.2 89.3 55.7 93.9 83.8 86.5 97.1 89.3 73.5 86.6
86.1 58.6 54.9 85.6 54.7 81.8 33.1 41.9 62.1 77.5
approach maintains consistent performance without task-specific optimisation. This characteristic makes ComBo more readily applicable to new tasks compared to previous probing approaches.
4.2 Probing multiple backbones
We now evaluate ComBo's capacity to probe multiple FMs simultaneously. As a baseline, we consider an alternative approach to combining representations from multiple FMs: distillation. Specifically, we compare against the ViT-B/16 model from the RADIOv2.5 family [5). trained through distillation to merge representations from four FMs: DFN CLIP (3), DINOv2 (6) SAM (3) and SigLIP (36). We compare four approaches: (1) applying ComBo to each individual FM that contributed to RADIOv2.5, (2) using ComBo to probe all models simultaneously, (3) using ComBo to probe the top two models selected for each task with our task-relevance estimation mechanism (Sec. B) and (4) fine-tuning the distilled RADIOv2.5 model with Adapter+. Results are presented in Tab.
key finding is that ComBo applied to all models simultaneously consistently matches or outperforms probing the best individual model across all tasks. However, individual model contributions vary considerably, with SAM consistently underperforming other models on VTAB- tasks. Our task- relevance estimation mechanism (Sec. 3 addresses this by identifying the most relevant models
7

---

Table 3: Performance when combining VIT-B models individually tuned with Adapter+ [32] VTAB-Ik. We report accuracy (%) on the test set. We compare individual adapted models against two approaches for combining them: Multi-Adapter+ (our baseline that trains adapters on all models simultaneously with shared classification head) and ComBo (our method that probes all adapted models). While DINOv2 achieves the highest global average (79.9%), both combination approaches excel on more individual tasks (Multi-Adapterr 8/19, ComBo: 5/19). ComBo offers comparable performance to Multi-Adaples with significantly lower computational requirements. Best results are in bold and second-hest hest are underlined.
Natural
Specialised
Structured
Global Avernge
Caltech
CTFAR- 100
DTD
Flowerst 122
Sun397
SVHN
Average
Camelyon
EunSAT
Resise45
Retinopshty
Average
Clev-Count
Clev-Dist
DML ab
dSpe Loc
dSp--Dii
TITTI-Dist
SVORD-Azim
SNORD- Azim
sNORB-Eles
-lle
Average
Pets
Individunl adopted models
CLIP
53
94.5
76.2 78.9 98.1 89.9
54.8
93.1 K3.7 87.3 97.1
90.3
73.8 87.1 92.0
60.7
59.9
88.2
56.0
82.7
41.2
53.2 66.7
79.2
DINOv2 6
92.9 83,5
78.7 99.7
93.3
55.2
92.8
85.2
89.2 96.9
88.4
73.9
87.
91.1
61.5
58.5
86.7
57.8
83.0
43,9
56.8
67.4
79.9
SAM 3
75.2 32.6
57.7 74.7
51.4
18.6
87.1
56.7
82.9
95.4
72.7
72.1
80.8
81.2
60.1
45.0
78.1
46.7
81.3
28.1
47.0
58.4 65.3
SigLIP 36
92.0 70.0 78.8 97.9 91.3
51.6
92.7
82.0
85.5
97.3
89.3
75.0
86.8
92.8
61.8
59.1
90.6
55.7
83.0
42.1
60.4
68.2 79.0
Appraaches combining modelss above
Multi-Adapter 95.1 83.8 80.9 99.7 93.8
57.8
93.1
86.3
87.5
97.2
91.7
76.2
88.2
79.9
54.9
56.6
80.0
59.3
82.6
32.2
47.7
61.6 78.7
ComBo (ours)
95.2 78.9 78.7 99.4 90.8 46.3 93.9 83.3 S8.4 97.4 89.4
74.4
87.4
90.3
62.6
58.2
87.4
56.2
85.3
41.2
57.5 67.3 79.3
per task. Selecting the top two backbones by importance score for each task achieves the strongest overall performance 778.6% global average), improving upon the all- all-models approach (78.2%) while requiring fewer backbones to process. Thus, while probing all models is preferable to any single model, selecting only the most relevant backbones further improves performance.
Comparing against distillation, both ComBo variants (all four models: 78.2%; top two: 78.6% outperform RADIOv2.5 with Adapter+ (77.5%). While results are comparable on natural and specialised tasks, ComBo demonstrates consistent advantage on structured tasks (64.8% and 65.3% vs. 12). ComBo achieves these competitive results without requiring the computationally expensive distillation process necessary for RADIOv2.5. This performance gap may stem from limitations in the distillation process, either incomplete capture of teacher representations or capacity constraints imposed by the combination in single ViT-B/16 architecture. A key advantage of our approach is that ComBo ensures direct access to the original representations from each FM, preserving their complementary features. Furthermore, ComBo offers greater flexibility than distillation-basel approaches, allowing for seamless incorporation of new models.
4.3 Prohing tuned backhones
As established in Sec. 4.1 parameter-efficient tuning methods like Adapter+ produce powerful task-specific representations. Here, we explore how ComBo performs when applied to FMs that have already been tuned for specific tasks, addressing two questions: (I) whether ComBo can effectively combine multiple tuned models, and (2) which layers are most informative in tuned models.
Combining multiple tuned backbones Building these findings, we investigate how effectively ComBo can combine multiple FMs already individually tuned to specific tasks. We compare our approach against a strong baseline, Mlulti-Adppter which jointly fine-tunes all models with Adapter+ and combines their outputs through a trained linear head. While both approaches use tuned models, Multi-Adapter+ requires joint optimisation with backpropagation through all models during training, making it computationally expensive but potentially powerful.
Results in Tab, 3 compare tuned individual models with Multi-Adaptert and our ComBo approach. DI- NOv2 with Adapter+ achieves the highest average accuracy (79.9%), showing strong general-purpose purpose performance. However, it leads in only 3 of 19 tasks, suggesting its strength lies in consistency, unlike combination methods that can be influenced by weaker models on certain tasks.
Examining the combination approaches, Multi-Adaptesn excels on 8 of 19 tasks and achieves the highest performance on natural tasks (86.3%) while ComBo wins on 5 of 19 tasks and demonstrates superior performance on structured tasks (67.3% vs. 61.6%). Overall, ComBo achieves global aver- age accuracy (79.3%) that outperforms Multi- Adati-- (78.7%) and approaches DINOv2 (79.9%), confirming its effectiveness despite using more efficient combination strategy.
8

---

available Table 4: Effect of probing all layers versus only the last layer with ComBo, using the original ViT-B/16 trained on ImageNet-21K [7] and versions adapted with Adapter+ [3]]. We report VTAB-IK accuracies (%) on the vol set. Results demonstrate that while probing all layers is critical for non-adapted models, adapted models concentrate task relevant information in the final layer.
Probing the original model
Probing the tuned model
Natural Specialised Structured Average
Natural Specialised Structured Average
Last layer
72.2
77.6
41.1
63.6
82.8
87.1
60.7
76.9
All layers
82.1
87.0
60.5
76.5
82.1
87.0
60.5
76.5
These results demonstrate that ComBo, despite working with independently tuned models rather than jointly optimised ones, offers comparable performance to Multi Multi--Adaperr+. This is significant given that ComBo requires only forward passes of pre-tuned models, making more efficient and scalable when dealing with large FMs or when combining models. The performance gap between combination methods and individual models like DINOv2 also indicates potential for further improvements in how ComBo manages models of varying quality, such as with our task-relevance evaluation mechanism.
Importance of probing different layers in tuned backbones We also investigate the relative importance of different layers when probing tuned models. In Tab. 4 we compare the performance of ComBo when applied to either all layers or only the last layer of a ViT-B/16 model, with and without prior adaptation using Adapter+. While probing all layers is crucial for non-tuned models, this advantage disappears for models tuned with Adapter+. This finding aligns with the intuition that tuning processes concentrate task-relevant features in the final layers where classification heads are applied. This entails that the efficiency of ComBo can be improved when probing multiple tuned models, restricting probing to models' final layer reduces computational overhead without sacrificing performance.
4.4 Selecting subsets of backbones
Table 5: Validation of models' task-relevance estima- tion. Accuracy (%) on VTAB-IK Ik val set using different strategies to select from 4 ViT-B backbones (DFN CLIP, DINOv2, SAM, SigI.IP) to probe with ComBo. Best
In Tab. 5 we validate our model importance scoring method from Sec. 3 by probing mod- els with ComBo, where models are selected with different strategies from the four FMs from Sec. 4.2 We establish an upper bound with an exhaustive search over all model combinations. As naive selection strategies, we report perfor- mance using all models, only DINOv2 (the best single model), and random pairs (averaged over all possible pairs). We compare these against selections based on our method's produced im- portance scores: top-1, top-2, and top-3 mod- els. The large gap between exhaustive search and baselines highlights the importance of good model selection. Our importance scores consis- tently outperform naive strategies, with top-2 performing best overall when using fixed number of models n across tasks. However, the ideal number of models varies by task. By selecting the optimal n individually for each task from importance rankings (best per-task top-n), performance approaches the exhaustive search upper bound, validating the ability of our method to estimate models' relk-rele Details of the models' importance scores and per-task optimal in the appendix (Sec. B.7
per-task top-n consists in selecting leading to the hest performance for each task.
Nat.
Spec.
Struct.
Avg.
Uning exhoustive search Best combinations (upper bound)
83.6
88.1
64.0
78.6
Using naive selection
All 4 models Only DINOv2 (best single)
81.8
86.4
60.3
76.2
80.7
86.2
61.1
76.0
Random models (avg.)
80.2
Uxing our estimuted importuncee score Top-1
81.0
86.5
61.2
76.2
Top-2
82.8
87.0
60.1
76.7
Top-3
82.6
86.1
61.1
76.6
Best per-task
83.3
87.4
63.7
78.1
Table 6: Validation of ComBo design choices. We report accuracy (%) on the VTAB-Ik vol set when adapt- ing the ViT-B/16 model trained on ImageNet-21K.. K...
4.5 Design validation
Tab. evaluates key components of the ComBo adapter. We validate that feature map normali- sation is essential, as its removal causes perfor- mance decreases in all task categories. Architec- ture comparisons validate the value of using transformer, as replacing it with a linear layer or the MI.P architecture used in SMP 12 leads to important drops in performance.
Nat.
Spec.
Struct.
Avg.
ComBo adapter
79.3
85.6
56.5
73.8
w/o feature map normalisation
79.2
84.9
50.5
71.5
Replacing the transfirmer
w/ linear head
67.4
82.6
45.4
65.1
w/ the MLP from SMP [2)
71.6
83.2
47.8
67.5
9

---

Table 7: Comparison of adapter sizes, computational costs, and memory requirements when adapting ViT-B models. report number parameters and FLOPs relative to full fine-tuning, and peak GPU memory usage.
Results are reported for processing a batch of 32 images of size 224 x 224, which was the highest power of 2 that fit in our GPU memory for all methods.
Size
Train
Inference
Method
Updated params
Rel. FLOPs
Peak GPU mem.
Rel. FLOPs
Peak GPU mem.
Adapting one backbone
Tine-Tuning
100.00% 0.01%
100.00%
4,736
100.00%
0.595
Linear Probing
33.18%
0.603 GB
100.00%
0.595 GB
Adupter+
0.38%
65.58%
2.993 GB
100.33%
0,596 GB
ComBo adapter
2.79%
35.78%
0.929 GB
103.00%
0.825 GB
Adapting four backbones
Fine-Tuning
100.00%
100.00%
21.675 GB
100.00%
1.738 GB
Multi-Adee Adapter+
0,38%
65.67%
13.246 GB
100.33%
1,742 GB
ComBo udapter
1.71%
34.41%
3.212 GB
101.60%
3.159 GB
4.6 Computational efficiency
ComBo offers several computational benefits over existing baselines, partieularly when combining multiple FMs. To compress multiple models into one, distillation methods require expensive training (c.g.. SAM-CLIP and RADIOv2.5 used 64 large GPUs. with the latter training on 614M examples 4,5) and lack fiexibility to incorporate new models without retraining, In contrast, ComBo can be trained ~ minutes to probe four models on VTAB-IK Ik task using a consumer GPU.
Tab. 7 computational costs of different adaptation methods for ViT-B/16 backbones. ComBo requires significantly fewer training FLOPs due to the absence of backpropagation through backbones: only 35.78% vs. 65.58% for Adapter+ (single backbone). and 34.41 % vs. 65.67% for Multi Aulti-Adapter++ (four backbones). Memory requirements also substantially lower, with ComBo using ~ GB vs. 13 GB and 22 GB for Multi-Adapter+ and fine-tuning. For instance, using an NVIDIA RTX 3090 Ti (24GB), ComBo can probe large models like DINOv2-g/1 14 that would exceed memory limits when adapted with Adapter+ without additional memory optimisation techniques. While ComBo introduces modest inference overhead due its additional parameters and transformer processing, the impact is minimal in practice, adding at most 3.00% increase in FLOPs to the original backbone(s).
5 Conclusion
Motivated by the important role of foundation models in computer vision and by the differences between representations learned by existing models, we propose ComBo, a probing- -based adapter designed to process feature maps from multiple frozen models. ComBo outperforms prior probing- based adapters without requiring task- task-specific hyperparameter tuning and demonstrates the ability to effectively combine features from multiple models on the VTAB-I k benchmark. Furthermore, introduce a method using ComBo to efficiently estimate the task-relevance of different models. This enables practitioners to improve both efficiency and performance by focusing computational resources on the most relevant models. While ComBo alone does not yet match the performance of tuning- based approaches such Adapter+, we show that it can successfully integrate the representations of multiple models that have been fine-tuned individually, offering a promising solution for efficient model combination.
Limitations Our work has several limitations. First, ComBo assumes consistent feature map shapes across models. While we use interpolation to handle heterogeneous architectures, this may degrade representation fidelity. Second, using a single affine projection for all layers may limit scalability to include more model layers. More flexible or hierarchical projection mechanisms could improve scalability and representational capacity. Third, ComBo inherits representational biases from frozen models, which may affect downstream predictions. Fourth, while scoring mechanism successfully ranks models by task-relevance, it remains unclear how to automatically determine the optimal number of models maximise performance without evaluating each possible option. Finally, our experiments focus on ViT-B architectures and VTAB-Ik's low-data classification tasks. Evaluation on larger models, higher-resource settings, and dense prediction tasks would help to establish the broader generalisability of ComBo.
10

---

Acknowledgements
This work was supported by EPSRC Programme Grant "From Sensing to Collaboration" (EP/V000748/1), the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (EP/S024050/1), Oxa, and the Digital Research Alliance of Canada.
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 7748-8763.
PMLR, 2021.
[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15979-15988, 2022.
[3] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023.
[4] Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mchrdad Farajtabar, Sachin Mchta, Mohammad Rastegari, Oncel Tuzel, and Hadi Pouransari. SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3635-3647, 2024.
[5] Greg Heinrich, Mike Ranzinger, Hongxu Danny Yin, Yao Lu, Jan Kautz, Andrew Tao, Bryan Catanzaro, and Pavlo Molchanov. RADIOV2.5: Improved Baselines for Agglomerative Vi- sion Foundation Models. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22487-22484 2025.
[6] Timothee Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers Need Registers. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[7] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers. Transactions Machine Learning Research, 2022.
[8] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1188-1997,0 2017.
[9] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[10] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark, 2020.
[11] Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2Toe: Utiliz- ing Intermediate Representations for Better Transfer Learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 6009-6033, 2022.
[12] Zhi-Fan Wu, Chaojie Mao, Xue Wang, Jianwen Jiang, Yiliang Lv, and Rong Jin. Structured Model Probing: Empowering Efficient Transfer Learning by Structured Regularization. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), pages 16849-1688110 16858, 2024.

---

[13] Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do Vision Transformers Learn? A Visual Exploration, 2022.
[14]
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa. Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou. Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the Opportunities and Risks of Foundation Models, 2022.
[15] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, 2023.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, 2021.
[17] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The Platonic Representation Hypothesis. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20617-20644 PMLR, 2024.
[18] Matthew Gwilliam and Abhinav Shrivastava. Beyond Supervised vs. Unsupervised: Repre- sentative Benchmarking and Analysis of Image Representation Learning. In 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9632-9642 9642. IEEE,
2022.
[19] Tom George Grigg, Dan Busbridge, Jason Ramapuram, and Russ Webb. Do Self-Supervised and Supervised Methods Learn Similar Visual Representations? In Proceedings 2nd Workshop on Self-Supervised Learning: Theory and Practice (NeurIPS), 2021.
[20] Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching Matters: Investigating the Role of Supervision in Vision Transformers. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7486-7496. IEEE, 2023.
[21] Arkabandhu Chowdhury, Mingchao Jiang, Swarat Chaudhuri, and Chris Jermaine. Few-shot image classification; Just use a library of pre-trained feature extractors and a simple classifier. In Proceedings of the International Conference on Computer Vision (ICCV), pages 4454-9454, 2021.
[22] Lu Liu, William L. Hamilton, Guodong Long, Jing Jiang, and Hugo Larochelle. A Univer- sal Representation Transformer Layer for Few-Shot Image Classification. In International Conference Learning Representations, 2021.
12

---

[23]
Yi-Lin Sung, Linjie Li. Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An Empirical Study of Multimodal Model Merging. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[24] George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Ziplt! Merging Models from Different Tasks without Training. In The Twelfth International Conference on Learning Representations, 2024.
[25] Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Korbllith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems, 2022.
[26] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the 39th International Conference on Machine Learning, volume 162. pages 23965-23998, 2022.
[27] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12490-1250M 2024.
[28] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015.
[29] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter- Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97,
pages 2790-2799, 2019.
[30] Edward Hu. yelong shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[31] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In Proceedings of European Conference on Computer Vision (ECCV), page 709-727, 2022.
[32] Jan-Martin O. Steitz and Stefan Roth. Adapters Strike Back. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23449-23459, 2024.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the s1 International Conference on Neural Information Processing Systems, page 6000-6010, 2017.
[34] Iyya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, 2019.
[35] Ross Wightman. PyTorch Image Models.
https:///github.com/rrrgg
pytorch- mww.g-mddeeeellssssseee 2019.
[36] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre-Training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975-11986, 2023.
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathcesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vision, 115(3):211 1-222, 2015.
[38] Li Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4):994-611, 2006.
[39] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[40] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing Textures in the Wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 6606-3613 2014.
13

---

[41] Maria-Elens Nilsback and Andrew Zisserman. Automated Flower Classification over Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729, 2008
[42] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pages 3498-3505, 2012.
[43] Jianxiong Xiao, James Hays. Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages
2010.
[44] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading Digits in Natural Images with Unsupervised Feature Learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[45] Bastiaan S. Veeling, Jasper Linmans. Jim Winkens, Taco Cohen, and Max Welling. Rotation Equivariant CNNs for Digital Pathology. In Alejandro Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López. and Gabor Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pages 210-218, 2018
[46] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-
2226, 2019.
[47] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote Sensing Image Scene Classification: Benchmark and State of the Art. Proceedings of the IEEE, 186(10))1865-11 2017.
[48] Emma Dugas, Jared, Jorge. and Will Cukierski. Diabetic Retinopathy Detection. https
2015. Kaggle.
[49] Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich
Küttler, Andrew Lefrancq. Simon Green, Victor Valdés, Amir Sadik, Julian Schritwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. DeepMind Lab, 2016.
[50] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites: Disentangle- ment testing Sprites dataset.
wttgggtthwh.oooeeeeeed/d///pppie dataset/, 2017.
[51] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets Robotics:
The KITTI Dataset. International Journal of Robotics Research (IJRR). 2013.
[52] Y. LeCun, Fu Jie Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume pages II-104 Vol.2, 2004.
[53] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data Filtering Networks. In Proceedings of the Twelfth International Conference Learning Representations (ICLR), 2024.
[54] Brigit Schroeder and Yin Cui. FGVCx fungi classification challenge, 2018.
[55] Aliaksandra Shysheya. John F Bronskill, Massimiliano Patacchiola, Sebastian Nowozin, and Richard E Turner. FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification. In The Eleventh International Conference on Learning Representations, 2023.
14

---

A Additional details
A.1 Additional dataset details
We summarise the details of the datasets used in the VTAB benchmark 10 in Tab.
Table Details the datasets in VTAB benchmark [0] used. When experimenting on VTAB-IK setting or estimating models' task-relevance, we use 800 training samples and 200 validation samples. When running our final tests on the VTAB-Ik setting, we use the 1000 training samples.
Ik setting
full setting
Dataset
#Classes
Train
Val
Train
Val
Test
Natural
Caltech101
102
800/1,000
200
2,754
306
6,084
CIFAR- 100
39
100
800/1,000
200
45,000
5,000
10,000
DTD 40
47
800/1,000
200
1,880
1,880
1,880
Flowers 102
102
800/1,000
200
1,020
1,020
6,149
Pets 42
37
800/1,000
200
2,944
736
3,669
Sun397 43
397
800/1,000
200
76,127
10,875
21,750
SVHN 44
10
800/1,000
200
65,931
7,326
26,032
Specialised
Patch Camelyon
45
2
800/1,000
200
262,144
32,768
32,768
EuroSAT [66
10
800/1,000
200
16,200
5,400
5,400
Resise45
47
45
800/1,000
200
18,900
6,300
6,300
Retinopathy
48
5
800/1,000
200
35,126
10,906
42,670
Structured
Clevr/count
8
800/1,000
200
63,000
7,000
15,000
Clevr/distance
6
800/1,000
200
63,000
7,000
15,000
DMLab [49
6
$00/1,000
200
65.550
22,628
22,735
dSprites/location
50
16
800/1,000
200
589,824
73.728
73,728
dSprites/orieenttiin
[50
16
800/1,000
200
589,824
73,728
73,728
KITTI/distance [51)
4
800/1,000
200
6,347
423
711
SmallNORB/azimuth
52
18
800/1,000
200
24,300
12,150
12,150
SmallNORBBelleattion
52
18
800/1,000
200
24,300
12,150
12,150
A.2 Additional backbones details
We share details of all backbones used in experiments in Tab.
For ComBo we do not use cls or register tokens. For other baselines, the pooling used before the linear head depends on the model and is specified in the table.
Table Details backbones used our experiments. We only report pooling strategies when they were used our experiments, such as when tuning the model with Adapter+. models are loaded and adapted from except for RADIOv2.5 which loaded from the PyTorch Hub. The "-CPE" suffix indicates the use
timm 35
of Cropped Position Fmbeddings as implemented in RADIOv2.5 (5)
Model
Architecture
Model puth in timm PyTorch Hub
Pooling used
CLIP 1
VIT-B/16
timm/vit_ wwweeettbbbeeeeeeeeeeeee Lat___bbusspl______e wtt_ppucc___hhhpep 1166__li__2____ 224.openai
DEN CLIP
58
VIT-B/16
timm/vit base base_patch1e wttppppppp___ 221_2224.f22
cls token
DINOV2
VTT-B/14
timm/vit. base casse_patch1e 14_ reg4 dinov2. dinov2.lvd1422m 1vd1122m
cls tnken
ImageNet-21K
VIT-B/16
timm/vit, base wpppcccc166.... 2266_2224rrrnnieee orig__iilllk in21k
token
MAE 2
VIT-B/16
timm/vit base ppppppaac..... batpiiaaaaaapeee6ee_aaa
224.mae
RADIOv2.5
1
WTT-B/16-CPP CPI
NVlahs/RA _ v2.5-b
summary (equivalent to clas token)
SAM 3
VIT-B/16
timm/samvit, base fllp_paachh.sss)
average pooling
LiP
36
VYT-B/16
timm/vit, base Bccee_pacch66,.1 sigllspggppppp_ webl
average pooling
15

---

A.3 Additional training details
Computational details All our experiments are performed using BFloat16 Mixed precision. Our experiments were mainly performed machine with one NVIDIA GeForce RTX 3090 Ti GPU and an Intel(R) (R) Silver 4210R CPU @ 2.40GHz with 40 logical CPUs. With our system, training one ComBo adapter probing one backbone one VTAB- Ik dataset takes approximately 5 minutes, and training one ComBo adapter probing four backbones on one VTAB-IK dataset takes approximately 10 minutes.
Tuning and probing baselines Results presented in Tab. are reported from Adapter+ [32 for the tuning methods and the linear probing result, and from SMP 12 for Head2toe and SMP.
Linear probing different layers of FMs Results of Fig. and Fig. 5 are produced using the following procedure: (1) peraae-ppooling all tokens at the output of the selected block, (2) applying LayerNorm layer, and (3) learning a linear classifier on top of the pooled output. We train for 100 steps with batch size of 128 using the AdamW optimizer [4 We use weight decay and no learning rate scheduler. The learning rate is selected based on validation performance from the set [1.0, 0.1, 0.01, 0.001, 0.0001, 0.00001].
Training Adapter+ and Multi-Adapter For all experiments involving Adapter+ applied to foundation models, we use the implementation from hattps://githii:
plus We follow the recommended settings from the original paper [32], which include post-adapter placement, channel-wise scaling, and Houlsby initialization. We use a rank of 16 for our experiments as it led to the highest Global Average. We train the adapter with the same optimisation settings and schedule as the original paper and our method, discussed in Sec. 4 Following the authors' recommendations, we apply stochastic depth during training. For the frozen network, we use linearly increasing drop rates from O to 0.1 as function of network depth. For the adapters, we use fixed drop rate of 0.1.
For Multi- Aulpter, we first apply Adapter+ blocks to all backbones. We then pool their outputs using model-specific strategies (see Sec. [A.2 for details). Finally, we stack the resulting pooled features before applying a linear classifier.
A.4 Additional details about the generation of images that maximise FMs activations
Images in Figs. and were generated using the method from Ghiasi et al. 13 We used the imple- mentation available at httpp://gittiiei
com/ anonamouu00222 cml /ViTViS Neurons maximised to produce images are randomly selected for each layer and model. We used 400 optimization steps with total variation coefficient of 0.1 and strength λpp 0.0005. The learning rate was set 1.0 for CLIP and MAE, and 0.1 for other models.
A.5 Computational requirements computation
The computational cost comparison in Tab. 7 is based on batch size of 32. These measurements were obtained when adapting to the dSprites/location dataset from the VTAB-1k benchmark. This dataset contains 16 classes, which affects the number of parameters in the classifier head. However, the head represents only a small fraction of the compute load for full fine-tuning, Adapter+, and ComBo. Therefore, the relative differences between adaptation techniques would remain similar across all VTAB benchmark tasks, regardless of the number of classes.
The results for adapting one backbone are based on adapting the ViT-B/16 model trained on ImageNet- 21K. The results for adapting four backbones are based on adapting the DEN CLIP, DINOv2. SAM, and Lig ViT-B models.
A.6 Image and icons licenses
The image of space probe used in Fig. 3 is from work performed by ATG under contract for ESA and is shared under a CC BY-SA IGO 3.0 Ticense. Icons used in our figures come from Font Awesome and are shared under CC BY 4.0 license.
16

---

B
Additional results
B.1 Additional visualisations of images that maximise neuron activations
CLIP
DINON2
ImogeNet 21K
Randont
Layere
Figure 4: Additional visualisations extending Fig. 2 These images are optimised to maximise activations of different neurons across different layers using the technique from Ghiasi et al. [13.. We also include visualisations for a ViT with randomly initialised weights. All images generated from neurons that randomly sampled.
In Fig. we present additional images from the experiment of Fig. that maximise different neurons of different layers of We include an additional "Random" model with ViT-B/16 architecture and randomly initialised weights validate that the observed patterns come from learned representations.
17

---

Our images show that different layers of different models tend to have different representations, and the most relevant layer to gather features for downstream task is task duskddepeddent. This finding validates the value of probing architectures that can simultaneously access features from all layers of multiple models, such as ComBo.
B.2 Additional results from probing different layers of different foundation models on VTAB-Ik
In Fig. we also provide extended results of our linear probing experiment presented in Fig. la with all datasets of the VTAB- 1k benchmark.
Our observations hold, with the best-performing layer and model depending the dataset. For natural tasks such as Caltech101, CIFAR-100 100, DTD, Flowers 102, Pets, and Sun397, we can see that models trained with more semantic-based supervision such as CLIP, DINOv2, and ImageNet-21K tend to outperform MAE and SAM, usually with features from the penultimate or last layer. However, this pattern does hold for different tasks, with intermediate layers being sometimes better suited (e.g., SVHN) or MAE performing best (e.g., Clevr/distance).
B.3 Training adapters with more training samples
To evaluate how different adaptation methods scale with training data size, we conducted experiments using the full setting of the VTAB benchmark (which provides access to complete training datasets, as opposed to the few-shot setting) and present results in Fig.
For this experiment, we randomly sampled 1000, 5000, 10000, and 20000 images from each dataset's training split and trained different adapters on these subsets to examine how the performance of ComBo and other methods scales with increasing training data. For linear probing, we applied it to the cls token representation after the final transformer layer.
The results demonstrate that ComBo shows consistent performance improvements data becomes available across all tested datasets. Notably, on the SmalINORB/azimuu dataset, ComBo surpasses Adapter+ performance at higher sample counts. ComBo's ability benefit from more training data can be attributed to its transformer-based approach to processing different tokens, which naturally benefits from larger training datasets.
training
Interestingly, on the Sun397 dataset, linear probing initially outperforms ComBo, though this per- formance gap narrows with additional training data. This observation highlights the effectiveness of the regularization inherent in training only a linear classifier on top of the most relevant pre- trained features, which aligns with earlier findings from the linear probing experiments Sun397 (Fig. 5g). However, the diminishing gap suggests that ComBo's more flexible architecture increasingly leverages additional data to close this performance difference.
B.4 Training adapters with more classes
We also validate the behaviour of ComBo on a classification task with various number of classes. To do this, we use the dataset from the 2018 FGVCx Fungi Classification Challenge [4] which contains images of fungi with 1394 species classify.
We experiment with various number classes by randomly selecting different numbers of subsets of classes, and evaluating adaptation results the resulting dataset. We consider 5, 10, 20, 50, 100, 250, 500, 750, 1000, and 1394 classes. For each number of class, we randomly select classes to keep and use all available data for these classes. This is done over three seeds for each number of classes, leading to different subsets of classes and data for each run.
We present results in Fig. 7 where all methods display the same behaviour. As more classes are included, the task becomes more difficult and the performance of all methods drops. Linear probing in particular suffers more from the increase in number of classes.
18

---

1.00
1.00
Vol Accuracy
Accurasy
0.75
CLIP
MAE
0.50
0.50
DINDV2
SAM
ImageNet-21K
Rondom
0.25
0.25
0.00
0.00
Layer
(a) Legend for all plots
(b) Caltech101
(c) CIFAR
39
1.00
1,00
1.00
Accurcc
0.75
0.75
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Layer
(d) DTD
40
(e) Flowers 102
(f) Pets 42
1.00
1.00
1.00
48 Accuresy
Arcuragy
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
(g) Sum397
(h) SVHN
Patch Camelyon 45
1.00
1.00
1.00
Accurasy
0.75
0.75
0.50
0.50
0.50
3.255
0.25
0.25
0.00
0.00
(i) FuroSAT 46
(k) Resise45 47
()) Retinopathy
48
1.00
1.00
1.00
Jal Accurss
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
(m) Clevr/count
Clevr/distance
S
(o) DMLab 49
1.00
1.00
Vel. Acturady
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
Layer
(p) dSprites/location
50
(q) dSprites/orientation 50
(r) KITTI/distance 51
1.00
g8l Accursty
0.75
0.50
0.50
0.25
Sal
0.25
0.00
6
8
Layer
Layer
(s) SmallNORB/azimuth
52
(t) SmallNNORB/clvattion
52
Figure 5: Results linear probing as presented Fig. Ta for all VTAB-Ik 10 datasets. The plots present validation accuracies of linear probing on activations from different layers of VITs trained from CLIP (.) DINOv2 (6). ImageNet-21K 77. MAE (2). SAM (3, as well as randomly initialised VIT.
19

---

0.29
Acaplan
Velletton
Cordl
Canta
juddrg
0.36
012
Numbor traininn samples
Number of training samples
(a) Clevr/distance
(b) SmallNORB/azimulh
0.72
0.30
0.73
Acaptar-
proting
Adgparr
ComBe
0.40
Linar proling
Number training samples
Namber of training samples
(c) Sun397
(d) SVHN ARI
Figure 6: Results of using various adaptation methods with a ViT-B/16 trained on ImageNet-211 IK 7 for datasets from the VTAB benchmark [10] with increasing number of training samples. Results reported on the full validation set. The results are averaged over runs, with error bars showing standard deviation.
Adapter+
0.90
ComBo
Validation accuracy
probing
0.75
0.60
0.45
0.30
10
10a
10a
Number
Figure 7: Results of using various adaptation methods with V/T-B/16 trained on ImageNet-211s 21K 7 the FGVCx Fungi Classification Challenge dataset (54] with increasing number of classes classify. Results reported on the validation set. The sampling of subsets classes done over three seeds, our plots display average accuracy with error bars showing standard deviation.
B.5 Additional ablations ComBo
We also present additional ablation results in Tab. 10 validate design choices.
Keeping the els token In our ComBo experiment, we decide not the cls token register tokens to simplify the design choices when combining multiple models which might have different variations of these tokens. Moreover, relevant information needed for the task with own cls token, compensating for the absence of any original global token from the pre-trained backbone. Our ablation shows minimal improvement (74.1% 5% vs 73.8% for the Global Average) when keeping the original cls token.
expect transformer to be able learn attention to
When probing multiple models, different strategies to handle the cls token are possible, such as assigning them to new compressed tokens to feed our transformer, but we have not explored this thoroughly find the use of tokens associated to original image patches more general and simple.
Taking activations elsewhere Our experiments also rely on probing activations from the output
of each block. However, previous probing methods [ 12 use multiple locations within each block to extract features. We find that ComBo's performance competitive without the need to add
20

---

Table 10: Validation of ComBo design choices. We report accuracy (%) on the val set when adapting the ViT-B/16 model trained on ImageNet-2IK.
Natural
Specialised
Structured
Global Average
ComBo adapter
79.3
85.6
56.5
73.8
Probed features (default: no cls or register tokens, prohing the outpu of all blocks) Keeping the cls token
79.5
86.1
56.5
74.1
Probing after the second normalisation layer each block
78.4
86.1
53.5
72.7
Probing only the ouiput the first blocks
56.5
83.3
55.2
65.0
Prohing only the output of the last blocks
77.6
82.5
47.3
69.1
Probing the outputs of only blocks 2, 6, 8. 10, and
78.9
85.5
56.2
73.5
Differend ComBo frans settings (default: 6 blocks, embedding dimension of 128, 2 heads)
With a depth of block
79.3
84.5
53.9
72.6
With depth of blocks
79.2
86.0
56.1
73.8
depth
blocks
78.8
85.5
55.5
73.2
With an embedding dimension of 64 and head
78.
85.5
55.8
With an embedding dimension of 256 and heads
79.6
86.1
53.3
73.0
Different clossification head (defadt:: linear) With a Gaussian Naive Bayes classifier head
77.3
69.8
54.8
67.3
additional locations, and prefer to keep fewer locations within each block to maintain lower number of parameters of our affine projection.
In our ablation, we validate that the output of the block is good choice to collect activations by comparing using the output of the second normalisation layer in each block. This could be suitable location as features are already normalised, but the performance we obtain is slightly lower (72.7% vs 73.8% for the Global Average), indicating that the output of the block might have more informative activations.
Probing different combinations of layers We also validate the importance of probing all layers rather than only specific ones. We find that only probing the first 6 blocks leads to significant drop in performance (65.0% vs 73.8% for the Global Average), with the largest difference appearing in the Natural tasks (56.5% 79.3%), which usually rely more semantic features located in late layers. On the other hand, probing only the last 6 blocks also leads to a drop in performance (69.1 % vs 73.8% for the Global Average), but with the largest difference appearing in the Structured tasks (47.3% vs 56.5%). Using 6 layers better spread across the network (2, 4, 6, 8, 10, and 12) leads to a much stronger performance (73.5% 5% for the Global Average), but still does not reach the performance of probing all layers (73.8%).
Different transformer sizes The architecture choice of our transformer also validated in Tab. 10 with different embedding dimensions or number of blocks leading to drops in performance.
We expect that this is related to the small amount of training data in the VTAB-Ik 1kk setting and the complexity of the tasks. We believe that more complex tasks with more data could benefit from larger transformer architectures within ComBo.
Different classification head In FiT (55]. Shysheya et al. observed that using a Naive Bayes final layer classifier performed better than linear classifier few-shot setting. We also experiment by using the same head as the FiT-LDA variant, but find that the linear head still outperforms it in our case, with a significant difference in the specialised domain. However, we note that we did not experiment with using an episodic fine-tuning setting, which the authors note is crucial to obtain the best results with the Naive Bayes head.
B.6 Removing individual models when probing of multiple foundation models
In Tab. II we experiment with probing multiple FMs and removing some of them to see how the performance of ComBo is affected. As observed in Tab. 2 SAM performed much worse than other models considered when probed. This is reflected in the best performance of ComBo when removing SAM features (76.9% vs 76.2% for the Global Average when probing all models). On the other hand, DINOv2 is the best best-performing model overall when probed individually (Tab. 2 and removing its
21

---

Table 11: Performance when probing different combinations of FMs. We report accuracy (%) the val set. compare probing all four FMs considered with ComBo (DFN CLIP [3 DINO,2 66 SAM [ and SigLIP) [36], and removing each one from the set of probed models. Best results are in bold and second-best are underlined.
Natural
Specialised
Structured
Global Avurage
Caltech101
CIFAR-I00
Flowers 102
Sun397
SVHN
Average
Camdyon
EuroSAD
Rosise45
Retinopushy
Aweage
Clev-Coont
Clev--Dia
DMLab
dSpeLoc
dSpc Oni
KITTI-Dist
SNORB- -NORB-Azia Azim
&NORB Elev Elev
DTD
Pets
All I'M
97.2 76.0 73.3 100.0 93.2 46.2
87.0
81.8
87.2
97.1
89.8
71.2
86.4
89.7
64.5
49.0
92.8
48.0
72.2
18.7
47.7 60.3
76.2
Removing individiual backboner from probing
w/o DEN CLIP
97.8
74.3 74,2
99.8
94.0 48.3
85.8
82.1 88.0
97.0
90.5
69.8
86.3
91.8
63.0
48.8
91.2
47.3
70.7
22.2
45.3
60.0
76.1
w/o DINOv2
97.0
67.2
72.3
99.3 93.5
47.7
82.7
80.0
90.0
72.8
86.1
90.0
63.7
90.2
50.3
81.8
20.5
42.8
who SAM
97.2
76.1
75.5 99.8 93.0
49.8
86.5
82.6
87.7
97.8
90.2
72.8
87.1
90.7
63.2
52.2
91.
49.3
70.8
21.2
49.2
61.0
76.9
SigLIP
97.3
99.8
86.7
22.5
features from the probing leads to the highest drop in performance (75.6% vs 76.2% for the Global Average).
B.7 Detailed models' task-relevance estimation results
Tab. 12 presents the detailed importance scores generated by our method for each model on each VTAB-IK Ik task. These scores reflect each model's estimated task-relevance and are used to select which models to probe for each dataset. In Tab. and Tab. we demonstrate how these importance scores guide model selection, with Tab. 2 specifiicllly showing results when selecting the top two models (those with the highest importance scores) for each task.
The importance scores align with the observations from Sec. B.6 DINOv2 consistently achieves the highest or second-highese importance scores across most tasks, reflecting its strong performance on VTAB-IK. Conversely, SAM generally receives lower importance scores, corresponding to its weaker performance on these visual recognition tasks.
Notably, the optimal number of models varies substantially across tasks, shown in the bottom row of Tab. 12 Even within a single category, the best ensemble size ranges from single model to all four models, highlighting the importance of task-specific model selection.
Table Estimated importance scores each model on each VTAB-Ik task. Scores are computed using our proposed method and averaged over three training seeds. The bottom row shows the optimal number of models (selected by highest importance scores) that achieved the best performance each task. Highest scores are in bold and second highest underlined.
Natural
Speciulised
Structured
Globul Average
Caltehh011
CIFAR 100
Flower
SVIN
Awerge
Camelyon
Resise5 45
Retionpaty
Awerge
Clevr-Count
Clevr-Dist Dist
DMLab
dSpeLoc
dSpc-On
KETTI-DRS
SNORB Azim
-NOR--Elev
Average
DTD
Pot
DEN CLIP
2.91
3.61
2.70 3.21 5.92 3.97
1.29 3.37
2.98 0.40 6.50
3.86 3.44
2.50 6.74 6.43
4.56
5.55
2.80
2.50
4.37
4.43
3.75
DINO>2
4.51
6.20
4,09
4.72
6.27
5,08
3.38
4,89
3,52
2,07
6.51
3622
3
3,13
6.14
6.65
2.80
6.70
3.45
4,94
7.15
5.12
4.65
SAM
2.36
2.03
2.44
4.2
1.82
0.56
2.32
2.11 0.32
4.98
3.19
2.65
2.20
5.26
5.85
3
4.92
2.78
2.10
3.14
3.72
2.89
SPPLIP
2.93
2.55
2.48
3.05
6.52
4.22
0.38
3.16
2.00
15
6.02
2.79
2.74
2.66
6
6.26
5.01
3.11
4.67
197
1.30
3.56
3,57
3.15
Best # of models
1
2
2
2
2
2
2
2.125
2.04
B.8
Standard deviations for test result tables
Finally, we report the standard deviations for all test results presented in the main paper Tabs.
15
22

---

Table 13: Standard deviations (%) over the three seeds for produced results in Tab.
For the average columns,
we report the average of the standard deviations across all datasets in each category.
Natural
Specialised
Structured
Caltech101
38
CIFAR- 100
30
DTD 40
Flowers 102
Pets 44
Sun397 43
SVHN
44
Average
Cumelyon 4S
EumSAT
Resine45
47
Relinopathy
4%
Average
Cleve Count
Clavr- Divr-Dit
DMLab +0
dSpr-Loc 30
dSp--Ori
KITTI.
S1
NNORB-
SN ANOR--22 slcv
S
Average
Global Averagge
ComBo (ouns)
0.03 0.47 0.18 0.74 0.34
0.27
0.25 0.55 1.63
0.10
0.88
Table 14: Standard deviations (%) over the three seeds for produced results in Tab. For the average columns,
we report the average of the standard deviations across all datasets in cach category. Structured
Natural
Specialised
ANORD-Azim Azim
Global Average
Caltech101
CIFAR. 100
Flowen 102
Sun397
SVHN
Avezage
Camelyon
EurSAT
Resise45
Redinopatty
Average
Clew-Couat
Clevn-Dit
DMLab
dSpr Loc
dSpe-Ori
KITTI-Dis
SNORD-
SNORB-ElOC Eley
Avezuge
DED
Pets
Probing individunt foundation models wäth ComBo (ours)
foundiatt
DEN CLIP 53
0.60 0.42 0.45 0.13 0,27 0.19 0.80 0.41 0.96 0.14 0.05 0.32 0.37 0.51 0.38 0.24 1.08 0.25
0.75
1.19
0,15 0.57 0.45
DINOy2
6
0.13
0.08 0.18 0.02 0.35 0.40 0.68 0.26 0.44
0.18
0.58
0.51
0.43
0.52
0.80
0.97
5.11
0.44
0.43
0.63
1.37
1.29
0.66
SAM
3
0.74
0.92 0.64 0.57 0,38 0.18 0.63 0.58
0.30
0:42
0.76
0,35
0.46
0.18
0.35
0.76
2.29
0.21
0.99
1.58
0.44
0.85 0.63
Sigl
0.09 0.33 0.61
0.26
0.25
0.54
0.54
1.45
0.96
0.04
1,36
0.72
Pmobing all four foundation models
ComBo (ours) 0,23 0.48 0,22 0.02 0.33 0.44 0.28 0.29 0.81 0.16 0.61 0.41 0.50 0.90 0.25 0.62 1.83 0.61
1.90 2.77 0.90 1.22 0.67
Distilbigg four foundution models above and nuning the resudting model with Adapters A3
RADIOv2.5 [S] 0.89 0.35 0,28 0.28 0,19 0.31 0.17 0.35 0,21 0,32 0.44 0,31 0.32 0,30 0.37 0,51 0,16
0.65
0.65 0.32 1.70 0.58 0.42
Table 15: Standard deviations (%) over the three seeds for produced results in Tab. we report the average of the standard deviations across all datasets in each category.
3
For the average columns,
Natural
Specialised
Structured
Global Averaae
<NORB-Avin Azim
-NORB-Elev
Calteelo11
CIFAR-I 000
Flowers102
Sun397
SVHN
Average
Camelyon
EuroAAT
Resise45
Retiopaliy
Averge
Cleve-Count
Cleve- Dist
DMLab
dSpr Lac
dSpx-Oni
KITTI-Dist
Average
DTD
Pots
Individual adapted modells
DEN CLIP 533
0.11 0.11
0.11 0.55 0.05 0.30
0.23
0.50 0.10
0.10
0.11
0.20
0.41 0.53 0.29
0.20
0.30
0.76
0.42
1.59
0.56
0.33
DINO<2
0.23 0:21
0.07
0.02
0.06
0.06
0.35
0.14
0.41
0.15
0.28
0,35
0.29
1.26
0.34
0.21
0.57
0.59
0.63
0.97
3.00
0.95
0.46
SAM
0.76
1.51
0.27
0.45
0.35
0.29
0.29
0.92
0.72
1.26
0.42
SigT.IP IP
0.42 0.52 0.29 0.18 0.24 0.43
0.13
0.31
0.17
0.14
0.15
0.55
0.25
0.30
0.04
0818
0.38
0.25
0.40
0.22
0.53
0.28
0.28
Appmarhes combiniing models ahowe
Multi- Adapter+ 0.02 0.04 0.11 0.02 0.35 0.14 0.44
0.16
0.51
0.17
0.05
0.61
0.34
5.40
0.61
0.53
1.06
0.47
1.21
0.23
0.96
1.31 0.60
ComBo (ours) 0.64 0.20 0.30 0.10 0.40 0.35 0.07
0.30 0.10
0.06
0.28
1.77
0.55
0.87 0.25
1.32
0.14
0.71
0.57
1.88
0.07
0.73 0.53
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims in the abstract and introduction regarding the novel probing method (ComBo), its performance, scalability, and ability combine features from multiple foundation models accurately reflect the paper's core contributions. The claims are validated on the VTAB- 1k benchmark.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations part of the conclusion, in Sec. 5 This
section describes several limitations, including potential scaling issues with the linear affine layer.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Sec. exxplins our method, Figs. Tb and provide illustrations of the architec- ture, and Sec. details our settings and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
25

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: No code available at the time of submission, but the code will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wwtpcc/pccppcpcceeeooooooop httpc//gguiieeeeeeeuuubiiiiiiiii weneiisssnnoooiiiio
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Sec. dec.Adettais our settings and parpe-pprameters, and provides justifications for their choice.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Our results are all based on the average over three seeds, and the standard deviations are provided in the appendix to avoid overcrowding the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Tab. provides information the required number of FLOPs for training and inference, size of our models, and peak GPU memory usage. Moreover, we include details of compute and hardware used in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the code ethics and strived to adhere to its standards.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work involves pre- pre-trained models, adapters, and public datasets. is not tied to any partieular application with direct societal impact, beyond academic research.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles. surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
This paper does not provide any new dataset, and the proposed approach applics to existing released pre-trained models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our implementation, which will be made publicly available at a later date, utilises open-source libraries such as PyTorch and timm as external dependencies rather than including their source code or binary files directly. Consequently, our codebase itself does not bundle third-party licenses, and these foundational libraries are appropriately cited in the paper, see Sec. Other assets are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
28

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not provide any assets at submission time. will include the required details when we will release the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
29

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: While our approach could apply to LLMs, they are not core components of our method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy hbtpeee//earrrpsccc///ooorreerer for what should or should not be described.
30


Reviews:
[{"review_id": "ZzfxzehsuF", "reviewer": "Reviewer_YFvs", "summary": "Foundation models learn diverse features based on their training data and objectives. For downstream tasks, different models or layers within these models may provide useful representations. Existing strategies such as fine-tuning and distillation are unable to exploit these complementary strengths or are too expensive. In this paper, the authors present ComBo, a lightweight, scalable adapter based on the probing framework that (i) extracts feature representations from selected layers of multiple models, (ii) normalizes and aligns them spatially via interpolation, (iii) projects them into low-dimensional embeddings using a shared linear projection, and (iv) processes the compressed embeddings with a small transformer. The method operates on frozen models and does away with the expensive backprop into FMs. It also avoids dataset-specific tuning. The approach was tested on 19 vision tasks across natural, specialized, and structured domains. ComBo beats previous probing methods (Head2Toe, SMP), and matches or surpasses full fine-tuning in many structured tasks. It also beats distilled models like RADIOv2.5.", "questions": "The paper assumes that spatial features from different foundation models can be aligned via interpolation to a shared spatial resolution. This may work well for ViT-style models, but how generalizable or robust this assumption is, particularly for architectures with fundamentally different tokenization strategies.", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- doesn't require backprob through foundation models making the approach computationally efficient\n- handles activations from multiple models and layers without task-specific tuning unless previous probing methods\n- outperforms existing probing approaches and matches/surpasses full fine-tuning in structured tasks \n- uses fixed hyperparameters making it robust and easier to apply\n- works with both pretrained or tned models and integrates new models without any retraining\n- lower parameter overhead compared to multi-backbone fine-tuning and distillation\n\nWeaknesses:\n\n- The paper assumes that spatial features from different foundation models can be aligned via interpolation to a shared spatial resolution. This may work well for ViT-style models, but how generalizable or robust this assumption is, particularly for architectures with fundamentally different tokenization strategies.\n- novelty of the paper is more system integration than in algorithmic design", "was_revised": true, "final_justification": "Thank you for the detailed response. I believe the revisions suggested will strengthen the final version.", "rebuttal": "We thank the reviewer for their positive evaluation and constructive feedback. We appreciate your recognition of ComBo's key strengths: computational efficiency, robustness without task-specific tuning, and ability to integrate multiple models without retraining each separately.\n\n**Overview**: We address your concerns about generalisability to different architectures and clarify the novelty and practical contributions of our approach.\n\n## Replies to your concerns and questions\n### 1. Generalisability to different architectures and tokenisation strategies\n\nWe thank the reviewer for pointing out that this was not discussed. We agree that this aspect and the requirements to support new architectures should be clarified in the paper, and we will include a discussion about it.\n\n**ViT-based approaches are standard**\n\nWhilst we agree that our experiments consider limited architectures, we believe our findings remain generally useful.\nOur current evaluation focuses on ViT-based architectures as they represent the dominant paradigm in modern foundation models. As such, we expect the application of ComBo to remain straightforward for most recent, and potentially future, foundation models.\n\n**The general methodology of ComBo could be adapted to other architectures**\n\nMoreover, the core principle of ComBo can be extended beyond this specific architecture family: The core methodology remains (1) extract features from multiple layers of multiple models, (2) align them spatially in an architecture-dependent manner, (3) compress them via a learned projection, (4) process the compressed features with a transformer. The spatial alignment step can be customised for different architectures whilst maintaining the overall framework.\n\n**Tokenisation discrepancies and Convolutional Neural Networks (CNNs) could be managed through interpolation**\n\nWhilst tokenisation discrepancies might appear within ViT-based models, we expect some spatial alignment to be possible. In fact, this is the case in our experiments, where DINOv2 uses 14×14 patch sizes compared to 16×16 for the other models we consider. This required us to combine the models by spatially interpolating their resolution, which produced a simple and successful combination strategy.\n\nWe believe CNN feature maps can likely be dealt with similarly, by the same interpolation approach on different layers, treating spatial locations as tokens.\n\n### 2. Addressing novelty\n\n**ComBo demonstrates novel insights about multi-model feature integration and provides practical capabilities not previously shown.**\n\nWe agree that ComBo represents system integration rather than purely algorithmic design. However, we believe this system provides sufficient novelty through the insights it generates and its practical contributions:\n\n### Novel insights from our work\n- Transformers can effectively process multi-model features: We demonstrate that transformers can successfully integrate features from multiple layers across different foundation models, even in a low-data setting such as VTAB-1k.\n- Different features from different foundation models can be relevant, depending on the task: Our results highlight the importance of selecting the correct foundation model and layer to extract features for a given task.\n\n### Practical novelty\nWhilst individual components of ComBo exist, the design of ComBo enables capabilities not previously demonstrated:\n- Probing multiple large foundation models together without backpropagation through any of them\n- Universal hyperparameters that work across diverse vision tasks\n- Memory-efficient approach that makes large model combinations accessible to practitioners with limited resources\n\n### Value of simple systems\nWe argue that demonstrating how simple integration, albeit the result of significant design exploration, can match complex alternatives represents valuable novelty for the field. Our work provides practical insights about what components are necessary for effective multi-model combination, which we believe has clear value for the community.\n\n## Summary of updates we will make to the final paper\nFollowing your feedback, we will include:\n- Extended architectural generalisation discussion covering the application of ComBo to other backbone architectures and various tokenisation approaches\n- Clarifications on the novelty of our work, discussing its design choices and strengths in its simple architecture"}, {"review_id": "abIaI8NXOq", "reviewer": "Reviewer_UTfc", "summary": "Authors propose ComBo, a method to combine features from multiple foundation models. Specifically a two-stage approach is adopted: features from multiple layers and models are concatenated and projected, which is then passed into a lightweight transformer for further processing. Experiment results show that ComBo is better than other baselines like Head2Toe and SMP.", "questions": "see weaknesses", "limitations": "n/a", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. Overall the paper is easy to follow.\n2. The proposed method ComBo does not require extensive hyperparameter tuning.\n\nWeakness:\n1. The method proposed is not practical. When combining features from multiple models, it requires forwarding multiple models and obtaining their features. In Table 6, if one puts the distillation baseline in it, it should be far more efficient than ComBo since it does not require multiple models’ inference. \n2. When compared with LoRA, the proposed method has worse performance while being less efficient during inference time. Admittedly, ComBo does not require back propagation through the foundation model. However I doubt if the real world has cases where trading inference’s speed for training efficiency is needed. I encourage authors to justify the real-world use case here.\n3. In spite of the limited practicality, the ideas raised in future work sections are potentially interesting and it would add in this paper’s value if some of the analysis could be done: e.g. analyzing the parameters to help with model selection/pruning. This may bring insight on understanding different FMs, adding to the value of this work.", "was_revised": true, "final_justification": "I notice that in the new result provided in rebuttal, Dinov2 is selected 18 out of 19 cases. And for all but 1 single-backbone cases, DINOv2 is the choice. It seems that the conclusion from the current set of experiments boils down to \"just use DINOv2\", as also shown by the high accuracy by Combo+DINOv2 (77.9%). This result seems to defeat the paper's story on combining multiple backbones, if one single backbone can achieve 77.9%, there is less reason to add in other backbones for ~0.4% accuracy improvement. Therefore, the current experiment setting where one vision backbone is dominating may not be enough to show ComBo's motivation on leveraging strength of multiple backbones.\n\n=======edit========\n\nAuthor's additional results show the informativeness of ComBo as a criteria to select and prune multiple pretrained vision backbones. I raise my score to 4.", "rebuttal": "We thank the reviewer for their feedback and for acknowledging that the paper is easy to follow and that the method does not require extensive hyperparameter tuning. We appreciate your constructive concerns about practical applications, which have motivated us to develop novel solutions that directly address inference efficiency. \n\n**Overview**: We address your main concerns about practicality (Weaknesses 1-2) by clarifying the training vs. inference efficiency trade-offs, providing new model pruning results (Weakness 3) that reduce inference costs, and demonstrating ComBo's advantages in memory-constrained scenarios.\n\n## Replies to your concerns and questions\n### 1. (Weakness 1) Inference cost against a distillation baseline\n\n**ComBo addresses cases where distillation is impractical due to resource constraints, whilst offering automatic model selection to reduce inference costs.**\n\nWhile we agree that a distilled model would be more efficient at inference for combining multiple models than ComBo, which still requires forward passes from each backbone, we believe there is a strong need for an approach such as ComBo.\n\nThe resources required to distil foundation models are extremely significant, with SAM-CLIP [4] requiring 64 A100 GPUs, and RADIOv2.5 [5] also using 64 GPUs to train over 614M examples. These resources are not accessible to many practitioners, who would need to rely on distilled models being released. If a new foundation model relevant to their tasks becomes available, it would be costly or impossible for many practitioners to include it in a distillation process. In contrast, ComBo can be trained in ~10 minutes to probe 4 models on a VTAB-1k task, using a standard consumer GPU.\n\nMoreover, distillation is not a straightforward process, with potential issues such as \"mode switches\" appearing based on resolution [5].\n\nFinally, our additional pruning approach (addressing your Weakness #3, detailed below) allows us to automatically select a subset of backbones for a given task, thereby reducing the inference cost significantly.\n\n### 2. (Weakness 2) Comparison against LoRA\n\n**ComBo enables multi-model combination and requires 4× less GPU memory than PEFT methods during training.**\n\nWe agree that LoRA and other parameter-efficient fine-tuning (PEFT) techniques are often very practical, offering strong performance with minor inference cost overheads.\n\nOne key advantage of our probing approach is that it does not require backpropagating through the backbone. If this is manageable, we agree that PEFT methods might be a better choice. However, our objective is to access features from multiple, potentially large, backbones. LoRA and other PEFT techniques are not designed for this setting, and the need to backpropagate through the backbones strongly limits which backbones can be combined. In contrast, ComBo is specifically designed to make best use of multiple models efficiently.\n\nEven in the single backbone case, the GPU memory requirements can make ComBo a valuable choice for practitioners with limited computational resources. We have measured GPU memory requirements during training (which we will include in the paper) and found that ComBo requires 4× less memory than Adapter+.\n\nAs a practical example, in our setting with an NVIDIA RTX 3090 Ti (24GB) GPU, we are able to probe the Giant version of DINOv2 (ViT-g/14) with ComBo, but are unable to adapt it with Adapter+ due to memory limitations without needing additional tricks (e.g., accumulating gradients). \n\nFinally, ComBo's inference cost is slightly higher than some PEFT methods, but we find the difference is very minor in practice (<3% additional FLOPs relative to the original backbone).\n\n\n### 3. (Weakness 3) Novel Model Pruning Method (Developed in Response to Multiple Reviewers)\n\n**Our new automatic backbone selection reduces the average number of backbones from 4 to 1.42 whilst maintaining performance (78.1% vs 78.2%).**\n\nIn direct response to concerns about inference efficiency raised by you and other reviewers, we have developed and evaluated a novel automatic model pruning method that significantly reduces inference costs while maintaining performance.\n\nOur two-stage method is inspired by Head2Toe [11], but instead of selecting features to retain, it selects entire backbones.\n\n**Stage 1 - Learning Backbone Importance**: We first train ComBo with a novel regularisation term applied to projection weights Λ. For each backbone model, we compute the L2 norm of all its projection weights across all layers to obtain an importance score. We sum importance scores from all backbones and add the result as a regularisation term to minimise in our loss.\n\n**Stage 2 - Pruning and Retraining**: After the first stage of training, we measure the resulting importance scores for each backbone. We prune all backbones whose norms fall below 90% of the highest-scoring backbone, then retrain ComBo from scratch without regularisation using only the selected backbones.\n\n### Experimental Results\nWith our new pruning mechanism:\n- Performance is maintained: ComBo with pruning achieves 78.1% vs 78.2% for full ComBo (only 0.1% drop)\n- Efficiency is improved: On average over the 19 datasets, we select 1.42 backbones per dataset vs 4.0 for full ComBo\n\nWe provide details of our selected backbone combinations (models with an \"X\" are selected) for each dataset with our new approach, and the corresponding test accuracies: \n\n| Dataset | Category | ComBo pruned accuracy | DFN CLIP | DINOv2 | SAM | SigLIP | Models Used |\n|---------|----------|--------------|----------|---------|-----|---------|-------------|\n| CIFAR100 | Natural | 79.4% | - | X | - | - | 1 |\n| Caltech101 | Natural | 94.7% | - | X | - | - | 1 |\n| DTD | Natural | 78.4% | - | X | - | - | 1 |\n| Flowers102 | Natural | 99.7% | - | X | - | - | 1 |\n| Pets | Natural | 93.1% | X | X | - | X | 3 |\n| SVHN | Natural | 85.9% | - | X | - | - | 1 |\n| Sun397 | Natural | 49.3% | - | X | - | - | 1 |\n| Camelyon | Specialized | 86.0% | - | X | - | - | 1 |\n| EuroSAT | Specialized | 96.2% | - | X | - | - | 1 |\n| Resisc45 | Specialized | 91.1% | X | X | - | X | 3 |\n| Retinopathy | Specialized | 73.7% | X | X | - | - | 2 |\n| Clevr-Count | Structured | 92.6% | - | X | - | - | 1 |\n| Clevr-Distance | Structured | 61.9% | X | X | - | X | 3 |\n| DMLab | Structured | 54.3% | X | X | - | - | 2 |\n| KITTI-Distance | Structured | 82.0% | - | X | - | - | 1 |\n| dSprites-Location | Structured | 81.9% | X | - | - | - | 1 |\n| dSprites-Orientation | Structured | 53.3% | - | X | - | - | 1 |\n| SmallNORB-Azimuth | Structured | 36.4% | - | X | - | - | 1 |\n| SmallNORB-Elevation | Structured | 55.6% | - | X | - | - | 1 |\n\nWe will also update Table 2 with these new results, with a compact summary shown here:\n\n| Method | Natural | Specialized | Structured | Global Average | Avg. Models/Dataset |\n|--------|---------|-------------|------------|----------------|---------------------|\n| ComBo (DFN CLIP) | 79.3% | 85.8% | 61.6% | 75.6% | 1 |\n| ComBo (DINOv2) | 82.6% | 85.9% | 65.0% | 77.9% | 1 |\n| ComBo (SAM) | 55.2% | 79.4% | 56.9% | 63.8% | 1 |\n| ComBo (SigLIP) | 80.5% | 85.4% | 62.6% | 76.2% | 1 |\n| RADIOv2.5 | 83.8% | 86.6% | 62.1% | 77.5% | 1 |\n| ComBo (all 4 backbones) | 83.3% | 86.5% | 64.8% | 78.2% | 4 |\n| ComBo (pruned backbones) | 82.9% | 86.7% | 64.8% | 78.1% | 1.42 |\n\n\n## Summary of updates we will make to the final paper\nFollowing your feedback, we will include:\n- Expanded computational details detailing training costs of distillation and PEFT techniques compared to ComBo, including detailed memory usage comparisons\n- Enhanced practical use case discussion highlighting scenarios where the computational trade-offs offered by ComBo are valuable\n- Novel automatic model selection methodology and results (developed in response to multiple reviewers' concerns) showing how our new pruning approach reduces inference costs whilst maintaining performance"}, {"review_id": "hbLsBEqee7", "reviewer": "Reviewer_ARhL", "summary": "The authors present a method to combine the features of multiple pretrained backbones to improve classification accuracy on the VTAB dataset. In particular, the method is a probing based adapter, called ComBo, that utilizes a transformer to compress image features generated from patches at multiple layers from multiple pretrained backbones to a lower dimensional space. ComBo surpasses the performance of competitive probing methods.", "questions": "- Have you considered using something other than a linear classifier to predict class probabilities? It has been shown that Linear Discriminant Analysis (LDA) outperforms a linear classifier on VTAB-1K [1]:\n\n[1] Shysheya, A., Bronskill, J., Patacchiola, M., Nowozin, S., & Turner, R. E. (2022). Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. ICLR 2023.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written and easy to read.\n- The experiments are thorough and demonstrate the merits of the approach.\n- The idea of using a transformer to amalgamate the features from multiple pretrained backbones both spatially and layer-wise is novel.\n\n**Weaknesses**\n- The biggest weakness in the approach is the increase in inference time resources (memory, computation) that is proportional to the number of pretrained models $K$. The relatively small accuracy gains (78.2 with ComBo using 4 models from Table 2 versus 77.6 using Adapter+ with a single model) would be difficult to justify in practice given the 4 times increase in inference cost.\n- As a new probing method when utilizing just one pretrained backbone, ComBo falls short of PEFT methods (see Table 1).\n- The paper would be improved by expanding the evaluation to include few-shot datasets such as Meta-Dataset (Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.)\n- A couple of early methods that present various approaches for combining the features of multiple backbones were missed in the references and related work:\n\n    - Chowdhury, A., Jiang, M., Chaudhuri, S., & Jermaine, C. (2021). Few-shot image classification: Just use a library of pre-trained feature extractors and a simple classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 9445-9454).\n\n    - Liu, L., Hamilton, W., Long, G., Jiang, J., & Larochelle, H. (2020). A universal representation transformer layer for few-shot image classification. arXiv preprint arXiv:2006.11702.", "was_revised": true, "final_justification": "I will keep my score at 4. The new work on pruning backbones is a nice improvement, but may still be impractical versus fine-tuning with PEFT due to the increase in inference cost.", "rebuttal": "We thank the reviewer for their positive evaluation and thorough assessment. We appreciate your recognition that the paper is well-written, experiments are thorough, and the transformer-based approach for amalgamating features is novel.\n\n**Overview**: We address your concerns about inference cost vs. accuracy trade-offs, performance relative to PEFT methods, and provide additional experimental analysis on alternative classifiers, while acknowledging suggestions for future work.\n\n## Replies to your concerns and questions\n### 1. Inference cost being proportional to the number of models\n\n**Our new automatic backbone pruning reduces the average number of backbones from 4 to 1.42 whilst maintaining performance (78.1% vs 78.2%).**\n\nIn direct response to concerns about inference costs of using multiple backbones raised by multiple reviewers, we have developed and evaluated a novel automatic backbone pruning method that directly addresses the $K$× inference cost issue.\n\nOur method reduces average backbones per dataset from 4.0 to 1.42 while maintaining competitive performance (78.1% vs 78.2% for full ComBo).\n\n### Methodology: Two-stage training approach\nOur method is inspired by Head2Toe [11], but instead of selecting features to retain, it selects entire backbones.\n\n**Stage 1 - Learning backbone importance**: We first train ComBo with a novel regularisation term applied to projection weights Λ. For each backbone model, we compute the L2 norm of all its projection weights across all layers to obtain an importance score. We sum importance scores from all backbones and add the result as a regularisation term to minimise in our loss.\n\n**Stage 2 - Pruning and retraining**: After the first stage of training, we measure the resulting importance scores for each backbone. We prune all backbones whose norms fall below 90% of the highest-scoring backbone, then retrain ComBo from scratch without regularisation using only the selected backbones.\n\n### Experimental results\nWith our novel pruning mechanism:\n- Performance is maintained: ComBo with pruning achieves 78.1% vs 78.2% for full ComBo (only 0.1% drop)\n- Efficiency is improved: On average over the 19 datasets, we select 1.42 backbones per dataset vs 4.0 for full ComBo\n\nWe provide details of our selected backbone combinations (models with an \"X\" are selected) for each dataset with our new approach, and the corresponding test accuracies:\n\n| Dataset | Category | ComBo pruned accuracy | DFN CLIP | DINOv2 | SAM | SigLIP | Models Used |\n|---------|----------|--------------|----------|---------|-----|---------|-------------|\n| CIFAR100 | Natural | 79.4% | - | X | - | - | 1 |\n| Caltech101 | Natural | 94.7% | - | X | - | - | 1 |\n| DTD | Natural | 78.4% | - | X | - | - | 1 |\n| Flowers102 | Natural | 99.7% | - | X | - | - | 1 |\n| Pets | Natural | 93.1% | X | X | - | X | 3 |\n| SVHN | Natural | 85.9% | - | X | - | - | 1 |\n| Sun397 | Natural | 49.3% | - | X | - | - | 1 |\n| Camelyon | Specialized | 86.0% | - | X | - | - | 1 |\n| EuroSAT | Specialized | 96.2% | - | X | - | - | 1 |\n| Resisc45 | Specialized | 91.1% | X | X | - | X | 3 |\n| Retinopathy | Specialized | 73.7% | X | X | - | - | 2 |\n| Clevr-Count | Structured | 92.6% | - | X | - | - | 1 |\n| Clevr-Distance | Structured | 61.9% | X | X | - | X | 3 |\n| DMLab | Structured | 54.3% | X | X | - | - | 2 |\n| KITTI-Distance | Structured | 82.0% | - | X | - | - | 1 |\n| dSprites-Location | Structured | 81.9% | X | - | - | - | 1 |\n| dSprites-Orientation | Structured | 53.3% | - | X | - | - | 1 |\n| SmallNORB-Azimuth | Structured | 36.4% | - | X | - | - | 1 |\n| SmallNORB-Elevation | Structured | 55.6% | - | X | - | - | 1 |\n\nWe will also update Table 2 with these new results, with a compact summary shown here:\n\n| Method | Natural | Specialized | Structured | Global Average | Avg. Models/Dataset |\n|--------|---------|-------------|------------|----------------|---------------------|\n| ComBo (DFN CLIP) | 79.3% | 85.8% | 61.6% | 75.6% | 1 |\n| ComBo (DINOv2) | 82.6% | 85.9% | 65.0% | 77.9% | 1 |\n| ComBo (SAM) | 55.2% | 79.4% | 56.9% | 63.8% | 1 |\n| ComBo (SigLIP) | 80.5% | 85.4% | 62.6% | 76.2% | 1 |\n| RADIOv2.5 | 83.8% | 86.6% | 62.1% | 77.5% | 1 |\n| ComBo (all 4 backbones) | 83.3% | 86.5% | 64.8% | 78.2% | 4 |\n| ComBo (pruned backbones) | 82.9% | 86.7% | 64.8% | 78.1% | 1.42 |\n\n\n### 2. Performance vs. PEFT methods\n\n**ComBo enables multi-model combination and requires 4× less GPU memory than PEFT methods during training.**\n\nWe agree that parameter-efficient fine-tuning (PEFT) techniques are strong baselines that ComBo might not usually outperform in the context of adapting a single backbone. However, we designed our approach to efficiently combine features from multiple models, which PEFT methods currently do not handle naturally.\n\nMoreover, not requiring backpropagation through the backbone offers other advantages, even with a single backbone. We have measured GPU memory requirements during training (which we will include in the paper) and found that ComBo requires 4× less memory than Adapter+. This property can make ComBo a valuable choice for practitioners with limited computational resources. \n\nAs a practical example, in our setting with an NVIDIA RTX 3090 Ti (24GB) GPU, we are able to probe the Giant version of DINOv2 (ViT-g/14) with ComBo, but are unable to adapt it with Adapter+ due to memory limitations without needing additional tricks (e.g., accumulating gradients). \n\n### 3. Future work and missing references\n**We will include missing references and discuss few-shot learning capabilities studies as future work.**\n\n- Few-shot setting evaluation: We thank the reviewer for this suggestion and agree that validating ComBo's few-shot learning capabilities on a benchmark such as Meta-Dataset would be an interesting next step. Unfortunately, it was outside the scope of our current project and submission, but we will consider it in future work.\n- Missing references: We thank the reviewer for pointing out these very relevant methods we had missed in our related work. We will include them in the revised manuscript.\n\n\n### 4. Alternative classification head\n\n**Linear head consistently outperforms Naive Bayes classifier across all task categories in our setting.**\n\nThe previous observation from the FiT paper on an improved classification head on VTAB-1k is interesting, and we have now performed an additional ablation study to see its effect in our setting. We replaced the linear head in ComBo with a Gaussian Naive Bayes classifier following the approach used in the FiT-LDA method.\n\nWe obtained the following results (same settings as Table 5: averaged over three seeds, on the validation set of VTAB-1k), which we will include in our paper:\n| Classifier Type | Natural Tasks | Specialized Tasks | Structured Tasks | Average |\n|-----------------|---------------|-------------------|------------------|---------|\n| ComBo with linear head | 79.3% | 85.6% | 56.5% | 73.8% |\n| ComBo with FiT-LDA head | 77.3% | 69.8% | 54.8% | 67.3% |\n\nWe find that the linear head consistently outperforms the Naive Bayes classifier across all task categories. This differs from FiT's reported improvements, potentially due to our use of non-episodic training, which is typical in transfer learning, whereas the authors of FiT noted that episodic fine-tuning was crucial to obtain optimal Naive Bayes performance.\n\n## Summary of updates we will make to the final paper\n\nFollowing your feedback, we will include:\n- Novel automatic model selection methodology and results (developed in response to multiple reviewers' concerns) showing how our new pruning approach reduces inference costs whilst maintaining performance  \n- GPU memory requirement details for training ComBo and PEFT methods and discussion highlighting accessibility advantages for practitioners with limited computational resources during training\n- Comparison to the Gaussian Naive Bayes classifier from FiT-LDA in our ablation study\n- Additional citations with proper contextualisation of related feature combination approaches\n- Future work discussion of studying ComBo's potential in few-shot learning settings"}, {"review_id": "gCTNq263uB", "reviewer": "Reviewer_VFAP", "summary": "The manuscript builds on an empirical observation that contemporary foundation models learn different representations (Fig. 2). Consequently, the manuscript proposes a probing method which combines features from multiple frozen foundation models into a single representation. The method begins by taking activations from all layers across all models.  These activations are first processed to equalise the token count for all models. This is done by reshaping tokens from each layer into a 2D grid and applying bilinear interpolation. The tokens are then standardised and concatenated with their counterparts from other models. The resulting matrix S has a dimensionality of (token count * layer count) x (sum of all model dimensions). The matrix S is then affinely projected with learnable parameters (W, b) into a lower-dimensional space. The resulting matrix is then concatenated with a cls token and fed to a shallow transformer F that generates the final representation. Learnable parameters are the task head, transformer F, cls token, and the projection (W, b). Interestingly, the resulting procedure keeps foundation models frozen, which greatly reduces the computational requirements. Experimental evaluation on VTAB-1k indicates that the proposed method on average beats relevant previous works.", "questions": "Q1. It seems that the manuscript misses the most important experiment. Please answer W1 in detail.\n\nQ2. What if one or more foundation models are irrelevant for some downstream tasks. Could ComBo detect this and exclude such a foundation model during the inference? The benefit would be in inference speedup.\n\nQ3. The VTAB benchmark has a relatively low class count across the tasks (max <400). How does ComBo behave in setups with large class count (over 1k classes) and the VTAB full setting (Table 7)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strenghts: \n\nS1. The manuscript addresses an interesting problem.\n\nS2. The proposed method is intuitive and easy to follow. \n\nS3. The presented ablations are insightful.\n\nS4. The proposed method performs well on different tasks with a constant hyperparameter setting.\n\nWeaknesses:\n\nW1. Table 1 shows that ComBo improves over SMP and Head2Toe when probing a single foundation model. However, a similar comparison is missing in Table 2 (merging features from multiple foundation models). It is hard to conclude on the effectiveness of the proposed method w.r.t related work without these crucial results.\n\nW2. The method focuses solely on transformer architectures and does not comment on the integration of other architectures (e.g. convolutional models). \n\n\nTypos:\n- L144. Linear projection -> you mean affine projection (linear projection W + translation b)", "was_revised": true, "final_justification": "The manuscript proposes a probing method which combines features from multiple frozen foundation models into a single representation, which is interesting and potentially useful in practice. The author's response resolved my concerns, so I increased the score. Also, I thank the authors for a clear and elaborate response supported by sufficient experiments.", "rebuttal": "We thank the reviewer for their positive feedback on our intuitive method design, insightful ablations, and performance.\n\n**Overview**: We address your concerns about missing Head2Toe/SMP comparisons for multiple foundation models (Q1), clarify our approach to non-transformer architectures (W2), present new automatic model exclusion results (Q2), and provide scalability experiments (Q3).\n\n## Replies to your concerns and questions\n### 1. (Q1, W1) Lack of SMP and Head2Toe results in the setting with multiple foundation models\n\n**Evaluating SMP and Head2Toe with multiple backbones would be computationally prohibitive, while ComBo demonstrates superior performance with a single model.**\n\nWe agree that ideally, we would compare against SMP and Head2Toe when probing multiple foundation models; however, we decided against this for practical reasons.\n\nFirst, both Head2Toe and SMP rely on extensive hyperparameter sweeps for each dataset, including learning rates, regularisation strengths, training duration, and target pooling size. Extending this to multiple foundation models would require expensive hyperparameter sweeps for SMP and Head2Toe on each dataset-backbone combination, which we cannot carry out due to computational limitations. Furthermore, SMP's implementation is not public, challenging reproducibility.\n\nIn contrast, ComBo performs favourably against these methods with a single setting across 19 VTAB-1k tasks, without requiring hyperparameter sweeps. This makes ComBo more practical than existing approaches, and Table 1 clearly highlights ComBo's benefits over SMP and Head2Toe even in the single-model setting.\n\nGiven ComBo's strong performance in Table 1, its practical advantages, and the substantial challenges of evaluating SMP and Head2Toe with multiple backbones, we believe the current validation is sufficient and have preferred using our resources to focus on additional experiments, ablations, and studies.\n\n### 2. (W2) Integration of non-transformer architectures\nWe thank the reviewer for highlighting this missing discussion. We agree this should be discussed in the paper and will update it. We provide architectural considerations here.\n\n**ViT-based approaches are standard**\n\nWhilst our evaluation focuses on limited architectures, we believe our findings remain useful. Our evaluation focuses on ViT-based architectures as they represent the dominant paradigm in modern foundation models. We expect ComBo to remain straightforward for most recent and future foundation models.\n\n**The general methodology of ComBo could be adapted to other architectures**\n\nThe core principle of ComBo can be extended beyond this specific architecture family: The core methodology remains (1) extract features from multiple layers of multiple models, (2) align them spatially in an architecture-dependent manner, (3) compress them via a learned projection, (4) process the compressed features with a transformer. The spatial alignment step can be customised for different architectures while maintaining the overall framework.\n\n**Interpolation of feature maps is already used for different tokenisations and could apply to CNNs**\n\nWe already handle different patch sizes, with DINOv2 using 14×14 patches compared to 16×16 for other models. We manage this through bilinear interpolation. This approach can likely extend to CNN feature maps by treating spatial locations as tokens.\n\n\n### 3. (Q2) Novel automatic model exclusion method \n\n**Our backbone selection method reduces the average number of backbones from 4 to 1.42 whilst maintaining performance.**\n\nWe thank the reviewer for this suggestion on ComBo detecting and excluding irrelevant models, also mentioned by other reviewers. Following this feedback, we developed and tested a new automatic model selection approach to address these concerns.\n\nOur two-stage method is inspired by Head2Toe [11], but instead of selecting features to retain, it selects backbones.\n\n**Stage 1 - Learning backbone importance**: We train ComBo with a novel regularisation term applied to projection weights Λ. For each backbone, we compute the L2 norm of all its projection weights across all layers to get an importance score. We sum importance scores from all backbones and add this as a regularisation term to minimise in our loss.\n\n**Stage 2 - Pruning and retraining**: After first-stage training, we measure resulting importance scores for each backbone. We prune backbones whose norms fall below 90% of the highest-scoring backbone, then retrain ComBo from scratch without regularisation using only selected backbones.\n\n### Experimental results\nWith our novel pruning mechanism:\n- **Performance is maintained**: ComBo with pruning achieves 78.1% vs 78.2% for full ComBo (only 0.1% drop)\n- **Efficiency is improved**: On average over the 19 datasets, we select 1.42 backbones per dataset vs 4.0 for full ComBo\n\n**Detailed model selection results**: We provide details of our selected backbone combinations (models with an \"X\" are selected) for each dataset with our new approach, and the corresponding test accuracies:\n\n| Dataset | Category | ComBo pruned accuracy | DFN CLIP | DINOv2 | SAM | SigLIP | Models Used |\n|---------|----------|--------------|----------|---------|-----|---------|-------------|\n| CIFAR100 | Natural | 79.4% | - | X | - | - | 1 |\n| Caltech101 | Natural | 94.7% | - | X | - | - | 1 |\n| DTD | Natural | 78.4% | - | X | - | - | 1 |\n| Flowers102 | Natural | 99.7% | - | X | - | - | 1 |\n| Pets | Natural | 93.1% | X | X | - | X | 3 |\n| SVHN | Natural | 85.9% | - | X | - | - | 1 |\n| Sun397 | Natural | 49.3% | - | X | - | - | 1 |\n| Camelyon | Specialized | 86.0% | - | X | - | - | 1 |\n| EuroSAT | Specialized | 96.2% | - | X | - | - | 1 |\n| Resisc45 | Specialized | 91.1% | X | X | - | X | 3 |\n| Retinopathy | Specialized | 73.7% | X | X | - | - | 2 |\n| Clevr-Count | Structured | 92.6% | - | X | - | - | 1 |\n| Clevr-Distance | Structured | 61.9% | X | X | - | X | 3 |\n| DMLab | Structured | 54.3% | X | X | - | - | 2 |\n| KITTI-Distance | Structured | 82.0% | - | X | - | - | 1 |\n| dSprites-Location | Structured | 81.9% | X | - | - | - | 1 |\n| dSprites-Orientation | Structured | 53.3% | - | X | - | - | 1 |\n| SmallNORB-Azimuth | Structured | 36.4% | - | X | - | - | 1 |\n| SmallNORB-Elevation | Structured | 55.6% | - | X | - | - | 1 |\n\nWe will also update Table 2 with these new results, with a compact summary shown here:\n\n| Method | Natural | Specialized | Structured | Global Average | Avg. Models/Dataset |\n|--------|---------|-------------|------------|----------------|---------------------|\n| ComBo (DFN CLIP) | 79.3% | 85.8% | 61.6% | 75.6% | 1 |\n| ComBo (DINOv2) | 82.6% | 85.9% | 65.0% | 77.9% | 1 |\n| ComBo (SAM) | 55.2% | 79.4% | 56.9% | 63.8% | 1 |\n| ComBo (SigLIP) | 80.5% | 85.4% | 62.6% | 76.2% | 1 |\n| RADIOv2.5 | 83.8% | 86.6% | 62.1% | 77.5% | 1 |\n| ComBo (all 4 backbones) | 83.3% | 86.5% | 64.8% | 78.2% | 4 |\n| **ComBo (pruned backbones)** | **82.9%** | **86.7%** | **64.8%** | **78.1%** | **1.42** |\n\nWe observe that SAM is never selected with our approach, consistent with Table 10, where excluding it improves performance. Thus, our proposed model selection technique provides a solution to the reviewer's Q2.\n\n### 4. (Q3) Scalability to training examples and classes\n\n**ComBo demonstrates consistent performance gains with increasing training data and effective scaling on the Fungi dataset.**\n\nWe agree that validating ComBo's performance in different settings is valuable.\n\nWe included results using VTAB full setting subsets for datasets with large training samples in Figure 6 of our appendix. This demonstrates ComBo's performance with increasing training samples (1k, 5k, 10k, 20k), where ComBo consistently benefits from additional data.\n\nTo evaluate ComBo's performance on larger class counts, we produced new results on the Fungi dataset [a]. This dataset contains 1,394 species of mushrooms to classify. \n\n[a] Brigit Schroeder and Yin Cui. FGVCx fungi classification challenge 2018. github.com/ visipedia/fgvcx_fungi_comp, 2018.\n\nTo observe how performance evolves with class count, we produced subsamples with different numbers of randomly sampled classes and adapted a ViT-B/16 model trained on ImageNet-21K. We present results averaged over three seeds (each leading to different class subsets) compared against baselines:\n\n| Method         | 5 classes | 10 classes | 20 classes | 50 classes | 100 classes | 250 classes | 500 classes | 750 classes | 1000 classes | 1394 classes |\n|----------------|-----------|------------|------------|------------|-------------|-------------|-------------|-------------|--------------|--------------|\n| ComBo          | 77.8%     | 75.6%      | 75.6%      | 62.7%      | 60.8%       | 51.9%       | 44.8%       | 40.7%       | 39.4%        | 36.5%        |\n| Adapter+       | 93.3%     | 88.9%      | 81.1%      | 68.9%      | 66.9%       | 60.3%       | 52.8%       | 47.2%       | 44.3%        | 40.5%        |\n| Linear probing | 84.4%     | 77.8%      | 74.4%      | 63.1%      | 56.0%       | 46.5%       | 37.7%       | 32.2%       | 30.1%        | 26.6%        |\n\nComBo's behaviour with more classes is comparable to our baselines, with better performance with fewer classes but still decent performance with very high class counts.\n\n### 5. (Typos) Linear -> affine projection\nWe thank the reviewer; this is correct and we will update it in the paper.\n\n## Summary of updates we will make to the final paper\nFollowing your feedback, we will include:\n- **Extended architectural generalisation discussion** covering CNN integration possibilities and various tokenisation approaches\n- **Novel automatic model selection methodology and results** (developed in response to multiple reviewers' concerns) showing how our new pruning approach reduces inference costs whilst maintaining performance  \n- **Enhanced scalability analysis** including class count scaling experiments\n- **Terminology corrections** (linear → affine projection)"}]

Author rebuttal:

Meta-review: The paper introduces a probing method for combining features from multiple foundation models, aiming to leverage complementary strengths across modalities and architectures. Reviewers agree that the paper makes a focused contribution, backed by solid experimental results. Key weaknesses are the scope of experiments (e.g., limited to certain datasets and modalities) and the risk of not generalizing to more complex settings. The rebuttal provided additional evidence and new results, resolving some concerns. Overall, the strengths overweigh weaknesses, and I recommend to accept this paper.
Decision: Accept (poster)