Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling
Authors: Cristian Rodriguez-Opazo
Ehsan Abbasnejad
Damien Teney
Hamed Damirchi
Edison Marrese-Taylor
Anton van den Hengel
Venue: ICLR 2025 Poster (2025)
Keywords: CLIP
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper studies how different vision architectures trained with CLIP (e.g., ViTs and ResNets) exhibit distinct representations, classification performance, and robustness, even with the same training data and objective.

Abstract: Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers~(ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks.
This paper explores the differences across various CLIP-trained vision backbones.
Despite using the same data and training objective, we find that these architectures have notably different representations,
different classification performance across datasets, and different robustness properties to certain types of image perturbations.
Our findings indicate a remarkable possible synergy across backbones
by leveraging their respective strengths.
In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example. 
Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones.
The approach uses as few as one labeled example per class
to tune the adaptive combination of backbones.
On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 39.1\% over the best single backbone, well beyond traditional ensembles.

Published as a conference paper at ICLR 2025
SYNERGY AND DIVERSITY IN CLIP: ENHANCING PER- FORMANCE THROUGH ADAPTIVE BACKBONE ENSEM-
BLING
Cristian Rodriguez-Opazo" Ehsan Abbasnejad
Damien Teney
Hamed Damirchi" Edison Marrese- Taylor Anton van den Hengel*
'Australian Institute for Machine Learning University of Adelaide
"Idiap Research Institute, Switzerland "The University of Tokyo
(cristian rodriguezopazo ehs abbasne jad @adelaide edu
ABSTRACT
Contrastive Language-Image Pretraining (CLIP) stands out a prominent method for image representation learning. Various architectures, from vision transform- ers (ViTs) convolutional networks (ResNets) have been trained with CLIP serve as general solutions to diverse vision tasks. This paper explores the dif- ferences across various CLIP-trained vision backbones. Despite using the same data and training objective, we find that these architectures have notably different representations, different classification performances across datasets, and different robustness properties to certain types of image perturbations. Our findings indicate a remarkable possible synergy across backhones by leveraging their respective strengths. In principle, classification accuracy could be improved by over 40 per- cent with an informed selection of the optimal backbone per test example. Using this insight, develop straightforward yet powerful approach to adaptively ensemble multiple backbones. The approach uses as few as one labeled example per class tune the adaptive combination of backbones. On large collection of datasets, the method achieves remarkable increase in accuracy of up to 39.1 1% over the best single backbone, well beyond traditional ensembles.
INTRODUCTION
Large pre- rre-trained models are transforming machine learning, computer vision (Radford et al. 2021 He al. 2022 Kirillov et al., 2023 Liu al. 2023), and natural language processing (Devlin
et al.) 2018 Brown et al., 2020 Touvron et al. 2023). These models are typically trained with
self-supervised objectives, eschewing the need for manual annotations and enabling the use of very
large datasets (Jia et al. 2021; Schuhmann et al. 2022).
Contrastive Language-Image Pre-training (CLIP) (Radford et al. 2021 one such approach that enables various downstream applications involving vision and language. CLIP learns to align text and image representations when training a pair of vision and language encoders, which can be implemented with various architectures. These encoders to obtain representations of images and text, whose similarity can be simply evaluated with dot
then used in downstream applications
product. This enables e.g. zero-shot classification (Zhai al. 2022 Li al. 2022a Jia al. 2021
and cross- modal retrieval (Li et al. 2020a b. Yu et al. 2022]
Despite extensive prior work CLIP, there is still gap in comparing the representations from different vision backbones trained with this paradigm. Existing work has compared backbones for
their generalization capabilities (Goldblum al. 2023 Li Liet et al. 2022a Zhang al. 202T Gao
et al. 2021). finding that larger architectures generally perform better. This paper contributes to this area with an empirical study that compares CLIP-trainee backbones. We present new observations on how performance and robustness (e.g. invariance to image transformations) vary substantially across architectures. Within the same backbone family (e.g. different ViTs), different models present different patterns of performance across datasets (Figure 4.. The relation with model scale is also more complicated than suggested prior work (Figure

---

Published as a conference paper at ICLR 2025
Eusemble
Cascade
Noural Leggit Controlle
Bottest
Sape
NLC
1300
Buc (V)
Dus (V)
Sxagesd" Did
QLAP
Avenage
Uncestmationn
CLP
Input Image//I Trax
Backdbone Prodictions)
Arijust Logits
Enaemshle
Predict
Barkbone Prodictions
Confidens?? Add Model/Predicty
We propose a method to improve CLIP's effectiveness for image classification by combining the strengths of different backbones. (Left) For a given test image, the logits from different backbones are combined with a temperature scaling that weights their contribution to the final prediction. The scaling implemented the Neural Logit Controller (NLC, small MI.P) that is learned from as little as one labeled example per class. (Right) To reduce the computational load, our method can be combined with the Cascade framework (Wang et al. 2022a).
Figure I:
CALTECHI0 CARS Cieaado CIPARICO e CLEVR
Our empirical findings suggest that CLIP's ef- fectiveness for image classification could be im- proved by combining the strengths of different backbones. We thus present an ensemble method that combines backbones adaptively to each test example, using a temperature scaling that weights each backbone's 's contribution to the final predic- tion condition on the input image (see Figure [
COUNTRY211
EUROSAT FGVC
FLOWERS
FooD GTSRB
IMCNET-KK MNIST
PCAM
PETS SST2
Resisc45 STLI0
SUN399
1309
1208
100%
80%
40%
NLC Improvemexx
30%
A naive combination of backhones with traditional ensembling techniques [Lakshminarayana anna et al.) 2017 Dietterich 2000) (i.e. averaging the outputs) does not consistently enhance generalization per- formance. These approaches focus on prediction agreements rather than leveraging diversity. In contrast, our method uses a temperature scaling based on the idea of calibrating each backbone's s confidence. This intuition is further supported by the "modality gap" and variation in performance observed in prior work when adjusting CLIP's s log- its (Shi al. (023). We compare our approach more advanced ensembles, such as SuperLearner Cheng Ju and van der Laan (2018), which also combines the logits of multiple models by learning temperature scaling. However, unlike our method, SuperLearner does not adaptively adjust these tem- peratures based on the input features, implying that SuperLeerner cannot exploit the diversity of predictions (Tab. [.I]
20%
10%
0.2
Diversity of predictions
Figure 2: (Y axis) Relative improvement of NLC over best backbone vs. (X axis) predic- tions diversity 1, NLC always improves over the best backbone. Moreover, the clear correlation shows that higher diversity in predictions tends to result in greater improvements with NLC.
VITCBSS)) RN(S0) a V121B112 mu0) RS RI VIT E111144334-56 R8100100 19.j
Averze Accasay
108
83%
60%
Our proposed method, named the Neural Logit Controller (NLC), uses few labeled examples (as little one per class) to tune the combi- nation of backbones. Hence, we benchmark it against the oftte--f-theearr methods in a few-shot setting (Zhang et al. 2021; Radford et al. 2021] Zhou et al.] [002]] Gao et 2021). NLC shows remarkable performance and consistently outper- forms other approaches. Furthermore, we show that NLC complements existing methods like Tip-
Miclaal
-Cascada
-Esocailee lackiissc
$35
1,40 1,400
Figure 3: Average accuracy across 21 datasets for zero-shot ResNets and ViTs backbones, NLC Ensemble, and Cascade using 2 to 9 backbones, plotted against the Average GFLOPs. Demon- strates that ensembles can surpass the best zero- shot backbone with fewer GFLOPs.
'The diversity of predictions is measured as:
tsempice correctly predicted by all backbones Samples cossmully predicted by any backbons
2

---

Published as a conference paper at ICLR 2025
Adapter (Zhang al. 2021). Combining it with NLC in its original experimental framework yields improvements in accuracy of over 15%.
Finally, in-depth experiments reveal clear correlation between NLC's improvement over the best backbone and the diversity of correct predictions each dataset (see Figure 2 achieves over 120% of relative improvement when the diversity of predictions is close one, and never degrades the performance of the best backbone. In terms of efficiency, NLC outperforms the best backbone by combining only the top-four most efficient backbones, while using approximately 300 fewer GFLOPs
(see Figure 3). We also show how to use the Cascade framework (Wang et al. 2022a Varshney
and Baral 2022) with NLC to enhance performance while maintaining computational requirements
within the bounds of the original backbones.
Our contributions are summarized follows.
We perform extensive experiments across 21 datasets that reveal diverse predictions across CLIP backbones and distinct robustness properties to image transformations.
We evaluate the complementarity of backbones by measuring the potential improvement in classification accuracy (up to 43,5%) an optimal oracle selection of backbone per test example.
To leverage this complementarity, we propose adaptive ensembling method (NLC) that uses temperature scaling and requires a single labeled example per class. It improves accuracy over the best backbone by 9.1% average, surpassing previous ensembling frameworks.
We demonstrate significant advantages in computational efficiency. Combining the top four most efficient backbones enables NLC to outperform the best backbone with ~30 fewer GFLOPs. Integrating NLC with the Cascade approach (Wang et al. 2022a) maintains computational efficiency.
We demonstrate that NLC consistently outperforms state-of-the-ard few-shot methods. Moreover, integrating NLC with Tip-Adapter (Zhang al. 2021) gives the latter performance boost of over 15%.
INVESTIGATING DIFFERENCES ACROSS VISION BACKBONES
CLIP for zero-shot image classification. CLIP
(Radford et al., (02T) is a general technique to train a pair of vision and text encoders di and de on paired image/text data. The method uses self-
CLEVK
SUIRRAT
CHARIP
supervised objective to align the encoded represen- tations of matching images and text descriptions. Any image and text can then be mapped to a unified semantic space. This enables zero-shot image clas- sification by computing a compatibility score be- tween image and class description ("prompt"))
POVE
as score(x, = (x) (I). We follow previous work (Menon and Vondrick 2023) for the generat- ing of prompts e.g. an image of (label} which is (concept }. Subsequently, we calculate the compatibility score between the im-
UTTSUE
STI.DD
PCAM
Prrs
age feature (x) and the prompt representations (r()), where y represents label within the class The prompt with the highest similarity is then chosen the label for the given image.
RN2H
RVStndi
RKT0I
VIEB-32
VIE--15
VITL-14
VERL-18-3M
Detcca
Dioocaa
Figure 4: Zero-shot classification accuracy of various CLIP models on 21 datasets, and of the upper-bound "ORACLE" combination of ResNets (RN), ViTs, and all backbones.
2.1 COMPLEMENTARITY OF DIFFERENT BACKBONES
The vision encoder or "backbone" trained with CLIP can be implemented using various architectures. We want to identify whether different backbones have distinctive behaviours. We analyze the output of of them in zero-shot classification setting 21 standard benchmark datasets (see Section 4 for details). We propose two approaches to assess the differences in their zero-shot predictions and their possible complementary behaviour, First, define ORACLE prediction that combines the output
3

---

Published as a conference paper at ICLR 2025
IMAGENET- 1K
RN30
RN1O1
3114
RN00cIG
36359
RNS0GG
2038
6A8 FAR 921
1732
2557
255
1039
ESR
912
PON
506 A0T
1533
1749
857
516
990
753
42450
VII-B-22
VIT-B-16
$4108
VIT-L-14
SVIV
14-36-14-336
58285
2007
566
445
-204
4353
314
162
260
30
551
341
515
935
006
12070
VIT
ResNet
12426
1231
01500
Figure 5: Linear Venn diagrams showing the overlap of test images from ImageNet-Il correctly classified by different backbones (rows). Each column represents subset of images correctly classified by specific group of backbones (group size in column header). Row/column sums indicate the number of correct predictions per backbone/subset. We observe that (1) different backbones agree on a large part of the data. (2) They also make additional correct predictions on different subsets. (3) Accuracy usually grows with architectures size, but even within same family (VITs, ResNets), different models show different patterns of (in)correct predictions.
of the models per-image basis, using correct prediction if any of the models is correct, any other otherwise. This simulates a scenario where an informed choice of the optimal backbone could overlap
be made for each test image obtain the highest classification accuracy. Second, diagrams to visualize the patterns of (in)correct predictions across backbones and their different strengths on different subsets of images.
Figure 4 and Table B. summarize the performance of the different backbones. Note that they are pre-trained using the exact same dataset and objectives, as provided in the OpenClip project (Ilharco et al.) 2021). Our results show that backbones from different families (ViTs vs. ResNets) show sigmificant differences while controlling for model size (number of parameters). For example, ViT-B- 32 outperforms ResNet-50x4 in the CIFAR and CIFAR 100 benchmarks. However, this trend is not consistent across all datasets, e.g. on IMGNET- k. Additionally, the results indicate that backbones with more parameters do not always outperform smaller ones.
We then assess the ORACLE predictions using backbones from the same family (ORACLE RN for ResNets, ORACLE ViT for ViTs, and ORACLE All for combination of all backbones). The ORACLE All obviously outperforms the best individual backbone in every case. For instance, on EUROSAT, CLEVR, CUB, DTD, CIFAR100, and CARS, the ORACLE All shows improvements of 43.5%, 36.0%, 25.1% 21.8%, 16.6% and 16.0%. respectively. The large magnitude of these improvements highlights the complementarity of backbones. In other words, the correctly-classifet data is not simply growing as one considers larger and better models. Instead, different backbones perform well on different subsets of the data. This insight the key motivation for the ensemble method we propose in Section 3 We also found that similar findings have been made in domains
other than ours Roth et al. 2024 Zhong
(202T): Ramé et al. (0022).
We visualize in Figure 5 the overlap between different subsets of IMGNET-IK IK correctly predicted by different backbones. We observe comparable accuracy between the largest VIT (VYT-L-14-336, correct predictions) and ResNet (RN50x64) with 38, 285 vs. 36, 963 correct predictions. However, there are significant differences within each family: for example, among ResNets, only 22, 358 images correctly classified by all ResNets models, out of the 42, 426 ones correctly classified by any ResNet.
We further observe that every model has unique strengths by examining the performance of the oracle models. Specifically, ORACLE RN, ORACLE ViT, and ORACLE All increase the number of correct predictions to 42, 426, 42, 276, and 44, 506 out of 50, 000. Each backbone exhibits correct predictions unique to itself. In the ORACLE RN, RN50, RN101, RN50x4, RN50x16, and RN50x64 contribute 566, 455, 518, 782, and 745 exclusive predictions, respectively. In the ORACLE VIT,
4

---

Published as a conference paper at ICLR 2025
B-32, B-16, L-14, and L-14-336 present 903, 904, 543, and 818 unique predictions, respectively. Finally, when all backbones are used, ResNet contributes 2, 230 exclusive predictions, and ViT contributes 2, 080. It is important to note that correct predictions of any given model are not simply subset of those by the best model in its family.
2.2 OTHER SOURCES OF DIVERSITY
To better understand the effects of backbone complemen- tarity, and contrast these effects against the role of vari- ables that are known to drive performance in ensembles such random parameter initialization and learning via stochastic gradient descent, we propose to evaluate the diversity of correct predictions when combining multi- ple backbones, which as seen in our experiments above (Figure 2 strongly correlates with ensemble performance gains.
Doverstt
0.4
Concretely, we isolate the effects of three variables on prediction diversity: (1) backbone architecture, (2) pre- training dataset, and (3) number of training steps. For (1), evaluate the complementarity of ViT-L-14, ViT-B-32,
Barisborrs
Datswrts
Epoch
Figure 6: Diversity of predictions of (1) different Backbones with same pre- trained dataset, (2) same backbone with different pre-trained Datasets, and (3) same backbone and same pre-trained dataset in two different Epochs, Results show complementarity is higher when we combine different backbones.
and ViT-B-16 backbones using the same dataset. For (2),
analyze each VIT's performance using different pre-
training datasets (LAION-400M vs. OpenCLIP). For (3),
consider ViTs pretrained LAION-400M using the checkpoints at two different epochs (31 and 32).
Since, the best of our knowledge, there are no publicly released CLIP models that differ only on their initializa- tion, studies on this variable would, in principle, require us to train CLIP backbones from scratch, which is extremely compute-inted intensive.
TLR
GRAVSCALEE
Our diversity metric considers the aggregate of test ex- amples correctly classified and it is the ratio between the instances correctly classified by all backbones versus the instances correctly classified by any backbone It can thought of (I IoU), where loU is the "intersection over union" n sets correctly predicted samples, where n the number of methods considered. If every sample in
RM00L16
RN20054
KNN01
VEL--4
VIEL-14-336
this aggregate is correctly classified by all backbones (i.e. they all agree while being correct), diversity=0. Other- wise, if every sample can only be correctly classified by a single backbone, diversity=l. This allows us evaluate the complementarity of models.
BLUR
Impact on test classifica- tion accuracy of image transformations. Smaller values (closer to center) mean higher robustness, suggesting models are particularly robust to specific transforma- tions.
Figure 7:
F I.2) demonstrates that combining multiple backbones achieves higher average diversity than using the same backbone across different datasets or nearby solutions in the loss landscape. This indicates that training stochasticity and pretraining dataset are not the primary factors. Instead, the most meaningful complementarity comes from diverse backbones, as they excel different data subsets, as confirmed by the ORACLE results in Table I.]
2.3 ROBUSTNESS TO IMAGE TRANSFORMATIONS
To explore qualitative differences in behaviour of different backbones, we selected subset of each benchmark's test set and applied specific image transformations (resize, flip, grayscale, color jitter, and Gaussian blur). We then evaluate the zero-shot classification perfor- mance of each backbone on these corrupted images. 7 shows that ResNet 50 and ViT-B-16 the most resilient to flip. RN50x64 to res ze, and RN50x 16 to grayscale, VIT-I-114 and
5

---

Published as a conference paper at ICLR 2025
L-14-336 the most robust color jitter and Gaussian blur. These findings suggest that the backbones' ability to deal with out-of-distribttion images varies widely. Certain models prove particularly robust to specific transformations.
PROPOSED ENSEMBLING METHOD
The previous section showed that different backbones have different strengths and perform well on different types of data. We now use this insight to improve classification performance by combining multiple CLIP backbones.
Combining models by temperature scaling. Our approach is technically inspired by prior work on network calibration. which we apply an ensemble of models. Specifically, our technique can be seen as a variation of Platt scaling (Platt et al. 1999). This classical method uses the logits of model as features for logistic regression, which is trained on the validation set to produce calibrated probabilities. More precisely, given logit scores z; for an example i, Platt scaling learns two scalars a and and produces calibrated probabilities as q: = (az; + b). The parameters a and can be optimized using the negative log likelihood (NLL) loss over the validation set, with the model weights being frozen during this process.
A common of Platt scaling is known temperature scaling (Guo et al., 2017). In this approach, is used for all classes of a given model. The new, calibrated confidence prediction is given by q = max; softmax (z:/t)(A) where called the temperature, 2, are the logits for example returned by the uncalibrated model. Usually, is optimized with respect to the NLL on the validation set aiming to reduce the overconfidence the model on its predictions and to produce more reliable predictions, but because the parameter t does not change the maximum of the softmax function, the class prediction O, remains unchanged, meaning that the performance of a given model remains the same.
a single scalar parameter
Learning temperature coefficients. In our method, we aim to jointly optimize a set of temperature parameters to with b [1...., B] for a set of B backbones. The aim is to combine their predictions by adjusting each backbone's 's confidence depending on the confidence of the others, and the input example. We learn the temperatures 4a that weigh the logit 2 for a backbone and example x using the cross-entropy loss, and then we combine the logits via weighted sum.
Concretely, we train one-layer MLP (our Neural Logit Controller) to predict the set of temperatures that best calibrate the backbone mixture. The MLP takes input the concatenated representations obtained by passing the images through the encoder 6. of each backbone E IB. As depicted in Figure 1 The MLP directly produces a vector of temperatures E BR and is trained on a holdout set for the training set each target dataset using the cross-entrope loss between final predictions and ground truth labels.
4 EXPERIMENTS
Backbones. We consider the original selection of backbones described by Radford et al. (2021) This includes the ResNets (He et al. 2016) RN50, RN50x4, RN50x16. RN50x64 and RNI01, I, and the ViT's (Dosovitskiy al. 2021) B-16, B-32, L- and L-14-336. All models are obtained through the open-source project OpenCLIP (Iharco et al. 2021
Datasets. We use selection of 21 popular image classification datasets: CALTECH101 (Li et al. 2022b), CARS (Krause et al. (2133. 11FAR10 (Krizhevsky et al. 2009), CIFAR 100 (Krizhessky et al.
2009,, CLEVR (Johnson et 2017). CUB (Wah et al. [201] [ 1). DTD (Cimpoi et al. 2214). EUROSAT
(Helber et al. 2018), FGVC (Maji al. 2013), FLOWERS (Nilsback and Zisserman 2008). FOoD
(Bossard et al. 2014), GTSRB (Hubenn et al. 2013). IMGNET- 1k (Deng et al. 2009) MNIST (Deng
2012), PCAM Veeling et al.] 2018), PETS Parkhi al. (0122,, RenderedSST2 (Socher et al. 2013).
RESISC45 (Cheng et 20.7 7), STL10 (Coates et al. 2011) and SUN397 (Xiao et al. 2010). They
include diverse images of nature, animals, places, medical scans, satellite images, and man-made objects. The evaluation metric is simply the accuracy of each dataset's test split. The aggregated
performance is the average accuracy across all datasets weighted equally.

---

Published as a conference paper at ICLR 2025
Man 7
Accurccy
75%
4
60%
25%
Vore T-
.90
51
Loo Ave
0.5
4.1
6.7
0%
CALTECHIO
GIAR
CIFAR
CLEVE
COUNTRYS
CUD
EUROSAT
ROYC
POOD
GTSRB
INGNET
MNIST
PCAM
PrTT
SST2
RESSE4SS
STLM
SUN
33
19
96
39.1
Figure 8: Comparison of the top-performing single backbone (SINGLE- BEST) with NLC and other ensemble strategies on top of zero-shot CLIP backbones. In the table, Mean, Max and Min A summarize the difference in performance across datasets with respect to the SINGLE- BEST backbone.
Baselines.
Our experiments consider several ensembling baselines. Non-parametric approaches:
Logit averaging (LoG-AvG) simply takes the average of the logits (scores) of the models.
Voting (VOTE T-1) uses majority vote. Each model casts a vote for its max-scored class. The final prediction is the class with the highest number of votes. Compared LOG-AVG, this does not use the soft score values and the uncertainty reflected therein.
Parametric approaches:
Calibrated logit averaging (C-LoG-AVG) first calibrates each model independently using tem- perature scaling. They are then combined by simply averaging their logits.
Super Learner (SL) (Cheng Ju and van der Laan 2018) flexible ensemble learning framework that optimally combines predictions from multiple base learners. In contrast to our method, SuperLearner learns temperature scaling factors without considering the input. Thus, finding the best scaling for the total validation set.
Mixture experts (MoE) is popular method improve performance by combining specialized submodels of experts (Chen et al. [022] Lepikhin et al., 2020 Rau [019) Zoph et al., (2022). each one which focuses on a specific input region. A gating network determines which expert relevant for a given input. We use the Sparse MoE implementation (Zoph et al. 2022) by Rau
(2019), where the MoE layer's input is the concatenation of vision features 26, from Φb. We use standard hyperparameters: 9 experts trained for classification with cross-entropy loss with Adam (Kingma and Ba 2014) and a learning rate 2 for 300 epochs.
See Appendix C a discussion of additional baselines. The appendix also contains additional results on the combination of backbones and how they perform under distribution shifts the IMGNET- 1K dataset (Section E). Finally, Section H presents -ppecfc- results including overlap diagrams, an analysis of possible model combinations using NLC and our ORACLE, as well as the dataset-specific learned temperature values.
4.1 COMPARISON OF
METHODS
In this section, we evaluate the proposed NLC approach along with various non-parametric and parametric ensembling techniques. The non-parametric ones are "static" while the parametric ones use the training split of each target dataset to adjust the ensemble adaptively. The following results will show the importance of this adaptive setting to leverage the backbones' respective strengths, and the superiority of our method over baseline parametric techniques.
Figure g and Table C.I show the performance of each technique. Among non non-parametria baselines, LOG-AVG fails to enhance overall performance beyond the best backbone. One notable exception the COUNTRY dataset where LOG-AvG shows a substantial improvement over the best backbone. Overall, though, simple score averaging does not seem to exploit the complementarity of the back- bones, with an improvement in accuracy of only +1.3%. When the backbones are calibrated with C-LoG-AVG, performance improves compared to the non-calibrated version, across all datasets except GTSRB, MNIST, RESISC45, and SUN397. Vote T-1 is mostly ineffective and does not signif- icantly improve the best backbone. Among parametric methods, the proposed NLC shows the most substantial improvement, up to +399.1% on the EUROSAT dataset. On average, we obtain a commendable improvement of +9.1 1% over the SINGLF-BEST backbone across all datasets.
7

---

Published as a conference paper at ICLR 2025
BUNGLE-BUST
Vovn T-1
LoG-AvG
SL.
MoE
NLC
100%
Accuracy
Mrx
LOG-ANG
C-LOCLANG
44
Cu
CIFAR
CLEVE
COUNTRY211
Ctn
DTD
EUROSAT
POVC
FooD
GESRB
IMONET
MNNT
PCAM
PHIS
RESIN4S
STLIO
SUN299
MoE
-230
20.5
51.
1.6
0.0
5.4
Figure 9: Comparison of the top-performing single backbone (SINGLE-BEST) with NLC and ensemble strategies on top of CLIP backbones linear classifier, where Mean, Max and Min A summarize the difference in performance across datasets with respect to the SINGLE-BES,
4.2 INVESTIGATING BENEFITS BEYOND ANTASET-SPECING LINEAR CLASSIFIERS
We now design an additional experiment to investigate the benefits of the proposed NLC. A standard approach to adapt CLIP is to train a dataset-specific linear classifier on frozen visual features. In this section, we first train such linear classifiers, using each backbone's ' frozen features and each dataset's training split. Whereas NLC normally acts from these classifiers instead. Since the linear classifiers already are strong dataset-specific models, might expect that only small additional improvements are possible. Yet, the following results will show that the NLC obtains further performance gains thanks to the instance-level adaptation. In contrast alternative approaches, also never reduces the performance compared to the best single linear classifier.
CLIP scores,
train it to act on the logits
Setup. We follow the setup of Li al. 2022ap. We initialize the weights of the linear classifiers using language weights, which was shown to be more stable than random initialization. The output of each backbone Φs is L2-normalized before being passed to each linear classifier, which trained using 90% of the target dataset's 's training split. The remaining 10% are used to train NLC.
Results. Figure g and Table D.2
Table 1: Accuracy (%) in a few-shot setting
present the results combining linear classifiers with both non-parametric and parametric approaches. Similar to Section 4.1I, the proposed NLC con- sistently improves performance across all datasets, with an average improve- ment of +1.6% over the SINGLE- BEST linear classifier. As expected, the improvement is reduced compared the original NLC since some of the gains now realised by the dataset-specific linear classifier. Yet Table clearly shows that additional gains can be made over the linear classifiers, i.c. that there remains an exploitable complementarity across backbones. By examining the ORACLE performance in this setting. we can see that the potential benefits of com- bining multiple backbones remain. Interestingly, the MoE approach does not reach the performance level of the best backbone on several datasets. It shows degradation sometimes down to -0..9%. This suggests that MoEs may struggle partition the input space effectively into distinct clusters, specific to certain experts, potentially limiting its functionality.
Few-shot
2
4
8
16
Linear peohe CL.IP Radford et al.
2021
22.2
31.9
41.2
49.5
56.1
CooP al.
2024
47.6
50.9
56.2
59.9
63.0
CLIP-Adapeer Gao
2021
61.2
61.5
61.8
62.7
63.6
Tip-Adapter Zhang el
/
2011
60.7
61.0
61.0
61.5
62.0
Tip-Adapter-F Zhang 2021
61.3
61.7
62.5
64.0
65.5
NLC
78.2
78.1
78.2
78.3
78.4
4.3 COMPARISON WITH FEW-SHOT ADAPTER METHODS
We now examine the performance of the proposed NLCC in a few-shot setting. We also show that it complements existing few-shot adapter methods and improves their accuracy across various datasets.
We compare our method against Tip-Adapter, Tip- Tip-terpter-- (Zhang et al., 2021). few-shot L Linear- probe CLIP (Radford et al. 2021). CoOP (Zhou et al., 2022). and CLIP-Adapter (Gao et al. 2021). Linear-probe CLIP fine-tunes a linear classifier a few-shot training set the frozen CLIP backbones. CoOP (Zhou et al.) 2022) generates different prompt designs to make prompts learnable. CLIP- AdiPeed (Gao al. 2021) enhances few-shot classification by introducing feature adapter on CLIP's visual and textual encoders. Tip-Adapter (Zhang et al.) (02TT) achieves comparable performance to CL.P-Adapter without requiring training, using key-value cache model from the
8

---

Published as a conference paper at ICLR 2025
EUROSAT
FGVC
DTD
PETS
70%
83%
70%
60%
0015
6099
9015
00%
VIESB-23
4016
2005
KILD
4076
Shols
Shols
Sholks
Shols
Figure 10: Peformance of TiP-Adapter applied to different zero-shot CLIP backbones, and the combination of these with NLC, showing how our method provides complementary benefits.
few-shot training set. Tip-Adapter-F can further enhance performance by fine-tuning the cache. As shown in Table [] our NLC shows an outstanding performance over compared methods and consistently surpasses other few-shot methods by learning how to combine multiple backbones, which is complementary approach to any of these adapters. In the appendix, Table F.2 shows that current few-shot adapter methods also have performance differences with various backbones, and still, our NLC surpasses the best backhone reported by these existing methods.
Finally, we integrate Tip-Adapter (Zhang et al. 2021) with our NLC and conduct experiments EUROSAT, FGVC, DTD, and PETS. Initially, we apply Tip-Adapter and Tip-Adapter-F inde- pendently on each backbone, following the protocol from Zhang et al. (202T) with 1, 2, 4, 8, and 16 shots. Subsequently, we employ our ensembling mechanism fuse the adapted back- bones. To train NL.C, we follow their setting and use the same validation split as used by Tip-Adapter to linearly combine their logits with CLIP.
1.0
0.8
0.4
0.2
RN50
ANSOst
RN50026
RN50064
RX102
VIT--32
VITe
VITL-144
VITL-34-330
comparison of the perfor- mance of Tip-Adapter over different zero-shot CLIP backbones, and the combination NLC. Across the four evaluated datasets, NLC improves over each version of Tip-Adapter in the few-shot setting. Notably, NLC improves the performance of the best Tip-Adapter back- bone (L-14-336) of up to 10% for EUROSAT using 16 shots.
See Figure and Tables
these Tip- Tdp-Arapter versions with
Distribution of max- normalized temperature values learned by our NLC method on the CLEVR dataset.
Figure 11:
Moreover, when we compare NLC and SL in the Tip-Adapter settting, NLC shows better perfor- mance, Table F.3
4.4 EXAMINING THE LEARNED TEMPERATURE VALUES
We visualise the learned temperature values understand how the NLC method assigns weights to different backbones, revealing its strategy for leveraging the strengths of diverse architectures. Figure shows the learned temperature values for NLC on the CLEVR dataset (see Section for other datasets). By examining the learned temperatures, we can gain insight into the contribution of each backbone. We note that the backbone with the largest overall weight not consistently the deepest one, such as 101. The uniform distribution of learned temperatures across backbones indicates that NLC effectively utilizes the strengths multiple backbones, resulting a more balanced and accurate labeling process, complementing analysis in Figure ? These findings highlight that NLC benefits from the diversity in model predictions, reinforcing the advantage of combining various backbones.
5 RELATED WORK
Foundation models in computer vision. The trend in computer vision over the past decade has been to train larger and larger models on increasingly diverse tasks and increasingly larger datasets (Gadre et al. 2023 Schuhmaan et al., (002).. Scale and generality have proved to be key enablers to the
performance and robustness of these models (Fang et al.) 2022; Gan et al. 2022 Mayilvahanan et al.
2023: Santurkar et al. 2022 Tu al. 2024). Various methods have been proposed pre-train such

---

Published as a conference paper at ICLR 2025
models (He et al. 2022; Radford et al. 2021 Yu et al. 2022). CLIP (Radford et al.) 2021 is notable one, specifically designed to exploit noisy image-text pairs scraped from the internet. CLIP's significance lies in its scalability and its ability to generate meaningful alignments with prompts, facilitating zero-shot classification
2iiiip/ iip.ii2222eeeeateer CLIP's popularity is due to its scalability and ability to handle diverse text prompts that enable a variety of downstream applications (Gao et al. [021]] Zhang et al. [021] Li et al. (2022a). Various versions of CLIP
have been trained on different datasets and vision backbones such as ResNets (He al. 2016) and
ViT's (Dosovitskiy et al. 2021).
Comparing vision backbones, Goldblum al. (2023) recently conducted a comprehensive compar- ison of pretrained backbones across various tasks including image classification, object detection, segmentation, and image retrieval. In comparison, this paper presents complementary observations of the strengths of different backbones on different datasets and types of data. Moreover, we leverage these findings with a novel ensembling approach this complementarity of different backbones. Other works investigating differences across vision architectures include (Angarano et al., 20022] Pinto et al. 2022| 202I; Wang et al., 2022b) and others specific to CNNs (Abello et al. 2222 Hermann et al. 2020 and ViTs (Naseer et al.] 2021). ). Finally, we find that there are works analysing the diversity
of models for knowledge distillation Roth et al.] (2024), complementarity on models trained don different subset of data Ramé et al. (2022) and Zhong et al. (2021) exploring the diversity on LLMs.
Model ensembling. Combining multiple machine learning models is a classical approach for improving predictive performance. Simply averaging the outputs of several models is simple, effective technique (Bauer and Kohavi 1999, Breiman 1996: Dietterich] 20000] Lakshminarayan al. 2017) with studies dating back more than three decades ago. The approach has also been applied to deep neural networks (deep ensembles (Lakshminarayanan et al.) (0017), which has shown benefits in various domains including higher accuracy under distribution shift (Ovadia et al.. 2019) Teney et al. 2018). The diversity of the combined models (in terms of uncorrelated errors) has also been shown to be critical to these improvements (Hao al. 2024] Wortsman et al. 2022). The closest to our work is the SuperLeernee framework (Cheng Ju and van der Laan] 2018), which similarly investigates multiple classical ensembles and introduces a new ensemble that learns temperature scaling. However, but also explores the complementarity of CLIP backbones across 21 datasets. Furthermore, unlike SuperLearner, our method adapts its temperature scaling factors based on the input, which plays crucial role in ensembling CLIP backbones.
study not only evaluates the performance of various classical ensembles
CONCLUSIONS
This paper presents an analysis of vision backbones in the CLIP framework, focusing on the task of image classification. Unlike prior studies that span various downstream tasks (e.g. Goldblum et al. (2023) our emphasis lies in identifying the unique strengths of various backbones. Our experiments revealed a distinctive complementarity across architectures and an avenue for enhancing CLIP's performance by synergistic combination. We proposed an ensemble approach that reweights the logits from each backbone condition the input data to yield more accurate predictions in image classification across variety of datasets.
Limitations. First, our evaluation focused mostly on image classification, even though CLIP can be used for variety of downstream tasks. Although we test our method on out-of-distribttion Sec. and also obtain the upper bound performance on Image-Text retrieval Sec. K more work needs to be done on how to combine the representations. Second, our approach relies on a late fusion of backbones by adaptively adjusting their logits. Alternative approaches that leverage the same initial motivation use knowledge distillation, which requires training the student backbone again Roth jt al. 2024). Third, although we employed the Cascade method (Wang et al., 2022a) to reduce the computations, our method still needs to pass test image multiple times through vision encoders. This adds complexity and computational overhead, which can be important if a large number of backbones are considered. A possible solution would involve fusing backbones at the training stage to require only a single forward pass at test time.
Future work. We could emulate the behavior of given backbone by learning their representations using the main branch's 's early layers. Other efficient methods could also be developed to combine predictions from multiple backbones to enhance the scalability of the idea.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGE
This research is funded in part by the Australian Government through the Australian Research Council (Project DP240103278) and the Centre of Augmented Reasoning at the Australian Institute for Machine Learning, established by grant from the Department of Education. We thank the NVIDIA corporation for donating one of the GPUs used for the experiments. Finally, we would like to thank Stephen Gould for his valuable feedback on the paper.
REFERENCES
A. A. Abello. R. Hirata, and Z. Wang. Dissecting the nigh-frequency bias in convolutional neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 863-871, 2021.
Angarano, M. Martini, Salvetti, Mazzia, and M. Chiaberge. Back-to-bones: Rediscovering the role of backbones in domain generalization. arXiv preprint arXiv:2209.01 2022.
E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine learning, 36: 105-139, 1999.
L. Bossard, M. Guillaumin, and L. Van Gool. Food-101-ming discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 621 2014, Proceedings. Part VI 13, pages 446-461. Springer, 2014.
L. Breiman. Bagging predictors. Machine learning. 24:123- 140, 1996.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. arXiv preprint 200.v:2005.14166 2020.
Z. Chen, Y. Deng, Y. Wu, Q Gu, and Y. Li. Towards understanding the mixture-of-experts layer in deep learning. S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 23049-23062 Curran Associates, Inc., 2022.
G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-1885 1883, Oct 2017. ISSN 1558-2256.
A. B. Cheng Ju and M. van der Laan. The relative performance of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45(15)2280- 2818, 2018. doi: 10. N0.1080/02664763.1 2018 1441383. URL
02664763 2 2088 1441383
PMID: 31631918.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
J. Deng, W. Dong, R. Socher, L..J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-14,, 2012.
J. Devlin, M.-W. Chang. K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1100v: 04,00 2018
T. G. Dietterich. Ensemble methods machine learning. In International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
11

---

Published as a conference paper at ICLR 2025
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image worth 16x words: Transformers for image recognition scale. ICLR, 2021.
A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data deter- mines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 6216-6234, PMLR, 2022.
S. Y. Gadre, G. Ilharco, A. Fang. J. Hayase, G. Smyrnis, Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang. E. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, Song.
H. Hajishirzi, A. Farhadi, R. Beaumont, S. Oh, A. Dimakis, J. Jitsev, Y. Carmon, V. Shankar, and
L. Schmidt. Datacomp: search of the next generation of multimodal datasets, 2023.
Z. Gan, L. Li, C. Li, L. Wang, Z. Liu, Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends& in Computer Graphics and Vision, 14(3-4); 163-352, 2022.
Gao, S. Geng, R. Zhang, T. Ma, R. Fang. Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2 2021.
M. Goldblum, H. Souri, R. Ni, M. Shu, V. Prabhu, G. Somepalli, P. Chattopadhyay, M. Ibrahim,
A. Bardes, J. Hoffman, R. Chellappa, A. G. Wilson, and T. Goldstein. Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks, 2023.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1321-1330. PMLR, July 2017.
Y. Hao, Y. Lin, Zou, and T. Zhang. On the benefits of over parameetttttiiinooonnn for out-of-distribution generalization. arXiv preprint UrXiv:2403.17599 2024.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
K. He, X. Chen, Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners, In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
Helber, B. Bischke, A. Dengel, and D. Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pages 204-207. IEEE, 2018
D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and Gilmer. The many faces of robustness: A critical analysis of out-of-distributiio generalization. ICCV, 2021a.
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. CVPR, 2021b.
K. Hermann, Т. Chen, and Kornblith. The origins and prevalence of texture bias in convolutional neural networks. Advances in Neural Information Processing Systems, 33:19000-190 19015, 2020.
S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing, and C. Igel. Detection of traffic signs in real-worle images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks, 2013.
G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong. J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip. Technical report, July 2021. If you use this software, please cite it as below.
C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and Dision-langure representation learning with noisy text supervision. In International conference on machine learning, pages 4904-491 PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
J. Johnson, B. Hariharan, L. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE conference computer vision and pattern recognition, pages 2901-2910, 2017.
D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, Whitehead, A. C. Berg, W.-Y Lo, P. Dollár, and R. Girshick. Segment anything, 2023.
Krause, Deng, M. Stark, and Fei-Fei. Collecting large-scale dataset of fine-grained cars. Technical report, 2013.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. /6668, 2020.
C. Li, H. Liu, L. H. P. Zhang, J. Aneja, J. Yang, P. Jin, H. Hu, Z. Liu, Y. J. Lee, and Gao. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Neural Information Processing Systems, 2022a.
F.F. Li, M. Andreeto, M. Ranzato, and P. Perona. Caltech 101, Apr 2022b.
G. Li, N. Duan, Y. Fang, M. Gong. and D. Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI conference on artificial intelligence, 2020a.
X. Li, X. Yin, Li, Zhang. Hu, I.. Zhang, I.. Wang, Hu, Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020b.
H. Liu, C. Li, Y. Li, and Lee. Improved baselines with visual instruction tuning. 2023.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1066::1306.55/2 2013.
Mayilvahanan, T. Wiedemer, E. Rusak, M. Bethge, and W. Brendel. Does clip's generalization performance mainly stem from high train-test similarity? arXiv preprint arXiv:23 2023.
Menon and Vondrick. Visual classification via description from large language models. ICLR, 2023.
M. M. Naseer, K. Ranasinghe, H. Khan, M. Hayat, F. Shahbaz Khan, and M.-H. Yang. Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34:23296 23308, 2021.
M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan. and J. Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pages 13991-14002, 2019.
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and patlern recognition, pages 3498-3505, IEFE, 2012.
13

---

Published as a conference paper at ICLR 2025
F. Pinto, P. Torr, and P. K. Dokania. Are vision transformers always more robust than convolutional neural networks? In Advances in Neural Information Processing Systems, 2021.
F. Pinto, P. H. Torr, and P. K. Dokania. An impartial take to the emn transformer robustness contest. In European Conference on Computer Vision, pages
Springer, 2022.
J. Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
A. Ramé, K. Ahuja, J. Zhang. M. Cord, L. Bottou, and D. Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. arXiv preprint arXiv:2212.104455 2022.
D. Rau. Sparsely-gated mixture-of-experts pytorch implementation, 2019.
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet? In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 5389-5400. PMLR, 09-15 Jun 2019.
K. Roth, L. Thede, A. S. Koepke, O. Vinyals, O. J. Henaff, and Z. Akata. Fantastic gains and where to find them: On the existence and prospect of general knowledge transfer between any pretrained model. In The Twelfth International Conference on Learning Representations, 2024.
URL https: http::://ppereevve
n /ast/fooze forum? id m50ekkttt
Santurkar, Dubois, R. Taori, P. Liang, and T. Hashimoto. caption worth a thousand images? study representation learning. In The Eleventh International Conference Learning Representations, 2022.
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 35:25278-25294, 2022.
Shi, M. C. Welle, M. Björkman, and D. Kragic. Towards understanding the modality gap in CLIP. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
R. Socher, A. Perelygin. J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA, Oct. 2013. Association for Computational Linguistics.
D. Teney, P. Anderson, X. He, and A. Van Den Hengel. Tips and tricks for visual question answering: Learnings from the 2017 challenge. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4223 4232, 2018.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, F. Grave, and G. Lample. Llama: Open and efficient foundation language models. ArXiv, 13972, 2023.
W. Tu, W. Deng, and T. Gedeon. closer look at the robustness of contrastive language-image pre-training (clip). Advances in Neural Information Processing Systems, 36, 2024.
N. Varshney and C. Baral. Model cascading: Towards jointly improving efficiency and accuracy of NLP systems. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Con- ference on Empirical Methods Natural Language Processing, pages 11007-11021, Abu Dhabi, United Arab Emirates. Dec. 2022. Association for Computational Linguistics. doi: 10.18653/v1/ antho org 2022. emn amnnp-ma in. 756
2022. 2022.emnlp-main.77 mainn. JRL
B. S. Veeling, J. Linmans. Winkens, T. Cohen, and M. Welling. Rotation equivariant CNNs for digital pathology. arXiv preprint arXiv: 1806.03962, June 2018.
14

---

Published as a conference paper at ICLR 2025
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-uesd birds-200-2011. Technical Report UNS-TR-2011--0 California Institute of Technology, 2011.
J. Wang, K. Markert, M. Everingham, et al. Learning models for object recognition from natural language descriptions. In BMVC, 2009.
X. Wang, Kondratyuk; E. Christiansen, K. M. Kitani, MovshovitzAAtiaa, and Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. In International Conference on Learning Representations, 2022a. htt.btttss::/pereeeeeee
forum?
0d=M
Z. Wang. Y. Bai, Y. Zhou, and C. Xie. Can CNNs be more robust than transformers? arXiv preprint 0:Xi::2206.03452, 2022b.
M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lpees A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965 23998.. PMLR, 2022.
Xiao, J. Hays, K. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference Computer Vision and Puttern Recognition, pages 3485-3492, June 2010.
J. Yu, Z. Wang, V. Vasudevan, L.. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners image-text foundation models. arXiv preprini 2201v:2205.01117 2022.
X. Zhai, X. Wang, Mustafa, Steiner. D. Keysers, A. Kolesnikov, and Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1812 18123-18133, June 2022.
R. Zhang, R. Fang, P. Gao, W. Zhang, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training-free clip-adapter for better vision- languaee modeling. arXiv preprint arXtv:2111.03930. 2021.
R. Zhong, D. Ghosh, D. Klein, and J. Steinhardt. Are larger pretrained language models uniformly better? comparing performance at the instance level. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3813-3827, Online, Aug. 2021. Association for Computational Linguistics. doi: 10. /0.18653/v1/2021. findings-acl 334. URL
org/y/l1.ggg222.. wtte2....nnnee findings-acl. 334
K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348 2022.
B. Zoph, I. Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022.
15

---

Published as a conference paper at ICLR 2025
A REPRODUCIBILITY
The one-layer MLP is trained with ADAM optimizer with learning rate of 2e-4. We use a weight decay of 0.01. It receives as input the concatenation of the features, the hidden layer has a width of 128 and its outcome is the temperature values for the used backbones. All the backbones used in the paper are pre-trained using the same dataset called openai i and objectives from the OpenAl Fundation https: htt///gtthb/ www.ooommmm......... wwwfonnatttiiimmmmmmmmmm ons /open_ clip In the Cascade Wang et al. (2022a), we use the probability as the confident measurement with a threshold of 0.9. When the method is confident with a probability bigger than 0.9 we produce final prediction, otherwise, add the next backbone. All our experiments are done in one NViDIA GeForce RTX 4090 GPU, in a small server with 64 GB of RAM and 32 CPU cores.
B ORACLE PERFORMANCE
Table B.I presents the results depicted in Figure 4 from the main paper. It includes the number of parameters and GFLOPs/Image for each backbone, the performance of zero-shot CLIP (OpenAI) on the selected datasets, and the performance of the proposed ORACLE method.
We observe that backbones with similar number of parameters from different families can show significant differences in performance. For instance, ViT-B-32 outperforms ResNet-50x4 in the CIFARIO 1 and CIFAR 100 benchmarks. However, this trend not consistent across all datasets, as illustrated by the results on IMGNET-IK, 1k. Additionally, higher number of parameters does not always correlate with better performance across different backbone families.
Notably, the ORACLE method, which leverages all possible backbones, shows substantial improve- ments on several datasets: 43.5% 5% on EUROSAT, 36.0% on CLEVR, 25.1% % on CUB, 21.8% on DTD, 16.6% on CIFAR 100, and 16.0% on CARS. This clearly demonstrates the complementarity of each zero-shot CLIP backbone. Table B.2, we present the results of the artificial image variations shown in Figure In this experiment, selected subset of each test set from the benchmarks and applied specific artificial image variations -i.e., Resize, Flip, Grayscale, Color Jitter, and Gaussian Blur- to evaluate the zero-shot classification performance these "corrupted" images for each backbone.
The results indicate that ResNet-50 and V1T-B-16 are the most resilient to the Flip transformation, RN50x64 shows the highest resilience to Resize, and RN50x 16 performs best for Grayscale. ViT-L-14 and 14-3333 exhibit the most robustness against Color Jitter and Gaussian Blur transformations. These findings suggest that each backbone has a unique ability to handle specific types of image variations.
C ADDITIONAL BASELINES
To complete the experiments and establish comprehensive baseline compared to other approaches, we introduce two non-parmettric and one parametric more ensemble methods to our benchmarks:
Table B.I: Zero-shot accuracy performance comparison of CLIP backbones on 21 datasets, showcas- ing the number of parameters and GFL.OPs per image for each backbone. Additionally, explore the performance of our empirical upper-bound prediction, denoted as 'ORACLE', representing the ideal combination of ResNet (RN), ViT, and All backbones.
Num
inogo
S0NST
CLIRSST CL.P-aaaa
50
102.00 MM
15.2
77.8
54.2
71.5
40.3
21.6
15.4
46.5
41.2
28.2
17.1
06.1
77.8
35.1
59.6
45.1
63.9
85.8
56.6
44.0
96
58.6
101
119 TM
25.5
$1.88
61.1
80.38
47.6
24.4
169
49.5
43.7
26.4
18.6
65.2
61.9
37.5
63.1
4600
39.1
16.9
63.9
54.5
96
$7.3
50s4
176 3M
51.8
45.1
20.5
20.4
34.2
48.6
65.1
53.3
56.7
68.9
67.2
53.1
57.9
62.7
5tu64
85.1
248
48.0
47.
B::22
151.MM
14.9
K3.7
50.7
80.3
56.2
23.3
17.
$3.0
440
37.4
19.3
66.5
81.
12.6
63.1
38.1
62.5
67.5
58.6
54.3
97.1
61.3
B.16
149:00
41.1
86.6
64.6
60.38
56.9
21.1
22.8
35.3
25.1
44.1
24.4
714
WT.9
43.3
65.1
95.5
50.7
89.1
515
58.3
1622
63.7
L-14
75.8
19.4
31.9
63.1
553
47.2
75.5
73.3
5007
93.6
58.9
RN
90.2
91.3
96.5
76.8
56.2
412
82.5
73.5
81.9
52.3
56.0
95.1
68.2
84.9
00.
$1.0
98.1
95.5
82.3
19.3
82.7
VIT
91.9
$0.9
06.5
47.1
34A
440
78.9
67.3
12.0
$1.9
88.7
95.4
620
$4A
85.6
95.4
07.5
02.5
76.7
99.7
905
91.0
60.4
520
88.1
78.2
01.5
66.1
18.9
72.9
80.0
95.1
99.1
99.1
08.6
38.7
993
16

---

Published as a conference paper at ICLR 2025
Table B.2: Impact of artificial alterations of the input image on each backbone in terms of delta accuracy concerning the original performance
RN
VIT
50x16
50x64
1-14--36
Resize
-3.81
-3.47
-1.90
-1.25
-4.05
-4.41
-2443
-2.82
-2.11
Flip
-0.36
-0.55
Grayscale
-6.36
-7.37
-5.98
-7.01
-7.52
-8.94
-8.70
-7.73
66.9
Color litter
-2.71
-3.00
-2.19
-2.93
-2.64
-3.52
2255
-1.67
-1.23
Blui
-3.72
-2.06
This method utilizes the Shannon entropy assess the confidence of each backbone in making a prediction. The backbone with the highest confidence for a given prediction is selected as the source of the final prediction.
Confidence (CONF):
Voting (VOTE T-1 and VOTE T-3) Voting combines different classifiers to predict the class label. In the VOTE T-1 approach, we consider the top-1 prediction from each backbone, and the final prediction is determined by the label with the highest number of votes. In cases where multiple labels receive the same number of votes, we select the one with the highest probability. Additionally, we explore the VOTE T-3 method, which seeks consensus among the backbones by considering the three most likely predictions from each backbone. These predictions are weighted based on their position within the top-3 list.
Genetic Algorithm (GAC). To showcase the importance of adaptively adjusting the logits, we use a genetic algorithm to find the optimal temperatures for combining the backbones. These temperatures are fixed after training, ensuring they remain constant regardless of the input.
Calibrated confidence (C-CONF) For this approach, we first calibrate the probabilities of each backbone independently using temperature scaling. Then follow the procedure above and utilize the Shannon entropy of each backbone to select the one with the highest confidence
Table C.I presents the baselines' results in the combination of zero-shot CLIP backbones. In non- parametric baselines, consistently with the presented in the main paper, leveraging the confidence of each backbone in its predictions consistently fails to enhance the overall performance beyond that of the best backbone. The lack calibrated probabilities in the CONF contributes to overconfidence in some backhones, resulting in performance degradation when combined. Calibrating the confidence (C-CONF) leads to improvements, although the performance still falls short of matching the best backbone, except for DTD, CALTECH101, EUROSAT and CARS. This trend persists in the LOG- AvG approach, where averaging performance across backbones does not effectively exploit their complementarity between prediction, yielding average delta accuracy of 0.1%. Notably, the LOG-AVG approach shows substantial improvement for the CUB dataset compared to the best backbone. When the backbones are calibrated C-LOG-AVG, the LOG-AvG approach enhances its performances across all datasets except GTSRB, MNIST, RESISC45, and SUN397 when compared to the non-calibrated version. Intriguingly, conventional ensemble techniques such as Vote T-1 and Vote T-3 prove ineffecttvee in providing significant boost to prediction accuracy beyond that of the best backbone.
Table C.I: Our results on ensembling the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques, and also compare to the best-performing single backbone SINGLE--BssH Mean, Max and Min A summarize the difference in performance when we compare against the SINGLE-BEsS backbone across datasets.
CLEVE
JUN397
SuNgue: BEGLE-BEST
86.6
79.4
95.6
75.8
24-4
34.5
63.0
56.4
48.0
23.2
70.1
93.1
52.4
76.6
78.9
63.0
93.8
71.0
64.6
99.4
67.7
23.4
VorE 1-1
86.3
79.7
94.3
14.6
12.7
30.7
68.1
38.0
46.2
34.1
78.4
92.5
$3.9
76.1
70.0
62.7
01.9
71.3
67.3
98.8
70.0
-0.3
-9.0
Vors T:3
M14
31.3
44.5
77.0
33.4
32.7
69
99.4
11.7
35.0
41.0
11.9
76.8
74.0
M8.1
68.1
98
71.8
04
-19
6.3
L.0-Ave
M69
75.0
25.6
69.8
57.3
49.4
15.
72.9
54.8
76.6
74N
62.3
73.6
68.1
98
719
-1.1
C-COSE
68.9
76.2
14.1
14.5
532
65.2
53.4
12.
78.9
02.5
44.3
75.9
57.2
51.
01.5
64.5
96.
67.
432
-11.3
GAC
87.
R1.9
05.4
76.2
24.5
948
7000
38.5
31.0
36.7
79.0
93.8
54.2
16.0
66.7
61.5
94.8
71.6
68.6
98.
71.6
19
-2.
69
SL.
87.6
12.3
95.1
75.1
10.0
3011
20.4
53.6
13.1
16.
31.6
41.
31.
15.1
34T
21.1
U5.3
21.1
13.6
96.
21.4
33
58
17

---

Published as a conference paper at ICLR 2025
Table D.I: Linear classifier accuracy across multiple datasets and employing different backbones, showcasing the performance improvement achieved by the linear classifier compared to the zero-shot version. Also, the combination of ORACLE of linear classifiers. The final three columns present the delta statistics with respect to the zero-shot version of each backbone. In the case of ORACLE, the delta with respect to the best linear probing.
CIRNR 1000
25
50
61.8
77.5
58.5
68.8
61.3
17.9
65.6
M5.81
79.9
90.9
61.7
56.5
70.3
97.4
76.5
85.5
71.3
89.0
97.0
74.8
23.5
0.3
06.8
1064
65.3
71.7
53.8
21.0
76.0
T3.
94.
85.8
861
76.3
97.1
82.3
91.0
73.0
91.0
96.1
78.4
22.6
1.5
67.5
50:54
98.6
101
05.5
80.9
90.9
72.6
61.8
20.3
70L7
60.7
94.5
4..
92.7
84.5
83.5
71.4
91.4
15.5
89.1
66.1
90.7
98.8
77.1
22.8
1.3
68.1
8:32
76.8
SU.8
77.6
59.1
10.9
718
M5.1
43.3
92.6
K3.2
856
73.0
55.0
77.6
87.2
05.9
90.1
95.5
74.6
304
0.2
76.5
L-14-356
97.3
936
97.7
85.4
67.1
35.9
16.3
78.8
96.9
63.9
98.5
94.7
92.4
84.0
98.8
78.7
80.5
95.1
99.3
82.5
18.2
0.3
$1.9
96.2
19.5
96.0
963
99.5
Table D.2: Our results combining the LinearProbe CLIP predictions with different backbones, which we group into non-parametric and parametric techniques. Mean, Max and Min A summarize the difference in performance across datasets.
SONTRII
CLENE
DOVC
SINGLS-BEST
V1.7
85.5
58.9
97.1
63.9
98.5
$4.0
98.8
62.4
95.1
99.8
82.5
VoTE
87.1
71.0
97.5
63.5
95.6
85.0
Vorz 1:3
98.2
91.
07.2
97.0
72.0
37.4
96.5
90.9
97.7
64.0
580
05.0
93.4
83.4
99.2
50.4
94.2
80.5
95.9
99.6
85.0
0.5
-2.0
3.2
Cowe
06-5
86-1
01.1
31.3
57.
241
90.7
71.9
98.8
52.9
94.8
87.9
81.0
60.0
97.8
796
M1.4
71.6
91.2
98-
79.8
-13
-11.8
-0.
A00-A4i
08:0
$9.0
75.
40.5
48.0
66.7
98.6
64.5
80.8
96.1
99
-1.
C-Coap
52.7
Loc--Av
08.3
91.9
97.2
97.2
75.2
38.3
87.5
81.3
97.3
65.2
98.2
85.2
93.4
83.3
99.2
80.6
94.9
$0.5
96.0
99.6
85.0
-1.7
GAC
06.1
92.3
95.2
95.6
73.3
40.7
88.3
82.0
$1.9
65.3
98.6
95.2
94.3
14.8
99.2
82.6
05.1
$1.1
96.3
99.
71.6
-14
-452
48
Wi-
K1.1
19
84.8
96.9
45.0
25.4
98.
79.
96.5
988
85.0
NLC
98.3
91.
95.3
96.8
77.0
40.8
872
794
97.8
96.4
98.6
95.
94.4
84.7
99.3
83.4
80.9
96.3
99.8
85.9
In employing parametric methods, we utilize the entire training set of the target datasets. Our proposed approach NLC demonstrates a noteworthy capability to enhance the performance of the best backbone, achieving substantial improvement of up to 39.1% % in the case of the EUROSAT dataset. On average, our method exhibits a commendable improvement of 9.3% over the SINGLE-BEST backbone's 's performance across the evaluated datasets.
D NLC USING LINEAR CLASSIFIERS OF CLIP BACKBONES
The results of adapting CLIP using linear classifier shown in Table D.I R.500 demonstrates the most significant improvement, averaging an impressive 23.5% increase across all datasets, surpassing even ViT-B-32. However, RN50 still falls short of outperforming other backbones overall. ViT-L-14 and V-T-L-14-336 continue to be the top performers across all datasets, even after linear probing.
A notable observation is the superior performance of the ORACLE of linear probes compared to any individual backbone, highlighting the potential benefits of combining multiple backbones. Although the performance gap between the ORACLE and the best linear probe consistently observed across datasets, it is less pronounced than in the zero-shot scenario, indicating a smaller margin for improvement.
Table E.I: Detailed ImageNet robustness performance. IN is used to abbreviate for ImageNet
IN
IN-V2
IN-A
IN-R
IN- IN-Sketch
Backbone
76.6
70.3
89.0
60.9
NNC (Train on IN)
78.8
72.4
75.6
89.3
61.8
18

---

Published as a conference paper at ICLR 2025
Table F.1: Classification accuracy of models un- der few-shot settings.
Table F.2: Classification accuracy of models on various vision backbones using 16-shots.
2
16
Mindds
50
351
B:31
Tnneegrecbr
(nttationd 2011
49.
17.6
56.3
54.8
43.8
Adapee
CLR-Ada
Shastal H061
65.4
66.2
Tip-Alapte
Ty Argea
(311
663
71.
65.6
NLCC
78.2
78.1
78.2
78.3
78.4
E ROBUSTNESS UNDER DISTRIBUTION SHIFT OF IMGNET-IK
To assess how well the combination backbones performs under varying conditions, we test its robustness using natural distribution shifts in the ImageNet dataset. We evaluate its performance on four datasets representing different distribution shifts: ImageNet-V22 V2 Adversarial Hendrycks et al. b)) ImageNet Rendition Hendrycks et al. 222 Ia) and ImageNet Sketch Wang et (2009). Specifically, we employ a NLC trained on the ZeroShot CLIP backbones using the original IMGNET- 1 data Deng et al. (209). This allows to determine whether the learned combination of backbones can maintain its performance across these distribution shift datasets. In Table E.T observe that the learned combination of backbones, denoted as NLC, enhances performance in out of 4 selected benchmarks, with improvements ranging from 0.3% to 2. %. However, in the case of ImageNet Adversarial, the performance of the combination of backbones appears to suffer, possibly due to more complex decision boundary.
al. (2019). ImageNet
F THE COMBINATION OF NLC WITH TIA-ADA
We integrate the Tip-Adapter Zhang et al. 2021) with our NLC ensembling mechanism and conduct experiments EUROSAT, FGVC, DTD, and PETS datasets. Initially, we apply Tip-Adapter and Tip-Adapter-F independently on each backbone, following their protocol with 1, 2, 4, 8, and 16 shots. Subsequently, we employ our ensembling mechanism to fuse the adapted backbones. For training NLC, we utilize the validation set used by Tip-Adapter to combine their logits with CLIP linearly.
we compare the performance of different few-shot adapters of CLIP on the ImageNet dataset. It shows that current few-shot adapter methods also have performance differences with various backbones; still, our NLC surpassed the best backbone reported by their method.
In Table F.1
Table F.3 F44
and F.6 show the zero-shot performance of the CLIP used by Tip-Adapte It also shows the Adptter and Tip-Adapter-P performance on each backbone and when we combine all the Tip-Adapter versions and backbones with NLCC Across the four datasets used for this experiment, NLC improve the performance of each version of Tip- Adpperr..rr Notably, NLC for BUROSAT obtained an improvement of up to 15% with respect to the best Tip-Adapter backbone (L-14-336) using shot.
19

---

Published as a conference paper at ICLR 2025
Table F.3: Tip-Adapter and Tip-Adapter-F fused Table F.4: Tip-Adapter and Tip-Adapter-F fused with our NLC and applied to EUROSAT dataset
with our NLC and applied to FGVC dataset
ZeroShot
ZeroShot
ResNer
VI
ResNer
WT
50
50cl
50x16
50x64
101
B-32
B-16
L-14
L-14-336
50x4
50x16
50x64
101
B-32
B-16
L-14
L-14-336
Adapter
TiP- Adapter
Shots
Shots 4
Model
8
16
Model
16
50
55.5
60.8
68.1
66.3
70.3
50
19.0
21.4
22.5
25.9
29.6
ResNet
50x4
53.0
56.8
67.0
68.0
71.7
ResNet
50x4
24.1
27.8
29.3
31.5
37.8
50x16
54.5
61.0
62.7
65.1
71.4
50x16
29.0
32.4
35.7
38.5
42.6
50x64
65.1
66.5
70.5
73.5
73.9
50x64
35.0
38.5
41.0
44.8
49.9
101
47.9
49.7
64.7
67.6
101
20.4
24.8
28.3
B-32
54.1
61.7
68.2
70.6
69.8
B-32
21.6
23.9
23.7
27.2
31,6
VIT
B-16
66.9
68.8
75.1
72.9
78.1
B-16
28.0
31.1
32.3
36.7
39,6
L-14
73.0
71.1
78.4
79.2
82.4
L-14
35.6
41.7
44.9
48.9
52.3
L-14-336
73.5
72.9
79.2
80.4
L-14-336
37.7
43.6
46.5
53.8
With SL
77.8
81.5
85.6
86.4
85.4
With SI
39.1
44.2
47.7
51.2
54.8
With NI.C
88.4
86.8
89.7
90.1
90.7
With NLC
40.9
44.5
47.9
51.5
55.0
TiP-Ada Ae-a-datte-F
TiP- Addattttt
Shots
Shots 4
Model
2
8
16
Model
16
60.7
64.4
73.3
77.7
84.9
50
20.5
22.9
26.5
30.4
35.5
ResNet
50x4
59.5
61.9
81.9
84.9
ResNet
26.3
32.8
50x
61.4
68.7
75.9
80.8
83.7
50x16
31.2
36.9
38.3
43.6
49.4
50x64
71.2
69.7
78.7
81.4
86.6
50x64
37.0
40.8
45.0
49.8
54.8
101
62.7
57.7
75.4
78.8
83.6
101
21.7
24.0
26.9
32.0
38.0
B-32
59.4
70.1
79.9
84.9
22.7
27.5
32.9
VIT
B-16
66.6
71.0
79.1
83.8
88.9
B-16
30.2
34.1
36.1
40.9
44.6
L-14
74.9
75.0
86.1
86.4
90.6
L-14
38.6
44.1
48.5
51.9
57.4
L-14-336
72.5
76.4
86.1
85.3
91.0
L-14-336
40.9
45.2
49.6
52.7
59.0
With SL
84.5
86.7
89.0
93.1
46.7
53.4
57.4
62.4
NI.C
89.7
88.3
91.4
91.1
93.8
With NIC
46.8
49.2
53.6
62.6
20

---

Published as a conference paper at ICLR 2025
Table F.5: Tip-Adapter and Tip-Adapter-F fused Table F.6: Tip-Adapter and Tip-Adapter-F fused with our NLC and applied to DTD dataset
with our NLC and applied to PETS dataset
ZeroShot
ZeroShot
ResNer
VIT
ResNer
WT
50
50cl
50x16
50x64
101
B-32
B-16
L-14
L-14-336
50x4
50x16
50x64
101
B-32
B-16
L-14
L-14-336
Adapter
TiP-Adee Adapter
Shots
Shots 4
Model
16
Model
16
50
46.3
49.9
54.3
58.5
61.3
50
86.2
87.2
86.0
86.7
88.8
ResNet
50x4
52.5
56.3
59.8
62.4
65.8
ResNet
50x4
88.7
89.2
89.1
88.9
90.8
50x16
56.9
59.5
61.5
65.3
67.6
50x16
90.5
90.8
91.4
91.7
90.9
50x64
55.6
61.5
63.2
67.3
70.0
50x64
93.7
93.3
93.8
93.8
94.1
101
48.2
51.5
56.6
60.2
63.6
101
86.8
86.6
86.9
87.3
B-32
49.3
51.9
55.3
59.4
63.2
B-32
87.1
88.0
87.6
87.9
88.1
VIT
B-16
51.5
52.6
58.6
62.8
67.0
VIT
B-16
89.3
90.6
90.7
90.7
91.7
L-14
59.6
61.6
64.8
69.7
71.3
L-14
93.8
93.5
94.1
94.2
94.5
L-14-336
59.3
61.8
69.5
72.6
L-14-336
93.8
94.0
94.0
With SL
58.7
64.1
65.5
70.3
71.5
With SI.
93.8
93.6
94.3
94.4
94.5
With NI.C
63.7
66.1
68.4
71.6
73.0
With NLC
94.5
94.2
94.9
95.0
95.3
TiP-Adan Hi--Adppte-F
TiP- Ad-attpter-
Shots
Shots 4
Model
2
8
16
Model
16
48.8
53.5
56.6
61.9
66.5
50
86.6
86.8
87.2
87.5
89.5
ResNet
50x4
53.7
57.2
65.7
70.6
ResNet
89.7
90.7
90.7
50x
58.8
59.2
63.1
67.4
73.5
50x16
92.8
92.0
92.9
92.9
92.9
50x64
59.0
61.8
66.6
69.9
76.0
50x64
94.2
94.1
94.2
94.1
94.3
101
47.5
50.7
58.3
64.8
68.8
101
87.6
88.4
88.7
88.7
90.5
B-32
50.9
55.3
63.8
68.9
89.0
88.9
90.7
VIT
B-16
52.7
56.0
61.0
67.6
71.0
B-16
90.9
91.3
92.0
92.3
92.8
L-14
60.4
60.7
67.1
71.6
75.8
L-14
94.2
94.1
94.6
94.3
95.1
L-14-336
58.9
60.9
67.1
72.4
75.7
L-14-336
94.3
94.3
94.8
94.1
95.0
With SL
62.2
67.3
74.8
78.8
94.5
95.2
95.1
NI.C
67.2
68.7
73.3
75.4
79.1
With NIC
95.2
95.0
95.3
95.5
21

---

Published as a conference paper at ICLR 2025
Table G.I: Ablation of NLC performance when changing the number of samples used train. We use NLC(n) to denote NLC with samples per class. and showcase the improvement in performance compared with SINGLE- BEST backbone in zero-shot setting.
SUNSYT
SINGLE: BuST
86.5
73.4
95.6
758
34.
34.5
630
96.4
18.0
33.2
79.1
98.1
52.4
76.6
75
63.9
M93
71.0
66.6
99.4
67.7
SND44
87.1
81.3
189
154
10.2
96-9
35.8
95.1
15.3
18.1
816
62.
98.1
11.2
-13
73
NNC3
87.1
51.
95.6
16.2
26,9
36.1
70.3
59.1
54.0
37.0
80.3
88.1
54.9
78.3
825
67.
942
72.4
689
99.5
712
23
08
73
97.1
828
95.8
72.
23.
16.3
71.3
604
55.5
37.0
90.4
93.3
15.2
784
81.3
686
94.2
758
896
994
72.
08
9.3
MNC132)
97.3
36.1
604
51.6
78.1
853
724
G TRAINING NLC UNDER LIMITED SAMPLES
In our exploration of the effectiveness of the NLC approach, we extend our analysis to a scenario where we limit the samples combine the zero-shot CLIPs. This experiment allows assess the adaptability and performance of our proposed method under limited training data conditions. Table G.I] presents the performance of NLC by means of limited number of samples. Although the performance of NLC overall improves when it has more data available to combine the backbones, in most cases, just using one sample NLC(1) per class is enough improve its performance. Notably, there is a stop in performance degradation when we use more than 8 samples NLC(8) in all the benchmarks.
Moreover, run five different seeds train NLC on the different few-shots settings. Results show that the standard deviation of our obtained performance stable and low across all our studied datasets, suggesting that approach is not sensitive to the effects of different samples. It varies with the number of samples used with mean standard deviation of 0.62 when we used shot and 0.48 when used 32 shots.
H LEARNED TEMPERATURE VALUES
Figure H.1 and H.2 present the distribution of the temperature values using box plot for the NLC method, normalized by their maximum value. Notably, there is a dominance in the temperature values for the best backbones in each family, i.e., RN50x64 and ViT-L-14 ViT-L-14-336. Particularly prominent in STL10, PETS, DTD, FGVCand FLOWERS, making these backbones especially relevant for NLC. Interestingly, it observed that the most weighted backbone within the ResNet family is not consistently ResNet-101, despite its deeper architecture. This observation is evident in datasets such as PETS, CUB, FGVC and SUN397, where the mean value of the temperature corresponding to ResNet-50 surpasses that of ResNet-101.
Furthermore, the distribution of the temperature values across backbones for datasets such as CLEVR. EUROSAT, and GTSRB is more uniform compared to other datasets. This suggests that the NLC method is effectively leveraging the strengths of each backbone arrive at accurate labels for each sample, resulting in more balanced distribution of weights across different backbones, complement- ing our analysis on Fig.
I DIVERSITY AND ORACLE PERFORMANCE OF DIFFERENT COMBINATIONS
Table I.I shows the Oracle performance by combinaning different options of diversity. Table present the diversity of different source of complementarity.
J CASCADING AND ENSEMBLE
Figure J.3
shows the performance of multiple ensemble and cascading methods of 2 to 9 back- bones, notice that NLC obtains the best performance, and when we add cascade it maintains the computational requirements
22

---

Published as a conference paper at ICLR 2025
(a) CALTECH
(b) CARS
(c) CIFARIO
1.0
1.0
0.4
0.4
0.4
0.2
0.2
0.2
RM101 +
+
VIT--32 HLL
VIT V 40++
0.0
0.0
0.0
RMND
INS
VIT---2
VIT 336.
IM5 Owt
RAS0:1e:
RM550a64
VI-U14
RA50-54
VITB36
VF-14
VITL14
(d) CIFARIO0
(c) CLEVR
(f) COUNTRY211
6.8
0.8
0.8-
0.4
0.4
R150
R05500
AM5011
AA50064
RIL01
VTB36
VIFL-14
VITL
R550
1045004
ANCL
VT-8-32
VI 336
R850
RA50064
RM201
VIFL1
VITL-4333
(g) CUB
(h) DTD
EUROSAT
1.0
1.0
1.0
0.8
0.5
0.8
0.0
0.0
muxe
8150:16
INS0x54
RX2.0
WTF
VITLM 336
NMS0x4
R150-1
RNS0:54
VITe 36
VITL-24
VIT
ho5001e
RNS0x54
VIT 32
36
WIFL114
VT
FGVC
(k) FLOWERS
(I) FOOD
1.8
6.2
0.2
0.2
6.0
0.0
9.0
m900
RA201
Vire-32
MTT--4
DISS
ANSDUR
RA200
VIT--32
VITe
VILL1433
m800
mosod1
InsSovse
RM2.0
ViTe-30
VIB
VITU-1
333
Figure H.l: Alpha values for each dataset using NLC
23

---

Published as a conference paper at ICLR 2025
(a) GTSRB
(b) MNIST
(c) IMGNET-IE KK
0.6
0.5
0.4
0.4
0.4
0.2
0.2
0.2
WTL--44 10
In30x1e
VIT2-32
VTB
RAS004
Rh30036
RN50x54
RAIOL
VITe VIT8-36
VTT-44
V4 336
RA5004
Rec0x366
RA50x54
RH100
VIF144 336
(d) PCAM
(c) PETS
(1) SST2
1.0
6.8
0.8
0.8
0.5
0.6
9.6
H
RNAO
0b50x16
RAI00
VITL14 336
IWS50w
mosodee
VITe 32
VITS 16
WTL-14
INS50v
BNS0x64
RA103
WTL-1
VIT8-
VITe
RESISC4S
(h) STL10
SUN397
1.0
1.0
1.0
0.2
0
6.8
0.0
0.0
IN50x16
IM55004
RM102
VT-8-22
MTB
VITL-44
VTL 230
106044
mosocta
RM101
VIT---22
VIT 336
R050
100004
RN50015
VIT--32
vmr-11
TTLL4433 336
Figure H.2: Alpha values for each dataset using NLC
24

---

Published as a conference paper at ICLR 2025
Table I.1: ORACLE performance by combining: VIT backbones pretrained with same datasets. 2)
Same VIT backbone trained on different datasets 3) Same VIT backbone trained on same dataset but two different epochs.
CASTAON
Avarae
2OVE
2000
SNNST
LAION
93.3
59.7
LAION 400M
76.8
91.6
94.9
97.4
$5.9
39.5
30.1
83.3
70.5
67.5
36.8
81.5
93.4
59.8
90.0
76.6
80.2
96.1
98.3
77.3
98.9
82.1
N1B--1
13.7
90.8
88.9
95.9
10.11
45.7
75.1
00.0
57.2
31.3
79.2
92.8
55.8
77.6
76-7
50.
84.3
72M
76.2
76.1
51.9
72.9
NIB--115
62.9
55.7
54.2
02.0
72.1
39.2
16.5
65.8
51.8
45.4
16.3
70.0
65.1
46-4
67.8
5600
598
56.7
55.6
60.4
97.1
762
M--B-32
578
83.5
75.3
18.8
68.6
15.6
13.9
57.5
53.1
45.1
15.5
66.8
76.8
41.4
60.9
65.3
562
85.9
52.7
523
95.0
65.9
VIT--114
67.2
$8.1
90.0
94.9
78.1
24.6
23.6
74.0
60.5
50.0
25.
76.0
69
50.5
73.3
71.4
50.7
92.2
57.5
67.6
98.1
73.1
Table 1.2: Diversity of combining: 1) ViT backbones pretrained with same datasets. 2) Same VIT
backbone trained on different datasets 3) Same VYT backbone trained on same dataset but two different epochs.
ST1,0
Dutaee Bud
OpenAl
47.2
13.3
47.5
14.3
42.8
70.4
71.5
51.3
50.8
76.7
81.2
32.4
19.3
68.1
34.8
66.7
50.1
15.0
65.8
46.5
3.2
39.7
LAIONASMM e11
45.7
13.6
79.
1X.7
11.6
69.0
71.1
96.0
45.1
61-7
W1.0
20.
24.5
148
516
66.1
16.7
K2.5
47.7
4.9
14.6
LAION400L_e12
43.6
13.6
29.3
13.6
33.5
65.5
72.9
43.7
45.3
02.1
83.1
20.9
24.4
50.9
35.0
53.6
67.9
17.1
$1.8
47.5
4,9
34.5
NIT
497-4-14
21.5
23.3
96.1
94.8
71.3
94.9
21.2
18.9
12.4
52.5
10.1
1.8
27.1
N57-8-16
277
0.6
322
62
2.9
255
1.9
4.1
2.1
2.9
1.3
099
4.1
3.5
0.3
WYF-6-32
1.3
76
0.2
K IMAGE-TEXT RETRIEVAL
We present additional image and text retrieval experiments in Table K.3 that hint at the potential benefits of combining multiple backbones of CLIP. We evaluated the complementarity of CLIP backbones Flickr30k and MSCOCO. We measured the upper-bound improvement of an "oracle selection" of backbones as described in Section 2 Table K.3 shows patterns of complementarity across backbones comparable those in the classification setting. On Flick30k, the upper bound shows possible improvements of 20 percentage points on both text and image retrieval. On MSCOCO, >25pp.
Table K.3: Zero-shot retrieval performance of CLIP backbones on the Flickr30k and MSCOCO benchmark and the performance of our empirical upper-bound, denoted as ORACLE, representing the ideal combination of ResNet (RN), ViT, and All backbones
Text Retrieval
Image Retrieval
Flickr30k
MSCOCO
Flickr30k
MSCOCO
R@ 1
R@5
R@ 10
R@1
R@S
R@ 10
R@1
R@5
R@ 10
R@1 1
R@5
R@ 10
50
80.4
96.1
98.1
48.1
73.9
83.0
59.2
85.9
91.6
28.3
53.0
64.1
84.8
96.6
98.9
52.1
76.6
84.5
64.6
87.3
92.7
32.5
56.7
67.1
ResNet
50x 16
86.2
97.8
99.5
55.3
78.2
86.3
67.4
88.9
93.6
35.2
59.5
69.6
50x64
88.3
98.4
99.6
57.8
80.6
87.7
71.4
91.6
95.5
34.7
60.0
70,0
101
83.8
96.5
98.3
49.8
74.4
82.7
62.5
85.7
91.3
30.2
54,2
65.3
B-32
82.2
95.1
97.5
50.0
75.0
83.2
61.4
86.0
91.7
30.4
54.8
66.1
VIT
85.1
97.3
98.9
51.8
76.8
84.3
65.2
87.8
92.8
32.7
57.8
68.3
L-14
87.0
98.4
99.8
56.1
79.6
86.9
67.8
89.8
94.3
35.3
59.6
70.1
L-14-336
88.1
98.2
99.6
57.5
80.4
87.6
71.5
91.8
95.5
36.1
60.7
70.8
ORACLE ResNet
97.9
99.9
100,0
77.7
93.0
96.3
86.6
96.8
98.4
54,7
76.9
84,9
ORACLE VIT
96.3
99.5
100.0
73.0
90.4
94.6
84.3
96.1
97.9
50,9
74.1
82.5
ORACLE
98.8
83.3
90.3
98.8
61.9
82.1
88.9
25

---

Published as a conference paper at ICLR 2025
Combined Backbones
[BT( + (5{)0) - (BT + (50,00101)
BII(B32,B16,L14 + 10150,101,,00444500
3111143311,,111111333333 + RN (51500001,,0
80%
75%
Average Accuracy
70%
VIT L-T-3-14-33 13336
VIT-L-14
Method
65%
RN50x64
Vote-T1
PRNN 10X66
Log-Ave Avg
60%
ViT-B-16
C-Log-Ave
VIT-B-33
RRN50044
NLC
55%
RN101
Cascade of diverse backbones
RN50
Ensemble of diverse backbones
50%
200
400
600
800
1,430
1,440
Average GFLOPs
Figure J.3: Average accuracy across 21 datasets for zero-shot ResNets and ViTs backbones, LOG- AVG Ensemble, VOTE T-1 Ensemble, C-LoGGAvG Ensemble, NLCC Ensemble, and NI.C Cascade using 2 backhones, plotted against the Average GFL.Oss. Demonstrates that our NLC ensembles surpass the best zero-shot backbone, and the NLC cascade can also surpass the best zero-shot backbone with fewer GFLOPs. Moreover, our method surpasses the standard ensembling techniques.
26

---

Published as a conference paper at ICLR 2025
L OVERLAP DIAGRAMS FOR OTHER DATASETS.
In this section, we present the linear Venn Diagrams for each of the other datasets used in the
experiment section CALTECH (Figure L.4). CARS (Figure L.5). CUB (Figure CIFARIO (Figure
L.6), CIFAR 10 (Figure L.7p, CLEVR (Figure L.8). COUNTRY211 (Figure L.9). CUB (Figure
DTD (Figure
EUROSAT (Figure L.12,, FGVC (Figure L.13). FLOWERS (Figure
FOOD
(Figure L.15) GTSRB (Figure 16] MNIST (Figure .77 PCAM (Figure L. PETS FFigure L919)
RenderedSST2 (Figure L.20), RESISC45 (Figure
L.2 STL10 (Figure L.22p, and SUN397 (Figure
I.23 We can see that in each dataset, the CLIP backbones present possible complementarities that
could be exploited.
M POSSIBLE COMBINATIONS
Tables M.4 M.5
M. 10 and M.11 present the results of possible combinations of backbones using the non- non-parametric and parametric approaches proposed in the paper. Notably, the performance of NLC consistently emerges as the best across various backbone combinations and datasets when compared to other methods.
M.6 M.7
M.8 M.9
Notably, instances exist where the combination specific backbones yields a more substantial performance boost than utilizing all backbones together. For instance, in the PETS dataset, combining ResNet 50, 101, and ViT-B-32 results in a delta improvement of 2.37%, surpassing the 0.99% improvement achieved by using the five backbones selected for this experiment. This phenomenon is consistent across datasets with different backbone combinations. In CARS, there is a boost of 5.71% when combining ResNet-101 and ViT-B-32, compared the 2.55% boost when using all five different backbones, While the best delta improvement among backbones may not necessarily come from combining all backbones, the best overall accuracy is consistently obtained when using the combination of all backbones.
27

---

Published as a conference paper at ICLR 2025
CALTEC1111
RN50
1351
RNTOL
RN3004
1223
RN30664
1455
RN30816
1477
1917
VIT-B-32
1452
VIT-L-14
1491
Sets
ViT-I,-14-38 14-33
1493
16
1503
1591
ResNet
1565
Sels
VII
1594
Figure L.4: CALTECHIO1 011 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
CarS
RN50
A961
RNTOI
RN500l
5378
RN30x16
5880
RN30864
6102
3830
315
VIT-B-32
4803
5195
Setas
VIT L 14
8252
VIT-L-11-336
6388
900
100
7238
VIT
7228
Sete
Res
7300
S2U
4:3
Figure I.5: Overlap diagrams for the CARS dataset with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows colour backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
28

---

Published as a conference paper at ICLR 2025
CIFARIO
RN50
7155
RNN004
RN101
RN50s16 6
8131
RN30864
8511
5451
152 151
TOX 116
VIT-B-32
8983
V6T-B-16
9181
Sets
SHT--I--14-38
9400
VIT-L114
9557
106
55
98.96
ResNet
Sets
VII
98406
Figure L.6: CIFARIO 10 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
CIFAR 100
5
RNN0
RUN0044
4613
RN1OL
4763
RN30x16
5212
RNS0864
5088
VIT-B-32
6:22
B
6091
Setas
VIT L. 336
7438
VIT-L-14
7582
118
319
700
6712
ResNet
Seto
VIT
S712
2015
353
Figure L.7: CIFAR 100 Venn diagrams with the correct prediction of each backbone. The Top part of the Venn diagram shows the number of backbones that are predicting correctly a set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row the diagram shows in colour the backbone that correctly predicts certain set image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Venn diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
29

---

Published as a conference paper at ICLR 2025
CLEVER
444
RNSDS
2036
RNSO
RNGO
RN50864
3395
RN1O1
3664
0016 381
1978
9056
STL
0541
1337
79
715
515
1373
VIT-L-14
2915
V4T-1-14-336
3UIT
Sets
V6T-B-16
3181
V-T-B-22
3480
38
138
173 305 450 000
5157
VIT
5157
Sets
Nes
8436
1550
Figure L.8: CLEVER Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
COUNTRY
RN50
3253
RNTOI
RN500c
4206
RN30x16
51:55
RNS0864
0285
VIT-B-32
3632
1
Setas
VIT L 14
6723
336
7271
312
243
9278
ResNet
8688
Seto
VIT
9278
2233
Figure L.9: COUNTRY211 1 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows colour backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
30

---

Published as a conference paper at ICLR 2025
CUB
5
RN50
2635
RN1O1
RNS004
RN30816 6
3355
RNG0661
3635
1A77
153
215
VIT-B-32
3U77
V6T-B-16
3203
Sets
VIT-I-14
3198
14.
3652
2055
155
72 140
106
350
4573
VIT
1573
Sets
ResNet
4780
Figure 1.10: CUB Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows number backbones that are predicting correctly a set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images certain set. The right part the total amount of correctly predicted images per backbone.
DTD
3
RN50
775
RNTOI
RN500l
913
RN30864
998
RN308T6 6
10001
VIT-B-32
827
B
8:8
Setas
VIT L 14
1040
L 338
1060
1961
VIT
1264
Sete
ResNet
1382
207
Figure L.11: DTD Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row the diagram shows in colour the backbone that correctly predicts certain set image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images certain set. The right part the total amount of correctly predicted images per backbone.
31

---

Published as a conference paper at ICLR 2025
EUROSAT
RNTOI
1824
1981
1521
RNGO0T6
2224
RN
2500
176
63.31 T5M
TM
153
ViT-B-12
2015
V1T-B-16
2381
Sels
VIT-L-L4-336
2431
VIT-L-14
2548
796
21 89
425
635
65: 20
486
530
[00
12.
305
E29
3868
VIT
Sets
ResNet
$:29
Figure L. 12: EUROSAT Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number backbones that are predicting correctly set of images. Each column represents set of image instances that predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
FGVC
RN50
569
RN508
700
RN G0xLG
918
RN RNG0
1033
15
1744
VIT-B-32
05
ViT-B-16
Setos
VIT 14
1057
L 336
1106
938
1729
VIT
1729
Sets
ResNet
1743
100
Figure L.13: FGVC Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images a certain set. The right part total amount of correctly predicted images per backbone.
32

---

Published as a conference paper at ICLR 2025
FLOWERS
RNTOI
RN50
RN
2301
RNS0K16
4:29
RN30664
2077
ViT-B-12
4088
ViT-B-16
4392
Seta
VIT-L-14-336
4812
VIT-L-14
1861
5009
VIT
X269
Sets
ResNet
Figure 1.14: FLOWERS Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backhone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images a certain set. The right part the total amount of correctly predicted images per backbone.
FOOD
1967
RNTOL
20669
Sete
R.5504
21577
HN308T6 6
22638
RN30861
23009
V-T-B-12
20892
ViT-B-16
Seta
VIT-L-14
23312
VIT-L-11-336
23506
24265
ResNet
21268
Sets
VIT
414
24742
Figure 15: FOOD Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images a certain set. The right part total amount of correctly predicted images per backbone.
33

---

Published as a conference paper at ICLR 2025
GTSRB
RNGO
REN0044
RNTOI
4742
RNGOL1G
5039
RN30664
0150
ViT-B-32
ViT-B-10
Seli
VIT-L-14
6385
VIT-L-114-36
60211
5467
ResNet
8240
Sels
VIT
Figure L.16: GTSRB Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backhone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part the total amount of correctly predicted images per backbone.
MNIST
RNIOL
4803
RN50
4907
RN5004
5326
RNG0XL6
0271
RNG0661
7894
2531
V-T-B-12
3809
ViT-B-16
Ma4h
Seta
VIT-L-14
7331
L 338
7522
230
8500
VIT
8558
Sels
ResNet
Figure L.17: MNIST Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number images a certain set. The right part total amount of correctly predicted images per backbone.
34

---

Published as a conference paper at ICLR 2025
PCAM
RNG0661
7000
REN5044
18585
RNTOI
1904
RNGOX1G
20977
RINO
20922
412122
ViT-B-16
10014
VIT-L-1d
170/0
ViT-I-114-30
19885
V.T-B.22
20402
1784
2100
100
4779
1365
1628
1263
9901
883
5027
5:8
SAOW
$1369
ResNet
28505
VIT
31269
120
Figure 18: PCAM Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Fach column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
PETS
RN50
3145
RNTOI
RN50al
3360
RN30x16
3304
RNG0861
3456
VIT-B-32
3208
ViT-B-16
3268
Senos
VIT L 14
3131
L 338
3-12
156
3078
VIT
3578
Sets
ResNet
Figure 19: PETS Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row the diagram shows in colour the backbone that correctly predicts certain of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows number of images certain set. The right part total amount of correctly predicted images per backbone.
35

---

Published as a conference paper at ICLR 2025
RENDERSST2
5
RN50
183
RN1O1
RNS004
1223
RN50816 6
1250
RN30861
1203
5B6
710
VIT-B-32
1067
V6T-B-16
1102
Sets
VIT-I-14
1255
14-366
1285
515
174 161 172
115
1051
VIT
1681
Sets
ResNet
1736
111
Figure L.20: RENDERSST2 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
RESISC45
RNN0
RUN5044
RN1OL
3332
RN30x16
3749
RN30864
4125
ViT B.32
3:12
B
3073
Setas
VIT L 336
4012
VIT-L-14
4069
116
200
4953
VIT
1956
Seta
Nes
5187
Figure L.21: RESISC45 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that predicted correctly by some group of backbones. Each row in the diagram shows colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
36

---

Published as a conference paper at ICLR 2025
STLI0
RNGO
7535
RNS004
RN101
7744
RNG001G
7821
RN50864
7863
ViT-B-32
7771
ViT-B-16
7860
Sett
VIT-I-14
7949
V7T-I-114338
7054
7077
ResNet
Sets
VIT
7973
Figure L.22: STL10 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number backbones that are predicting correctly a set of images. Fach column represents set of image instances that are predicted correctly by some group of backbones. Each row the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
SUN397
RNJOL
12438
RN50
RN500l
2991
RN30x16
13826
RN30864
14550
VIT-B-32
13330
13851
Setas
VIT L
14511
VIT-L-14-336
14732
103
105
17512
VIT
17542
Sete
ResNet
Figure L.23: SUN397 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly set of images. Each column represents set of image instances that predicted correctly by some group of backbones. Each row in the diagram shows colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
37

---

Published as a conference paper at ICLR 2025
Table M.4: Our results on PETS dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non con-paraettiic and parametric techniques. Also, the best-performing single backbone (SINGLE-BEST) and the ORACLE performance. We present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when we compare against the SINGLE- BETT backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
PETS
SUNGLE-BCTT
90
101 B-33 Bi16 1.14
Vor T:1
Vorw T:3
Coow
Tve-Avn
C-Dra
GAC
X1C
E5.00 -0.00-0.0
16.85
-0.0
8745--11
15-0000
5047
0.00
93.59
00.00
10.16
18.30 131
114
N7.166227
18:09
1:25
$1.95001
$7.46
58.50 104
80.00
85.15 221
19400
05,90052
59.21 1.54
8948-212
19.45
L30
92643.18
20:33.000
90.00.84
89.07
09.83..70
0..48
09:720000
860.70
90.27
0409
04.00041
94.00041
03.810.2
93.33014
03.880.00
08736014
2.00
0.27
0.27
95.24 0.35
95.27
0.16
93700.11
90.100.11
2.00
80.07
80.560070
90.16 1.00
67.35 -1.72
50.41
1.34
59.45
90.38 1.31
90.27 1.2
50.39 1.25
$3.85 4.5
91.99
95796619
95766916
99.74
43.22
91.0 016
91736114
013801
00.09
03392.33
94.06-1.66
09.940235
98736114
55.92
96.19399
180 1:20
14.93
8M0722226
19:45
1M64 2012
94,092222
00.290222
50.51074
90.02.0.05
0.79
90.08
1.04
50.35
94.74..07
94.29-122
02.30-1100
V5L08 0..0
02.80
40.00
95.02 41.5
$2.95
05.19-341
40.27
83.59
02.97 -063
03.703.11
90.95
0.38
03.87.0.77
93.34..25
03.640225
94.99.
94.47.57
96073.38 3338
07
90.15
(00
$6.54 03.53
50.65 1.55
59
90:41 1:54
90.35
1:78
50.1 1:66
5512600
M1.99
12.20-1.0
V0.00 00.57
12.35
1.14
59.75
4134
92.70 409
92.94-1.55
92.91 -0:5
V5.46 -0.14
V5.46
93-43
41.16
43.37
40.0
03-19-111
-.11
01.90311
03.59
2063
-0.30
12.89
-0.71
95.51
40.00
93.13
-0.4
95.40-0.19
04.63
04.13
92.50
90.91 0.00
92.64
1.01
00.91 0005
91.99 -0.0
99.00 x1.11
93.51 41.05
91.33.99
61.77
83.59
12:91 2008
08.27 00.33
02.78 40.02
95.19 40.41
93.05 -0.55
93.21 -0.33
93.796.19
0..33
91.823.02
83.59
92:61 0.98
$3.00 0..
90.13
-1.47
92:94
0.065
52.91 0..68
93.05 -0:55
03.08
94.03 0144
07.79 4.
03.59
91.63 1..0
05.00 0,50
90.26 14
92:85 0.744
$2.89 .0.71
$2.94 0006
05.30 0,20
94.55
94.55-0.0
$5.05
4.47
2135
Max &
1.39
1,51
1.34
2.20
L10
1.90
2
3.30
0.22
-3.03
0.74
7.99
.034
.095
-074
7.00
38

---

Published as a conference paper at ICLR 2025
Table M.5: Our results on CARS dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones. which we group intro non-paran non-parametric and parametric techniques. and the ORACLE performance. We
Also, the best-performing single backbone (SINGLE-BES) present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when we compare it against the SINGLE- EEST backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
CARS
Koshit
WI
SINor-Baa Bber
Nee- Nee-Punneettc
Parsutt::
VotE
VOTE
C-LoG-VIG
52.33 -0.00
M132-000
0.00
02.41 256
02.67
53.41
6316
01:11
52.51 1/1
61.82 2.4
61:92.66
71.38
59.73
61.30151
62.26
6084 1:31
63,993366
61.14 1.41
02.52 2.79
63.43
3.68
63,40
71.50 11.76
64.11
06.23.006
65.38 005
53.94-1000 00.67
5600 1.41
6400 0.36
65.89 1.25
66.35 1.94
56.901.19
7338 9:34
77.25
77.58 0.17
71.26.01
77.09 0.06
17.24 0.51
77.47 0.29
77.64-0111
78.51
78.57
0.92
$3.96
644233
64.993.07
63 2:30
06.89
69.71
63.064544
66685-73
73,60
1.334173
0.:942.31
00.41
67799
3.35
75.08
23.35
71.55 023
73.86 0.11
77.55 022
11.42 4.34
77.52
0.24
71.60:005
0.96
74.96 1.11
$3.88
73.38
71.34-021
17.45 0.35
71.02
0144
0.14
77.18
4039
72.83-0112
78.88 1.15
74.00 1.16
84.16-655
6112
64:63 3.51
03.33 4.1
51.36-3.71
3.37
06.565443
63.00 2.59
66.01
4.80
66.76.633
66.703663
12.76
64.1
66:34
67.35
59.66
60.0030
2.04
67.77
67.85
6..20
50.66 16.00
76.33
16.99-1.16
66:24
67.06
56.24
07.943333
06.4633.0 56
77.50
-1.0
77.58
4041
77.34-0.11
0096
77.05
76.51-55
76.26
1.47
75.90-1150
1:55
76:76-4199
16.76
41.99
76.90
0078
76991-331
75:40-0005
74:01 1.76
87.139644
64.1
67.86326
66.95
4.34
0140042
19.364.15
2.44
04.1451
66.8-4.55
59.34 99.344444
16.76
15:04-2:11
1098-00
77.17-4139
18.89 4187
72.16 -037
72:14-011
79:24 0V
19.26
80.32
15.0-2215
75.00
0.5
70.37
19.02
1.31
67.725.11
69.543.93
60.99
06.77
08.56.305
94.15
73.55
73.81 3304
75:97-1.88
75.97 -1.78
76604 -1.72
77.08
0067
76.4-1132
78.56-051
79.12 1.37
89.37 11/22
76.9-1006
75:06-144
1.09
75.06
-3.05
76:41-11.4
76:41
-1.34
96.95 -0.07
76.47-1.5
79:01 53.
74.22 1.17
89400 15
77.25
145113325
7588-113
75.90
76.57-1.188
76.57
-1.18
77.0
00.75
76.05 -1.7
76.83 06
79.10
89.30
70.38
75:28 -2.47
75.08
-1:31
75.04-2.31
75.54
16663-1.14
4006
7691-0030 -030
79401 0.30
794359 7.425-1.55
1M-49
Meas 3
-041
0.63
2..45
1.M4
0.40
0.98
2.25
2.43
11.35
Ma.
3.11
2.29
5.77
2.99
4.89
5.11
39

---

Published as a conference paper at ICLR 2025
Table M.6: Our results on CUB dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (SINGLE-BEST) and the ORACLE performance. We present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when compare against the ŞINGL.F- FEST backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
CUB
ResNer
SIVOL P- BEST
New-
Psonan:
GRACLE
8:16 1-24
Vortl
Loo-ArG
C-LONF
C-Los-Ar
55.25
0.00
62.06
0.00
19.M
51.28
52.39.2.9
55.00
50.22
53.66
53
61:11110
$2.99
54.02
$8.00
53.72
53,00001
$6.13.52
57.68
11.65
55.38
56.01
57351
03.82.0.4
55.99031
58,44
01.01-1066
55,49.25
4.25
57.61
5404
57.42
1.55
56.77
59.05
65 10.06
0005
064
02:15
(1.01
61.85
61.7
03899
03355 1.30
55.28
57.573.99
56.5833
51.05
4.23
5.,97.559
57.40
59.634335
60 72 5.44
60.74
11.50
63.06
02.160051
61.70 1.71
67.62
0.55
64.547775
63.17.076
61172.0
61 2.53
62.06
02:63
63.36
1.29
54,87-73
64.90
02.29.033
63891183
06.07 30
0:300:33
71189911
$2.99
16.25
59.21
47.95
99.32
$4.00
14.001.2
58.1852
59.25
6.21
69.71
14/2
55.38
57.343.05
3.14
53.31
59.424.44
60,5553
60.65.54
70.00
15.0
01:55-001
60:00-176
05411335
61.96
06.36
64.3623
75,85
636715
64.19
55 6.61
65.77
619622
55.18
59.97 3,09
50.92 4.64
00.83 5:445
61.24
51.73.445
00.93 5.4
61.555
6..656.77
71.99 16:11
62.06
02
150.09
03.44 1:38
60
1.81
65.00
3:95
61.82 01.4
63.56
6533633
65.313324
75:42 13.76
82.06
824450005
64105 1.51
55.55
66.90
65.33
5:26
61112106
66.33 4.15
06.15
3.00
79.54
1544
63.06
63112101
54.06334
91.05-7.61
03:77071
64399773
06
06.05
1.29
76.00 11.5
55.38
59.444.1
(6.61..33
5:33
49.57 -5.71
61.81
57,90221
61.396111
61 6.09
7.21
75.00 75.0031111
02.00
0.60
64.10.04
78.30 16:36
54.75 7.334
62.20
7966713
60.06
64.35 2:31
54 7.889
65.09
63.35
06.21.114
69
79.01 16.95
82.66
03.32
64.26 2:11
53.71 -6.15
65.83
61:2
85.01230
6.14
31.21
A
2.43
-1.14
111
13.00
Ma 3
.081
0.36
4.35
1.52
-1.36
6.71
1.56
6.25
40

---

Published as a conference paper at ICLR 2025
Table M.7: Our results on DTD dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones. which we group intro non-parametric and parametric techniques. Also, the bert-performing single backbone (SINGLE-BEE -BETE) and the ORACLE performance. We present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when we compare it against the SINGLE- BEST backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
DTD
RoiNei
VII
Soec
Nas-hhnmam::
Paranetnc
8:32
Coso
NLC
41.22 -0.0
40.20
46.44
4121 6:04
47.14:107
(TA6
3.94
407
54.36 11.9
43.00
44.33 0.64
45000
41.22 3.71
4601 2.02
43.75 4.21
45.43 1.44
45.74
1.76
45.00 101
5L.037L
45.11
46.85 1776
47.15
2.02
40.57 4.33
11:992:29
45.85 0.34
47.15.211
47.87
2.71
41
1.29
53.241114
55.32
03.640333
56.20.8
41.33
13.99
55.64033
35.05 0.27
56.22
02.23600
47.54
4.445..41
4.11
47.333.4
47.98 3.0
55.01
55.01
1645
43.28
2.61
0:77
07.332.15
1.91
4740.25
55.32
96.55
1.01
03.56-11
76
96 23.35300 0M
51.53
11.32-00 -00
56.79 03
16/15
1:17
96
1:38
62:00.
58.32
52.0 1.7
50.50
2007
43.85
11.44
35.29 1036
36.40 1.26
56.161444
56.81
1.40
56.90
1.54
12/02
43.99
3.51
40:71
3:72
40:37
46.90
2365
42
09:26
301.11
56:26
45:35
40.41
44.08 0.43
4.14
3.19
5:21
30.21
60.99
44.95
35.993.09
47.342255
47.77
57.81
55.32
53,19
2.13
$3.43
41.56 13.40
54.79 0.59
53.11 4.21
54.73 0,50
56.06 0.74
36.40
36.49...3
1.17
10.37
55.32
53.90
1.35
96.35
1:01
07.4-117.55
15.01
0.69
05300-11
0:18
16.00 758
51.34
57.15
1:96
45.11
45.40
1.3
43.06.433
41.76
5.35
49.25
4.15
48.60
3.51
40035
4.64
40.20-11
ML31 15:1
55.32
14-07-1155
55.000077
44.37-00110
59.27
4009
59.20
-011
10000.00
16.90
1.34
57.06
1.90
69.20
54.99
$6.76
43.56-11..6
05.24
56.22
56.31
56.00
37.90
66.37
45.673356
49.50
4.53
3.35
09.41
35.32
54.36 00.56
55.16
42.50-12.52
54.73 0.599
53,00 0.32
$4.95 0.57
36.22 02
66.03
55.32
51.00 03.45
94.38
1.05
07:03-17
$5.00 01.13
05:53.71
96.11 00
$7.07 3.55
77.71..94
55.33
59.72 -1.6
35.96
42.50-12.80
$4.57 00.74
48.4
55.4800100 0.10
$0.38
56.97 1.65
67.71 12:59
55.32
54,88-0 41.64
11.13.1
44.47-00330
15:07
16.81 1.49
2.18
67.86 1233
Men 5
0.76
1.75
-7.65
1.61
1.00
1.87
2.36
2.63
11.00
Mix
3.56
3.19
3.40
3.51
3.11
41

---

Published as a conference paper at ICLR 2025
Table M.8: Our results on FGVC dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (SINGLE-BES)) present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when compare it against the SINGLE- BEST backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
and the ORACLE performance. We
FGVC
VII
SUNGLE-8155 BEST
Nee-Punmon
OBACLE
17.07 -0:30
18.87
19.02..30
19.33
19,60042
0..59..06
19
7.519
19.65
28.52 0.51
20.46.31
1662
-300
21.361.71
19:00
21.18.53
21,00 135
22 2.52
27.9
56.39
34.30
0.11
51.33 0.0
17.19 7.5
31.75.0 99
71.97 0.17
2535088
25.65 1:76
75
25.8915
31 661
91.71
31:47
0.24
31.47
31.47.0.34
31.47
0.34
31.66 0.03
31.99 0.12
30.550.84
00.45.078
53.57 1.86
37.59
19.00
21/00
21.11100
30.00
30/21108
22.0.330
21.31
23.41
24:311.5
51.20
34.18 0031
24.30
0.69
33.20
25.02.005
1.50
32.753.34
51.71
31.96 0.15
31.68
0.08
31.39 4..2
32.00 0.33
31.71
.00
32.79100 108
32
33.55 1,02
31.57 136
51.71
11.42
0009
52.1910.85
31.05 01.24
32.
1.15
31.92
32.881112
58.30
1.59
33.60 1.75
41.25 4.54
19:65
30.79 1:14
21.51 1.96
18.12-1.53
21.75
20:52 0.81
21.00.00
21.72
23.00 3.34
33,60 13.95
22.00-1.5
23.65
24.03 0.36
3.22
36.57 12.18
23.94
34.570.18
24.81
24.99.6
25.70
2.37
37.68 13:29
31.71
30,36
1135
31,29
-142
3003
0.78
32.07 0.36
31.20 0.12
32733102
00.25..54
34.74 3.03
43,53 11:22
31.71
30.63
1.08
11.17.016
10:77
11:11
17:00::00
11:00:009
19.951.66
33.06
135
34.00 1.08
15.06 13.55
24.39
23.79.00
34.15.03
20.35 0.64
25.00
0.63
23.00
0:59
3432033
25.90
26.31
1.92
13.7
51.71
10:20-1.00
14100-031
78000-4833
31.02
31.82-009
31:00
02.000.00
30.09
33.72 2011
4400
2337
22.99
24.35 0.55
21.31-1.0
24.00221
23.90
3.16
91.71
28,65
3:05
30.63 1.088
30:45
1.36
31.20 0.51
31.14 0057
32.040.33
52.37 0.66
34.30
3.09
47.04
91.71
29.46
2:25
34.12
0.00
29:35
-1.6
31.00 4.99
31.39 4033
32.30 0.6
$2.55 0.54
34.00 3.21
48.54 10.33
91.71
30.06
31.04
0.22
31,02
0.5077
0.06
33.21 1.5
35.63 90
35.37 3.66
40.38
31.11
29,02
31.30
30.51
32:56-0077
32.500.5
33.33
2.32
40.83 12.12
Mein 5
40.35
-0.5
1..64
0L:33
41.27
0.34
1.06
2.43
$3.01
2.10
2:27
4.17
20077
42

---

Published as a conference paper at ICLR 2025
Table M.9: Our results FOOD dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non- non-parametric and parametric techniques. Also, the best-performing single backbone (SINGLE-BEST) and the ORACLE performance. We present, for each combination of backhones, the improvement constancy and deterioration of accuracy performance for each method when we compare against the SINGLE-BEsT backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
FOOD
Honhee
SONGLE-BUS Best
Nan- Sur-Peranen
Prnnen:
GRACLE
108
Vore
Vors
Come
Loo
C-Cow
C-Lo0-Ave
NLC
22.M1
8791
0.0
V2322
-1000
81.86
92.63 0.78
$2.00
1.04
52.36004
13.08 1.22
$2.47 0.61
83.14 1,20
83.29
83.27 1.41
57,1653
89885
61.00
76.96
61.55
$3.00 100
80.001100
86.11
N7.50 4116
89.82
$1,90
$1.35 40.35
81.2 044
89.80-0.11
19.15
064
91.11
9490
16117076
87.87
04.34...3
02.25
12.59
92314
41.99
92315
41.34
91.85 41.48
6.11
12.00
14.11
87.91
35.08 0.16
98.18..21
82.55
5.36
88.25
0.33
05.00.0.1
88.31 0.4
58.70
0:85
58.74 0:83
91.54
00.11-07
09.71
15:15
07.00
41:15
03.01
-0.7%
91
99.14-500
V0.70
0/17
00.550411
94 3:41
92.52
92.08 1125
92.60
0134
92.45
1.:2
12.59
11:4
92.31 4100
02.990227
92.00
0.35
92.88
94.96 2.64
$4.56 1:0h
16.28
79.64
$3.99
$462
55:25
80.002
10.84
1.26
87.57
87.72 0.19
67.51 004
$7.69.0.28
0.20
59.73000
02.90
18.000.11
90.00
91.18
91.54
92,53.55
92.00
93.11
43.96
92.30-6.33
0.99
87.91
87.
00:13
88
0:27
$3.05
15.00
07.950005
18.37
19.07 111
50.95 1:01
93 5:45
M2332
M6.17
22:15
01.23 1.33
01.75
11.7
01.30
4195
91.54
41:76
V1.28 11315
M2M55100
M2.5 2001
00.30
91.54
01.08
02.06
92,07
91.87
0..45
92.02 0.3
92.93 0.6
92.90
0.65
90:99 3.56
00.73
41.01-4330
91.97-01
97.95
91.90 0.00
49.29 1:13
87.60
$7.00
$1.00 0.99
08.
$7.00
4125
18.100.19
19.00
1115
1.13
14.00
91.24
0.51
08
41.73 01.99
6.51
$7.91006
93.00
91.26 -1.6
91.50
(1.75
90.00 -1.22
V1.82-00
93.87 41.65
91.45 -0355
12.98
10.10 0075
16.35
92.32
00.08 :2255
90.80
1145
59.62 2.7
91.09 -1.24
91.43 0:
90.91 -1.41
02.01 0.59
03.07 0:75
06.00
Mess
M
-3477
-2.13
-785
-1.71
-1
-5.08
D-11
7.19
43

---

Published as a conference paper at ICLR 2025
Table M. Our results on FlOWERRS dataset for all the possible combinations of combining the zero- shot predictions of CLIP backhones, which group intro non-parametric and parametric techniques. Also, the best-performing single backbone (SINGLE-BEST) and the ORACLE performance. We present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when we compare it against the SINGLE-Bi B backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
FLOWERS
ResNer
VIT
SINGLD-BEST
Pataretic
ORACLE
D-33 B-16 1-34
Loe-Aoo
C-Lex--Avy
GAC
NLC
66.12 -100
65.20
0.00
-1130
-0.0
-0.00
06.12
6501
57395
57.30
57.25
28.422311
08.912.35
66
66:48
09:17:75
60.77 5:79
08:66
09
177
16.76775
66.77 5.79
09:78 75
70100:51
75/99
71.43
72445 1/12
12.68
1.35
72.08 0.6
72.48 1.0
11.89
71,09 1.21
72.42 0.99
73.12 1,09
166333
25.05
78.29 0.76
76.22 0.53
75.02
0.09
77.33 1.72
76.79
0.28
18.19 0.86
18.91 0.15
4.227.211
08.56-2.08
66.742266
68.35
68.32
68.04
74.3920
93.37.094
79.05
77.67
77.34-1.32
78.34
17:87-1.19
37.52
-123
79.25.0.18
71.43
72:91 1.49
12.69
1.27
05.15
72
12.16
12.93 1.0
72.04 1.31
79.88
76.296.30
74.05
78.50
0:95
76.31
43.55
73015
-0.6
77:41-165
78.35 007
-117
78.94 41:11
79
87.30 1.81
79345
18.00
-104
76001
-1.04
78.45 41.6
72.90-1.15
18:37 41.6
78.29 40.6
76.14 40.9
39,86 0.6
$1.34431
66.48
09.98 3.5
69.122.63
09.57
68.12.2.3
70,09 3.61
69.64 3.35
70.90.444
76.51 12:6
71.43
71.740311
71.78
72.48
71.49:0.07
1.09
73.00
70,46500
76:30-4.4
71.43
13.20
66.29
72.74
72.55
72.90 1.5
72.97
73.90.247
74115
75.87
76.89
76.06
76
18:24
1.12
74.55
76:84
-7.31
37:43
-1.61
78:08
0..5
77 33-3377
74.35 007
77-41-661
78.09
30135
1.79
$181177
71.43
71.890.26
12.35
1.12
06:34
5,.09
12:090.00
12:43 1.00
7282 3/4
71.24 01.3
75.83-24
9.34
7418
75.13
78:39
79.22
16 19:300
79.00-1.0 -1.38
76.45-2.6
$4.10
76.34
37.13
78,05-1
17.67-1.38
71.43
12:95
72:00
72.20
73.001.55
82.14
25.22 1389
75.69
77
77.89
75.52 0355
74.5
76.01
3.01
76533-944
74:00-1:06
76.36 -2.6
77:57-1.05
76.55-355
78.7% (1.1
30375 1.70
80.67 60.67660
7938
78.71 -2.34
39.22
-1.44
76.21 -2.55
26.31
-2755
18.04 41.9
72.8 -231
75.15 41.99
301 5.12
53777572
77.09
79:201
76
77.69-33
77.00 77.00-1.95
78.19
794900 03-00333
6575
75.70
76.25
75.54.2
16.25 2.00
Meas
4080
-0.27
-1.24
40.35
01300
1,57
6.76
Max 5
2.95
3.50
2.63
3.29
2.63
3.61
3.15
4.44
11/04
44

---

Published as a conference paper at ICLR 2025
Table M. Our results on IMGNET-IK k dataset for all the possible combinations of combining the zero- shot predictions of CLIP backbones, which we group intro con-parametric and parametric techniques. Also, the best-performing single backbone present, for each combination of backbones, the improvement, constancy and deterioration of accuracy performance for each method when we compare it against the SINGLE- ESS backbone. Mean, Max, and Min A summarize the difference in performance across methods and backbone combinations.
LANGLE-BESTE and the ORACLE performance. We
IMGNET-IK
RoiNel
VII
Soec BOST
Nes-itnmam:
Parmnttnc
8:32
Coso
NLC
-0:30
-1:30
0.00
00.00
02.30
63.26 1.46
64.16
59115
54612.11
61.22
64472217
M471 2:41
66.16
70.88
63.36
65.00 1.65
65.57 3.01
$4.40 1:34
65.78 2-42
64.39 1/3
6..54-2.1
65.86 251
06
06.07.71
7L768AL
08.34
10:47
0332
68.15
01.99
08:89
05.00 0.26
68.54.511
56000
1.35
09.20 1.22
7.51
25.55
75.15 004
75.27
0.27
60.25 15.3
75.09 0.45
74.51 1.02
14.57 0.50
76.17.66 75.970.4
75.97
0.64
90.14.6
66003
66441.00
60.20 3.07
3.63
3000
66770335
67,003.65
67.35300
73.09966
16,390000
18.901130
63.17
1000113
25.55
15.37 40.6
75.40 0.13
68.53
12.01
75.19 0.35
74.79 0.34
15.20
75.98
0.45
76331078
076
40.49496
15.55
15:47 41.6
73.01
0005
65.42
9.03
73.65 0.1
15.04 415
15.50
13.30-010
15.98
1144
76
0.73
90/01305
36
261157799
66.76 3.4
60.27
3.08
67.33 3:57
05:1 05511211
677063372
67.47
4.11
67.57.
13.3913.03
66.22 0.12
68.94
66:92 1.53
69:23.0.99
0.18
00.56..33
TOLEL
2.37
73.5996
70:08-1:44
04.58.034
63.72
12.90
74.35
12.235
74:36
74.24-13
15.32-1.21
7634
76.
75.53
13.95-1.55
1.15
74.95
(661
03.53 9:31
71.96-03
26.56-094
0095
7140--065
76.55
1:01
83.00173
08.34
66:13 (1:1
69.75
62.46
5:81
10012
1.58
05.66
04.5
2.043:2000
2.33
78.39 1014
75.55
11.12-2.31
24.11
0093
63.08
-1.00
74.63-00 -01
04.27
-122
10.18-0000
14.04
14:13:14
60.30 5.14
75.11 0042
75.14.0.0
16.09
76:73 1.2
83,00135
69.00.075
69.51
62.51
08.45
10.52
71.00:3 2.75
25.55
12.37 -3.16
73.71
-188
08.11
12.44
73.67-1.56
-1.6
-1.
73:62 -1.02
0..12.55
76 1.16
$4.88
75.53
13.07-2.06
74.06-1100
1.07
K7.07
71:06-1.00
74.21
-3.37
70.41-1000 17
65.65 1.00
54.77.64
25.53
73.54 0::0
24.00-1.13
67.56 67.56-77.95 7.895
74.39-1.31
74.39
-1.14
74.38
11.11
T4.39.1.1
7..14
76
1.11
8.7368.33
15.55
73.76
24.60
07.46 0.00
74.36 74.36-4199 41/09
04.30 -122
10.39-0144
11.09
766 1.21
$4.42.300
Men 5
.0.54
0.16
-7.00
0.59
0.40
0.21
1.25
1.58
7.95
Mix
1.14
3.97
2.18
4.53
45


Reviews:
[{"review_id": "KY3NlIJUFy", "reviewer": "Reviewer_Gar3", "summary": "This paper starts with the finding that existing CLIP models show diverse behaviours (error distributions) on classification tasks, even if they are pre-trained on the same dataset (WIT-400M). This indicates potential for model ensembling. The authors then propose a learnable ensemble module that adjusts the logits of each model with a learned temperature value conditioned on the input image, and sums up the logits of all models for the final prediction. This can also be combined with an existing cascade strategy for better efficiency. Experiments show consistent improvements over the best single backbones.", "questions": "1.1: How is the training data for the ensemble module curated and how is it split from other data? Please provide more details.\n\n1.2: When the training data for the ensemble module is only available in some classes, does it generalize well to novel classes? Also, when training with only some fixed prompts, does it overfit these prompts?\n\n1.3: Continue to 1.2, considering more complex concepts (eg, [ARO](https://github.com/mertyg/vision-language-models-are-bows) or [SugarCrepe](https://arxiv.org/abs/2306.14610) datasets), can the ensemble module correctly rerank CLIP models on novel compositions of concepts?\n\n2.1: When trained on the same dataset, different backbones in the same series (eg, R-50/101/50x4, ViT-S/B/L) make different mistakes. Does this finding only apply to CLIP or also apply to other methods (eg, vanilla supervised learning)?\n\n2.2: Is diversity always preferred when selecting models in a per-sample way, no matter whether using CLIP or not?\n\n3.1: When inferencing with the proposed method over a dataset, does the user need to load all CLIP models to the device? How much memory is needed compared to using single models, and how much inference time is needed?\n\n3.2: Can the authors also provide a comparison in computation and time cost with other ensemble methods (eg, SL, MoE)?\n\nMinor: L815: OpenCLIP is a project developed by LAION. The models mainly considered in this paper are trained by OpenAI, and the corresponding dataset is called WIT-400M.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper presents a finding that the logits of CLIP models are very diverse, and the more diverse their logits are, the higher performance can be reached by their ensemble.\n- This paper presents a comprehensive study on the diversity and properties of CLIP's logits and possible factors behind them.\n- The results produced by the ensemble model are considerably higher than single models. When compared with other ensemble methods, there is also some improvement.\n- The paper is clearly written and is easy to understand.", "weaknesses": "- The setting of training the ensemble module might harm generalizability and more clarifications are needed. It requires training data for each class regarding which model is better for each evaluation dataset, which is ad-hoc and remains questionable whether it harms CLIP's ability to generalize to unseen classes, concepts, or compositions.\n- It is not very surprising that more diverse models enable better ensembles, especially when the oracle model is picked per-sample, this adds a lot of flexibility compared to ensembling without data conditions.\n- The method itself is related to one prior method (Super Learner), and the main improvement is adding image conditions. But when compared with SL, the performance improvement is not big.\n- From a practical viewpoint, how much additional inference-time computation is required? I see the results of FLOPS, and it would be better also to report GPU memory and inference time.\n\nMore detailed classifications are elaborated in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3RSzSot6R", "reviewer": "Reviewer_7ZyN", "summary": "This paper focuses on vision-language backbones, especially two-tower CLIP like network structure. It evaluates the classification tasks across several datasets. The main idea is to propose an ensemble strategy in logic space by adaptively learning the temperature across different versions of pretrained CLIP backbones. Empirical results are based on a large group of image classification datasets. It also provides detailed analysis for a better intuition.", "questions": "Please refer to the weaknesses section above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Training large-scale backbone requires high computational cost. Therefore, thinking about how to combine existing pretrained backbones is a valuable research topic. Ensemble is a reasonable way to do so.\n2. Empirical results are extensive for image classification task including several datasets.\n3. Instead of making ensemble directly on logit from different backbone versions, it proposes a learnable module based on temperature scaling.\n4. Overall, the paper writing is in a good shape and easy to follow.", "weaknesses": "1. More evaluation tasks are valuable to be considered to further support the paper statement. Even if limited with CLIP backbone, adding more retrieval tasks will be supportive.\n2. Ensemble on logit is still relatively not novel enough as a research work. Compared with previous ensemble, it still relies on the output of each individual model and make the late fusion.\n3. Such ensemble increases the performance but requires much more test time, and it also requires to conduct a post-learning process. The overall pipeline is not that straightforward.\n4. It looks like the proposed pipeline cannot obtain improvement by adding more training shots, more analysis and intuition will be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D46i88SqRR", "reviewer": "Reviewer_6yyE", "summary": "This work investigates the differences across various CLIP-trained vision backbones with respect to classification performance and robustness, and proposes an adaptive combination of pretrained CLIP backbones, which is able to increase performance beyond regular enabling.\n\nI very much appreciate the author's systematic approach: they first show that there exist (large) differences between (zero-shot) classification performance of individual CLIP-trained vision backbones. Then they drive an oracle performance upper-bound and investigate  complementarity of different backbones concluding that different backbones classify different subsets of images differently. This motivates the authors to propose an adaptive ensembling approach called Neural Logit Controller (NLC) to combine individual backbones to one prediction.\n\nThe proposed NLC is based on network calibration using learned temperature scaling. To do so, the method needs to be informed by a supervised signal given by a single labeled example per class, as the authors claim.\n\nExperimental results on a large collection of datasets show that the proposed approach is able to outperform state-of-the-art few-shot methods. For some datasets by large (up to 39.1), on average by 9.1% over the best performing individual backbone and better than baseline ensembling methods. However, outperformance is much smaller when data-specific linear heads are trained (=1.6%). Additional results show a promising combination of Tip-Adapter with the proposed approach.\n\nAs mentioned before, I very much appreciate the systematic evaluation of the growing number of available vision foundation models. Having so many foundation models available, renders the question how *foundational* these models are. So maybe the research community should invest more time in the investigation and how to harness these foundation models. To do so, we would need to understand their differences, strengths and weaknesses and then derive from that propose a novel aggregation, ensembling or merging methods. ...and this is what this paper is doing.\n\nI also very much appreciate the large number of experiments and evaluations done by the authors to support this work - including the additional results in the appendix. One is just one thing, looking at the results and in particular the difference in performance between Fig. 8. and Fig. 9. I might be wrong but can the decrease of outperformance from 9.1% to 1.6% be linked to the additional supervision signal intrinsic to the proposed NLC method? Is this what we see here? And if yes, could the results in Fig. 8 be biased towards the supervision part of NLC? I will raise it again in the question section.\n\n\n----------------------reply to rebuttal----------------------\n\n\nI would like the authors for the information provided in the rebuttal. I went through the paper again and look in particular at the appendix and other reviewer / author discussions and given that I would like to increase my rating for this submission. I think this work contributes to the research community with its detailed evaluation and proposed ideas.", "questions": "- **(Q1)**: As mentioned above, here is the question raised earlier: looking at the results and in particular the difference in performance between Fig. 8. and Fig. 9. I might be wrong but can the decrease of outperformance from 9.1% to 1.6% be linked to the additional supervision signal intrinsic to the proposed NLC method? Is this what we see here? And if yes, could the results in Fig. 8 be biased?\n\n- **(Q2)**: why is the MoE missing in Fig.8? I assume that the linear heads from the experiment of Fig.9 are serving as MoE?\n\n- **(Q3)**: on page 6, lines 298, with one-layer MLP, do you mean a MLP with one hidden layer?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **(S1)**: this paper presents an in-depth analysis of different CLIP-trained vision backbones with respect to different dimensions such as few-shot image classification, complementarity, robustness, and diversity.\n\n- **(S2)**: this paper proposes a novel method to combine the strengths and weaknesses of available CLIP-trained vision backbones able to outperform regular ensembling methods on a large amount of datasets.\n\n- **(S3)**: this paper is easy to read and follow given the systematic build-up of the paper.", "weaknesses": "- **(W1)**: Maybe the large out-performance of the proposed method from Fig.7 is linked to the injection of a supervised signal and therefore the comparison against regular ensembles might be biased. I have raised this in **(Q1)**.\n\n- **(W2)**: I would love to see more details about the injection of the supervision signal given the one labeled example per class. I am not sure if this is what I can find in appendix G and Table G.1?\n\n- **(W3)**: not really a weakness but some minor things:\n- - page 3, line 134: train a pair of vision and text encoders φ_l and φ_v <- I think the first is the language and the second is the vision encoder.\n  - Fig 4: it is not clear which setup corresponds to the dashed lines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KtHgKQYIiV", "reviewer": "Reviewer_oxVz", "summary": "This paper analyzes the prediction diversity for CLIP models from OpenCLIP by analyzing the overlap between the sets of correctly predicted images from the test set of ImageNet1k via zero-shot classification. The paper finds that model size increases accuracy, but there exist images that are correctly predicted by smaller models (within same family as well). To combine the models, the paper proposes a method to combine logits of different backbones by scaling the logits using a learned temperature value per backbone. Temperature scaling is implemented usign a neural network called Neural Logit Controller (NLC) which takes the concatenated features of all the backbones and predicts the temperature. NLC is trained using a standard cross-entropy objective, and can be trained using only one image per class. The method boosts the zero-shot and linear probing performance on a number of downstream tasks, and the performance boost correlates with prediction diversity.", "questions": "1. What is the standard deviation of the accuracy when using different subsets of data for training the methods in few-shot setting? This is important to understand the robustness of the method when using different training samples.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper shows that smaller models have certain samples that they can predict correctly, and larger models can give incorrect predictions on these samples. This is an interesting observation, and highlights the need for careful thought when scaling models.\n\n2. The proposed NLC is simple but effective, and can be trained in a data-efficient manner as demonstrated in Section 4.2 and 4.3.", "weaknesses": "1. The authors claim on line 200 that the results are non-obvious is not entirely true. Prior work has shown similar results across vision and language domains, albeit on different models [1, 2, 3]. [1] have shown prediction diversity on a range of models trained in a supervised setting and propose a method for combining models using distillation. They show that weaker models are more specialized in certain subsets of data. While the metric for measuring diversity is different, they study the same core problem. [2] show that finetuning models on different domains and then averaging model weights can improve out-of-distribution performance (this assumes the same model size, but studies a similar problem of different models being good at different subsets of data).\n\n2. **Incomplete comparisons in few-shot experiments (Section 4.3)**:\n- The results in Section 4.3 (Comparison with Few-Shot Adapter Methods) are incomplete: Figure 10 and Table 1, which show the zero-shot performance of TiP-Adapter applied to individial backbones and NLC does not show the complete picture - since NLC is an ensemble method, a proper comparison would be other ensemble methods. Similar is the case when NLC is combined with Tip-Adapter. In principle, any ensemble method can be combined with Tip-Adapter, so a proper comparison would be with other ensemble methods applied in a similar manner as NLC. Without these results, the comparisons are incomplete. \n\n3. **Number of training samples**:\n- Point 2 is especially more important considering that few-shot experiments in Section 4.3 are the only experiments which demonstrate the data efficiency of NLC, which is claimed to be a major contribution by the authors. From Section 4.1: \"The non-parametric ones are “static” while the parametric ones use the training split of each target dataset to adjust the ensemble adaptively.\" implies that NLC uses the entire training set for calibration. For few-shot experiments, if this data efficiency is applicable to other ensemble methods as well, this cannot be attributed to NLC, but would be a general property of ensemble methods.\n- From linear probing experiments in Section 4.2, Figure 9 shows that with a similar training budget, NLC does not provide a significant improvement over Super Learner (SL). Mean increase of 1.6 is the same, and SL also does not lead to any degradation over the single best model. The max increase seems to be a noisy metric to track, since MoE can show a maximum 20.8 increase but also shows degradation, indicating the max increase is really dataset specific.\n\n4. The paper claims that the method can be combined with Cascade, but does not provide the trainiing details. The paper also does not compare the performance of the proposed method with the original Cascade method. Minor: The paper also misses a citation for a cascade method [4].\n\n5. Although the paper mentions this in the limitations, fusing models at training stage has been done in [1] in supervised classification settings, hence the additional inference overhead of NLC is a major practical limitation. Furthermore, while the paper shows the potential for applying the method to other tasks like image retrieval, the paper does not actually study this using NLC, making the impact of the method limited.\n\n**References**:\n\n[1] Karsten Roth, Lukas Thede, A. Sophia Koepke, Oriol Vinyals, Olivier J. Hénaff, Zeynep Akata: Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model. ICLR 2024\n\n[2] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, David Lopez-Paz: Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization. ICML 2023\n\n[3] Ruiqi Zhong, Dhruba Ghosh, Dan Klein, Jacob Steinhardt: Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level. ACL/IJCNLP (Findings) 2021: 3813-3827\n\n[4] Neeraj Varshney, Chitta Baral: Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems. EMNLP 2022: 11007-11021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)