Synergy and Diversity in CLIP: Enhancing Performance Through Adaptive Backbone Ensembling

## Abstract

Abstract Contrastive Language-Image Pretraining (CLIP) stands out as a prominent method for image representation learning. Various architectures, from vision transformers (ViTs) to convolutional networks (ResNets) have been trained with CLIP to serve as general solutions to diverse vision tasks.
This paper explores the differences across CLIP-trained vision backbones.
Despite using the same data and training objective, we find that different architectures have notably different representations,
different classification performance across datasets, and different robustness properties to various image transformations.
These findings suggest a remarkable possible synergy across backbones
by leveraging their respective strengths.
In principle, classification accuracy could be improved by over 40 percentage with an informed selection of the optimal backbone per test example.
Using this insight, we develop a straightforward yet powerful approach to adaptively ensemble multiple backbones.
The approach uses as little as one labeled example per class
to tune the adaptive combination.
On a large collection of datasets, the method achieves a remarkable increase in accuracy of up to 35.1% over the best single backbone, well beyond traditional ensembling methods.

## 1 Introduction

Large pre-trained models are transforming machine learning, computer vision [radford2021learning, he2022masked, kirillov2023segment, liu2023improved], and natural language processing [devlin2018bert, brown2020language, Touvron2023LLaMAOA].
These models are typically trained with self-supervised objectives, eschewing the need for manual annotations and enabling the use of very large datasets [jia2021scaling, schuhmann2022laion].
Contrastive Language–Image Pre-training (CLIP) [radford2021learning] is one such approach that enables various downstream applications involving vision and language.
CLIP learns to align text and image representations
when training a pair of vision and language encoders, which can be implemented with various architectures.
These encoders are then used in downstream applications to obtain representations of images and text, whose similarity can be simply evaluated with a dot product. This enables
e.g. zero-shot classification [Zhai_2022_CVPR, li2022elevater, jia2021scaling] and cross-modal retrieval [li2020unicoder, li2020oscar, yu2022coca].

Figure: Figure 1: We propose a method to improve CLIP’s effectiveness for image classification by combining the strengths of different backbones. (Left) For a given test image, the logits from different backbones are combined with a temperature scaling that weights their contribution to the final prediction. The scaling is implemented in the Neural Logit Controller (NLC, a small MLP) that is learned from as little as one labeled example per class. (Right) To reduce the computational load, our method can be combined with the Cascade framework [wang2022wisdom].
Refer to caption: /html/2405.17139/assets/x1.png

Despite extensive prior work on CLIP, there is still a gap in comparing the representations from different vision backbones trained with this paradigm. Existing work has compared backbones for their generalization capabilities [goldblum2023battle, li2022elevater, zhang2021tip, gao2021clip], finding that larger architectures generally perform better.
This paper contributes to this area with an empirical study that compares CLIP-trained backbones.
We present new observations on how performance and robustness (e.g. invariance to image transformations)
vary substantially across architectures.
Within the same backbone family (e.g. different ViTs), different models present different patterns of performance across datasets (Figure [4](#S2.F4)).

Figure: Figure 2: (Y axis) Relative improvement of NLC over to the best backbone vs. (X axis) the diversity of predictions.^1 NLC always improves over the best backbone. Moreover, the clear correlation shows that higher diversity in predictions tends to result in greater improvements with NLC.
Refer to caption: /html/2405.17139/assets/x2.png

The relation with model scale is
also more complicated than suggested in prior work (Figure [5](#S2.F5)).

Our empirical findings suggest that CLIP’s effectiveness for image
classification could be improved by combining the strenghts of different backbones. We thus present an ensemble method that combines
backbones adaptively to each test example,
using a temperature scaling that weights each backbone’s contribution to the final prediction (see Figure [1](#S1.F1)).
A naive combination of backbones with traditional ensembling techniques [lakshminarayanan2017simple, dietterich2000ensemble]
(i.e. averaging the outputs of different models)
does not consistently enhance generalization performance.
In contrast, our method uses a temperature scaling
based on the idea of calibrating each backbone’s confidence.
This intuition is further supported by the
“modality gap”
and variation in performance observed in prior work
when adjusting CLIP’s logits [clip-understanding].

The proposed method is named the Neural Logit Controller (NLC).
It uses a few labeled examples (as little as one per class) to tune the combination of backbones. Hence we benchmark it against the state-of-the-art methods in a few-shot setting [zhang2021tip, radford2021learning, zhou2022learning, gao2021clip].
NLC shows remarkable performance and consistently outperforms other approaches.
We also demonstrate a complementarity of our improvements with the existing Tip-Adapter [zhang2021tip]. Combining it with NLC in its original experimental framework yields improvements in accuracy of over 15%.

Finally, in-depth experiments reveal a clear correlation between
NLC’s improvement over the best backbone and the diversity of correct predictions in each dataset (see Figure [3](#S1.F3)).
It achieves over 120% of relative improvement when the diversity of predictions is close to one, and never degrade the performance of the best backbone. In terms of efficiency, NLC outperforms the best backbone by combining only the top-four most efficient backbones, while using approximately 300 fewer GFLOPs (see Figure [3](#S1.F3)).
We also show how to use the Cascade framework [wang2022wisdom]
with NLC to enhances performance while maintaining computational requirements within the bounds of the original backbones.

Our contributions are summarized as follows.

- •
We perform extensive experiments across 21 datasets that reveal diverse predictions across CLIP backbones
and distinct robustness properties to image transformations.
- •
We evaluate the complementarity of backbones
by measuring the potential improvement in classification accuracy
(up to 43.5%)
of an optimal oracle selection of backbone per test example.
- •
To leverage this complementarity, we propose an adaptive ensembling method (NLC)
that uses temperature scaling and requires a single labeled example per class.
It improves accuracy over the best backbone by 9.1% on average.
- •
We demonstrate significant advantages in computational efficiency. Combining the top four most efficient backbones enables NLC to outperform the best backbone with $\sim$300 fewer GFLOPs. Integrating NLC with the Cascade approach [wang2022wisdom] maintains computational efficiency.
- •
We demonstrate that
NLC consistently outperforms state-of-the-art few-shot methods.
Moreover, integrating NLC with Tip-Adapter [zhang2021tip]
gives the latter a performance boost of over 15%.

## 2 Investigating Differences Across Vision Backbones

#### CLIP for zero-shot image classification.

Figure: Figure 4: Zero-shot classification accuracy of various CLIP models on 21 datasets, and of the upper-bound “Oracle” combination of ResNets (RN), ViTs, and all backbones.

CLIP [radford2021learning]
is a general technique to train a pair of vision and text encoders
$\phi_{l}$ and $\phi_{v}$ on paired image/text data.
The method uses a self-supervised objective to align the encoded representations of matching images and text descriptions.
Any image and text can then be mapped to a unified semantic space.
This enables
zero-shot image classification by computing a compatibility score
between an image and a class description (“prompt”) as $\operatorname{score}(x,l)=\phi_{v}(x)^{\top}\phi_{l}(l)$.
We follow previous work [menon2022visual] for the generating of prompts
e.g. an image of {label}, which is a {concept}. Subsequently, we calculate the compatibility score between the image feature $\phi_{v}(x)$ and the prompt representations $\phi_{l}(y)$, where $y$ represents a label within the class set. The prompt with the highest similarity is then chosen as the label for the given image.

### 2.1 Complementarity of Different Backbones

The vision encoder or “backbone” trained with CLIP can be implemented using various architectures.
We want to identify whether different backbones have distinctive behaviours.
We analyze the output of 9 of them
in a zero-shot classification setting on 21 standard benchmark datasets (see Section [4](#S4.SS0.SSS0.Px2) for details).
We propose two approaches to assess the differences
in their zero-shot predictions
and their possible complementary behaviour.
First, we define an Oracle prediction that combines
the output of the 9 models on a per-image basis, using a correct prediction if any of the models is correct, any other otherwise.
This simulates a scenario where an informed choice of the optimal backbone could be made for each test image to obtain the highest classification accuracy. Second, we use overlap diagrams to visualize the patterns of (in)correct predictions across backbones
and their different strengths on different subsets of images.

Figure: Figure 5: Linear Venn diagrams showing the overlap of test images from ImageNet-1k correctly classified by different backbones (rows). Each column represents a subset of images correctly classified by a specific group of backbones (group size in column header). Row/column sums indicate the number of correct predictions per backbone/subset. We observe that (1) different backbones agree on a large part of the data. (2) They also make additional correct predictions on different subsets. (3) Accuracy usually grows with architectures size, but even within a same family (ViTs, ResNets), different models show different patterns of (in)correct predictions.
Refer to caption: /html/2405.17139/assets/x3.png

Figure [4](#S2.F4) and Table [B.1](#A2.T1)
summarize the performance of the different backbones.
Note that they are pre-trained using the exact same dataset and objectives, as provided in the OpenClip project [ilharco_gabriel_2021_5143773].
Our results show that backbones from different families (ViTs vs. ResNets)
show significant differences whole controlling for model size (number of parameters).
For example, ViT-B-32 outperforms ResNet-50x4 in the CIFAR10 and CIFAR100 benchmarks. However, this trend is not consistent across all datasets e.g. on ImgNet-1k. Additionally, the results indicate that backbones with more parameters do not always outperform smaller ones.

We then assess the Oracle predictions using backbones from the same family
(Oracle RN for ResNets, Oracle ViT for ViTs, and Oracle All for a combination of all backbones).
The Oracle All obviously outperforms the best individual
backbone in every case.
For instance, on Eurosat, CLEVR, CUB, DTD, CIFAR100, and Cars, the Oracle All shows improvements of 43.5%, 36.0%, 25.1%, 21.8%, 16.6% and 16.0%, respectively.
The large magnitude of these improvements highlights the complementarity of backbones.
In other words, the correctly-classified data is not simply growing
as one considers larger and better models.
Instead, different backbones perform well on different subsets of the data.
This non-obvious insight is the key motivation for the ensemble method of Section [3](#S3).

We visualize in Figure [5](#S2.F5)
the overlap between different subsets of ImgNet-1k correctly predicted by different backbones.
We observe comparable accuracy between the largest ViT (ViT-L-14-336, correct predictions) and ResNet (RN50x64) with $38,285$ vs. $36,963$ correct predictions.
However, there are significant differences within each family: for example, among ResNets, only $22,358$ images are correctly classified by all ResNets models, out the $42,426$ ones correctly classified by *any* ResNet.^1

We further observe that every model has unique strengths by examining the performance of the oracle models.
Specifically, Oracle RN, Oracle ViT, and Oracle All increase the number of correct predictions to $42,426$, $42,276$, and $44,506$ out of $50,000$. Each backbone exhibits correct predictions unique to itself. In the Oracle RN, RN50, RN101, RN50x4, RN50x16, and RN50x64 contribute $566$, $455$, $518$, $782$, and $1,745$ exclusive predictions, respectively.
In the Oracle ViT, B-32, B-16, L-14, and L-14-336 present $903$, $904$, $543$, and $818$ unique predictions, respectively. Finally, when all backbones are used, ResNet contributes $2,230$ exclusive predictions, and ViT contributes $2,080$. It is important to note that correct predictions of any given model are not simply a subset of those by the best model in its family.

Figure: Figure 6: Impact on classification accuracy of image transformations applied to test examples. Smaller values (closer to the center) means higher robustness. Certain models prove particularly robust to specific transformations.

### 2.2 Robustness to Image Transformations

To explore qualitative differences in behaviour of different backbones,
we selected a subset of each benchmark’s test set and applied specific image transformations (resize, flip, grayscale, color jitter, and Gaussian blur).
We then evaluate the zero-shot classification performance of each backbone on these corrupted images.
Figure [6](#S2.F6) shows that ResNet 50 and ViT-B-16 are the most resilient
to flip, RN50x64 to resize, and RN50x16 to grayscale.
ViT-L-14 and L-14-336 are the most robust to color jitter and Gaussian blur.
These findings suggest that the backbones’ ability to deal with out-of-distribution images varies widely. Certain models prove particularly robust to specific transformations.

## 3 Proposed Ensembling Method

The previous section showed that different backbones have different strengths
and perform well on different types of data.
We now use this insight to improve classification performance by combining multiple CLIP backbones.

#### Combining models by temperature scaling.

Our approach is technically inspired by prior work on network calibration, which we apply to an ensemble of models. Specifically, our technique can be seen as a variation of Platt scaling [platt1999probabilistic].
This classical method uses the logits of a model as features for a logistic regression, which is trained on the validation set to produce calibrated probabilities. More precisely, given logit scores ${\bm{z}}_{i}$ for an example $i$, Platt scaling learns two scalars $a$ and $b$ and produces calibrated probabilities as $\hat{q}_{i}=\sigma(a{\bm{z}}_{i}+b)$. The parameters $a$ and $b$ can be optimized using the negative log likelihood (NLL) loss over the validation set, with the model weights being frozen during this process.

A common use of Platt scaling is known as temperature scaling [guoCalibrationModernNeural2017].
In this approach, a single scalar parameter $t>0$ is used for all classes of a given model. The new, calibrated confidence prediction is given by $\hat{q}_{i}=\max_{k}{\;\text{softmax}({\bm{z}}_{i}/t)^{(k)}}$, where $t$ is called the temperature, ${\bm{z}}_{i}$ are the logits for example $i$ returned by the uncalibrated model.
Usually, $t$ is optimized with respect to the NLL on the validation set aiming to reduce the overconfidence of the model on its predictions and to produce more reliable predictions, but because the parameter $t$ does not change the maximum of the softmax function, the class prediction $\hat{y}_{i}$ remains unchanged, meaning that the performance of a given model remains the same.

#### Learning temperature coefficients.

In our method, we aim to jointly optimize a set of temperature parameters $t_{b}$ with $b\in[1,\ldots,B]$ for a set of $B$ backbones.
The aim is to combine their predictions by adjusting altering each backbone’s confidence depending on the confidence of the others, and the input example.
We learn the temperatures $t_{b}$ that weigh the logit ${\bm{z}}_{i}^{b}$ for a backbone $b$ and example $x$ using the cross-entropy loss.

Concretely,
we train a one-layer MLP (our Neural Logit Controller) to predict the set of temperatures that best calibrate the backbone mixture.
As depicted in Figure [1](#S1.F1),
the MLP takes as input the concatenated representations obtained by passing the images through the encoder $\phi_{v}$ of each backbone $b\in{\mathbb{B}}$.
The MLP directly produces a vector of temperatures ${\bm{t}}\in{\mathbb{R}}^{B}$ and is trained on a validation set using the cross-entropy loss between final predictions and ground truth labels.

## 4 Experiments

#### Backbones.

We consider the original selection of backbones described by \textciteradford2021learning.
This includes the ResNets [He_2016_CVPR] RN50, RN50x4, RN50x16, RN50x64 and RN101, and the ViTs [dosovitskiy2020vit] B-16, B-32, L-14 and L-14-336.
All models are obtained through the open-source project OpenCLIP [ilharco_gabriel_2021_5143773].

#### Datasets.

We use a selection of 21 popular image classification datasets: Caltech101 [li_andreeto_ranzato_perona_2022], Cars [krause2013collecting], CIFAR10 [krizhevsky2009learning], CIFAR100 [krizhevsky2009learning], CLEVR [johnson2017clevr], CUB [WahCUB_200_2011], DTD [cimpoi14describing], Eurosat [helber2018introducing], FGVC [maji13fine-grained], Flowers [nilsback2008automated], Food [bossard2014food], GTSRB [Houben-IJCNN-2013], ImgNet-1k [deng2009imagenet] MNIST [deng2012mnist], PCam [Veeling2018-qh], Pets [parkhi2012cats], RenderedSST2 [socher-etal-2013-recursive], Resisc45 [Cheng_2017], STL10 [coates2011analysis] and SUN397 [Xiao:2010].
They include diverse images of nature, animals, places, medical scans, satellite images, and man-made objects.
The evaluation metric is simply the accuracy on the test split of each dataset. The aggregated performance is the average accuracy across all datasets weighted equally.

#### Baselines.

Our experiments consider several ensembling baselines. Non-parametric approaches:

- •
Logit averaging (Log-Avg) simply takes the average of the logits (scores) of the models.
- •
Voting (Vote T-1)
uses a majority vote. Each model casts a vote for its max-scored class.
The final prediction is the class with the highest number of votes.
Compared to Log-Avg, this does not use the *soft* score values and the uncertainty reflected therein.

Parametric approaches:

- •
Calibrated logit averaging (C-Log-Avg)
first calibrates each model independently using temperature scaling.
They are then combined by simply averaging their logits.
- •
Genetic Algorithm (GAC). To showcase the importance of adaptively adjusting the logits, we use a genetic algorithm to find the optimal temperatures for combining the backbones. These temperatures are fixed after training, ensuring they remain constant regardless of the input.
- •
Mixture of experts (MoE) is a popular method to improve performance by combining specialized submodels [NEURIPS2022_91edff07, lepikhin2020gshard, rau2019moe, Zoph2022STMoEDS].
An MoE splits a model into experts that each focus on a specific input region. A gating network determines which expert is relevant for a given input.
We use the Sparse MoE implementation [Zoph2022STMoEDS] by \textciterau2019moe, where the MoE layer’s input is the concatenation of vision features $x_{b}$ from $\phi_{b}$. We use standard hyperparameters: $9$ experts trained for classification with cross-entropy loss with Adam [kingma2014adam] and a learning rate of $2\mathrm{e}{-5}$ for 300 epochs.

See Appendix [C](#A3) a discussion of additional baselines.
The appendix also contains additional results on
the combination of backbones and how they perform
under distribution shifts on the ImgNet-1k dataset (Section [E](#A5)).
Finally, Section [H](#A8) presents
dataset-specific results including
overlap diagrams, an analysis of possible model combinations using NLC and our Oracle, as well as the dataset-specific learned temperature values.

### 4.1 Comparison of Backbone-Ensembling Methods

In this section, we evaluate the proposed NLC approach along with
various non-parametric and parametric ensembling techniques.
The non-parametric ones are “static” while the
parametric ones use
the training split of each target dataset to adjust the ensemble adaptively.
The following results will show the importance of this adaptive setting to leverage the backbones’ respective strengths, and the superiority of our method over baseline parametric techniques.

Figure [4](#S4.SS0.SSS0.Px3) and Table [C.1](#A3.T1) show the performance of each technique.
Among non-parametric baselines,
Log-Avg fails to enhance overall performance beyond the best backbone.
One notable exception is the Country dataset where Log-Avg shows a substantial improvement over the best backbone.
Overall though, simple score averaging does not seem exploit the complementarity of the backbones, with an improvement in accuracy of only $+1.3$%.
When the backbones are calibrated with C-Log-Avg,
performance improves compared to the non-calibrated version, across all datasets except GTSRB, MNIST, Resisc45, and SUN397.
Vote T-1 is mostly ineffective and does not significantly improve over the best backbone.
Among parametric methods, the proposed NLC shows
the most substantial improvement, up to $+39.1$% on the Eurosat dataset. On average, we obtain a commendable improvement of $+9.1$%
over the Single-Best backbone across all datasets.

### 4.2 Investigating Benefits beyond Dataset-Specific Linear Classifiers

We now design an additional experiment to investigate the benefits of the proposed NLC.
A standard approach to adapt CLIP is to train a dataset-specific linear classifier on frozen visual features.
In this section, we first train such linear classifiers,
using each backbone’s frozen features and each dataset’s training split.
Whereas NLC normally acts on raw CLIP scores,
we now train it to act on the logits from these classifiers instead.
Since the linear classifiers already are strong dataset-specific models, one might expect that little more improvement could be gained.
Yet, the following results will show that the NLC realises further gains thanks to the *instance*-level adaptation.
And in contrast to alternative approaches, it never reduces the performance compared to the best single linear classifier.

Setup. We follow the setup of \textciteli2022elevater. We initialize the weights of the linear classifiers using language weights, which was shown to be more stable than a random initialization.
The output of each backbone $\phi_{b}$ is L2-normalized before being passed to each linear classifier, which is trained using 90% of the target dataset’s training split. The remaining 10% are used to train NLC.

**Table 1: Accuracy (%) in a few-shot setting.**
| Few-shot | 1 | 2 | 4 | 8 | 16 |
| --- | --- | --- | --- | --- | --- |
| Linear-probe CLIP [radford2021learning] | 22.2 | 31.9 | 41.2 | 49.5 | 56.1 |
| CoOP[zhou2022learning] | 47.6 | 50.9 | 56.2 | 59.9 | 63.0 |
| CLIP-Adapter[gao2021clip] | 61.2 | 61.5 | 61.8 | 62.7 | 63.6 |
| Tip-Adapter[zhang2021tip] | 60.7 | 61.0 | 61.0 | 61.5 | 62.0 |
| Tip-Adapter-F[zhang2021tip] | 61.3 | 61.7 | 62.5 | 64.0 | 65.5 |
| NLC | 78.2 | 78.1 | 78.2 | 78.3 | 78.4 |

Results.
Figure [4.2](#S4.SS2) and Table [D.2](#A4.T2) present the results combining linear classifiers with both non-parametric and parametric approaches. Similar to Section [4.1](#S4.SS1), the proposed NLC consistently improves performance across all datasets, with an average improvement of $+1.6$% over the Single-Best linear classifier.
As expected, the improvement is reduced compared to the original NLC since some of the gains are now realised by the dataset-specific linear classifier.
Yet Table [D.1](#A4.T1) clearly shows that additional gains can be made over the linear classifiers, i.e. that there remains an exploitable complementarity across backbones.
By examining the Oracle performance in this setting, we can see that the potential benefits of combining multiple backbones remain.
Interestingly, the MoE approach does not reach the performance level of the best backbone on several datasets. It shows a degradation sometimes down to $-20.9$%. This suggests that MoEs may struggle to partition the input space effectively into distinct clusters, specific to certain experts, potentially limiting its functionality.

### 4.3 Comparison with Few-Shot Adapter Methods

We now examine the performance of the proposed NLC in a few-shot setting.
We also show that it complements existing few-shot adapter methods
and improves their accuracy across various datasets.

We compare our method against Tip-Adapter, Tip-Adapter-F [zhang2021tip], few-shot Linear-probe CLIP [radford2021learning], CoOP [zhou2022learning], and CLIP-Adapter [gao2021clip]. Linear-probe CLIP fine-tunes a linear classifier on a few-shot training set on the frozen CLIP backbones. CoOP [zhou2022learning] generates different prompt designs to make prompts learnable. CLIP-Adapter [gao2021clip] enhances few-shot classification by introducing a feature adapter on CLIP’s visual and textual encoders. Tip-Adapter [zhang2021tip] achieves comparable performance to CLIP-Adapter without requiring training, using a key-value cache model from the few-shot training set. Tip-Adapter-F can further enhance performance by fine-tuning the cache. As shown in Table [1](#S4.T1), our NLC shows an outstanding performance over compared methods and consistently surpasses other few-shot methods by learning how to combine multiple backbones, which is a complementary approach to any of these adapters. In the appendix,
Table [E.3](#A5.T3) shows that current few-shot adapter methods also have performance differences with various backbones, and still, our NLC surpasses the best backbone reported by these existing methods.

[Uncaptioned image]: /html/2405.17139/assets/x6.png

Figure: Figure 7: Distribution of temperature values learned by our NLC method on the CLEVR dataset (normalized by maximum value).
Refer to caption: /html/2405.17139/assets/x7.png

Finally, we integrate Tip-Adapter [zhang2021tip] with our NLC and conduct experiments on Eurosat, FGVC, DTD, and Pets. Initially, we apply Tip-Adapter and Tip-Adapter-F independently on each backbone, following the protocol from \textcitezhang2021tip with 1, 2, 4, 8, and 16 shots. Subsequently, we employ our ensembling mechanism to fuse the adapted backbones. To train NLC, we follow their setting and use the same validation split as used by Tip-Adapter to linearly combine their logits with CLIP.

See Figure [4.3](#S4.SS3) and Tables [F.2](#A6.T2)–[F.4](#A6.T4)) for a comparison of the performance of Tip-Adapter over different zero-shot CLIP backbones, and the combination of all these Tip-Adapter versions with NLC. Across the four evaluated datasets, NLC improves over each version of Tip-Adapter in the few-shot setting. Notably, NLC for Eurosat shows an improvement of up to 10% with respect to the best Tip-Adapter backbone (L-14-336) using 16 shots.

### 4.4 Examining the Learned Temperature Values

We visualise the learned temperature values to understand how the NLC method assigns weights to different backbones, revealing its strategy for leveraging the strengths of diverse architectures.
Figure [7](#S4.F7) shows the learned temperature values for NLC on the CLEVR dataset (see Section [H](#A8) for other datasets).
By examining the learned temperatures, we can assesses the contribution of each backbone. We note that the most weighted backbone is not consistently the deepest one such as ResNet-101.
The uniform distribution of learned temperatures across backbones indicates that NLC effectively utilizes the strengths of multiple backbones, resulting in a more balanced and accurate labeling process, complementing our analysis in Figure [3](#S1.F3).
These findings highlight that NLC benefits from the diversity in model predictions, reinforcing the advantage of combining various backbones.

## 5 Related Work

Foundation models in computer vision.
The trend in computer vision over the past decade has been to train
larger and larger models
on increasingly diverse tasks
and increasingly larger datasets [gadre2023datacomp, schuhmann2022laion].
Scale and generality have proved to be key enablers to the performance and robustness of these models
[fang2022data, gan2022vision, mayilvahanan2023does, santurkar2022caption, tu2024closer].
Various methods have been proposed to pre-train such models
[he2022masked, radford2021learning, yu2022coca].
CLIP [radford2021learning] is a notable one, specifically designed to exploit noisy image-text pairs scraped from the internet.
CLIP’s significance lies in its scalability and its ability to generate meaningful alignments with prompts, facilitating zero-shot classification [gao2021clip, zhang2021tip, li2022elevater].
CLIP’s popularity is due to its scalability and ability to handle diverse text prompts that enable a variety of downstream applications [gao2021clip, zhang2021tip, li2022elevater].
Various versions of CLIP have been trained on different datasets and vision backbones such as ResNets [He_2016_CVPR] and ViTs [dosovitskiy2020vit].

Comparing vision backbones.
\textcitegoldblum2023battle recently conducted a comprehensive comparison of pretrained backbones across various tasks including image classification, object detection, segmentation, and image retrieval.
In comparison, this paper presents complementary observations
of the strengths of different backbones on different datasets and types of data.
Moreover, we leverage these findings with a novel ensembling approach this complementarity of different backbones.
Other works investigating differences across vision architectures include [angarano2022back, pinto2022impartial, pinto2021vision, wang2022can] and others specific to CNNs [abello2021dissecting, hermann2020origins] and ViTs [naseer2021intriguing].

Model ensembling.
Combining multiple machine learning models is a classical approach for improving predictive performance.
Simply averaging the outputs of several models is a simple effective technique
[bauer1999empirical, breiman1996bagging, dietterich2000ensemble, lakshminarayanan2017simple]
with studies dating back more than three decades ago.
The approach has also been applied to deep neural networks (deep ensembles [lakshminarayanan2017simple]), which has
shown benefits in various domains including higher accuracy under distribution shift [ovadia2019can, teney2018tips].
The diversity of the combined models (in terms of uncorrelated errors) has also been showed to be critical to these improvements [hao2024benefits, wortsman2022model].
In contrast to most ensembling approaches that combine multiple instances of the same architecture,
our proposed method combines several vision backbones that are implemented with different architectures.

## 6 Conclusions

This paper presented an analysis of vision backbones in the CLIP framework, focusing on the task of image classification. Unlike prior studies that span various downstream tasks (e.g. [goldblum2023battle])
our emphasis lies in identifying the unique strengths of various backbones.
Our experiments revealed a distinctive complementarity across architectures and an avenue for enhancing CLIP’s performance by synergistic combination.
We proposed an ensemble approach that reweights the logits from each backbone to yield more accurate predictions in image classification across a variety of datasets.

Limitations.
First, our evaluation focused solely on image classification, even though CLIP can be used for a variety of downstream tasks.
Second, our approach relies on a late fusion of backbones by adaptively adjusting their logits.
Alternative approaches using early fusion have not been investigated.
These might provide further performance gains by leveraging the same initial motivation.
Third, although we employed the Cascade method [wang2022wisdom] to reduce the computations, our method still needs to pass a test image multiple times through a vision encoders. This adds complexity and computational overhead, which can be important if a large number of backbones are considered.
A possible solution would involve fusing backbones at the training stage to require only a single forward pass at test time.

Future work
could extend our analysis to other downstream tasks, such as image-text matching to understand the capabilities of backbones in more diverse contexts.
Other efficient methods could also be developed to combine predictions from multiple backbones to enhance the scalability of the idea.

Societal impact.
This work seeks to better understand the applicability
of computer vision models
and improve their reliability.
Notwithstanding the potential nefarious uses of such technologies, our contributions are seen as overwhelmingly positive.

## Appendix A Reproducibility

The one-layer MLP is trained with ADAM optimizer with a learning rate of 2e-4. We use a weight decay of 0.01. It receives as input the concatenation of the features, the hidden layer has a width of 128 and its outcome is the temperature values for the used backbones. All the backbones used in the paper are pre-trained using the same dataset called openai and objectives from the OpenAI Fundation [https://github.com/mlfoundations/open_clip](https://github.com/mlfoundations/open_clip).
In the Cascade [wang2022wisdom], we use the probability as the confident measurement with a threshold of $0.9$. When the method is confident with a probability bigger than $0.9$ we produce a final prediction, otherwise, we add the next backbone.
All our experiments are done in one NViDIA GeForce RTX 4090 GPU, in a small server with 64 GB of RAM and 32 CPU cores.

## Appendix B Oracle Performance

**Table B.1: Zero-shot accuracy performance comparison of CLIP backbones on 21 datasets, showcasing the number of parameters and GFLOPs per image for each backbone. Additionally, explore the performance of our empirical upper-bound prediction, denoted as ‘Oracle’, representing the ideal combination of ResNet (RN), ViT, and All backbones.**
|  |  | Num<br>Parameters | GFLOPs/<br>Image | Caltech101 | Cars | CIFAR10 | CIFAR100 | CLEVR | Country211 | CUB | DTD | Eurosat | FGVC | Flowers | Food | GTSRB | ImgNet-1k | MNIST | PCam | Pets | SST2 | Resisc45 | STL10 | SUN397 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CLIP-ResNet | 50 | 102.0M | 34.0 | 77.9 | 54.2 | 71.5 | 40.3 | 21.6 | 15.4 | 46.6 | 41.2 | 28.2 | 17.1 | 66.1 | 77.9 | 35.1 | 59.8 | 49.1 | 63.9 | 85.8 | 56.6 | 44.0 | 94.2 | 58.6 |
| 101 | 119.7M | 48.9 | 81.8 | 61.1 | 80.8 | 47.6 | 24.4 | 16.9 | 49.6 | 43.7 | 26.4 | 18.6 | 65.2 | 81.9 | 37.5 | 62.3 | 46.0 | 58.1 | 86.9 | 63.9 | 54.5 | 96.8 | 57.2 |  |
| 50x4 | 178.3M | 97.6 | 82.0 | 66.9 | 79.4 | 45.1 | 20.5 | 20.4 | 54.2 | 48.6 | 27.4 | 21.1 | 69.9 | 85.5 | 36.2 | 66.2 | 53.3 | 56.7 | 88.9 | 67.2 | 54.3 | 96.6 | 59.7 |  |
| 50x16 | 291.9M | 309.7 | 85.1 | 73.2 | 81.3 | 52.1 | 19.6 | 24.4 | 57.9 | 53.4 | 41.2 | 27.5 | 72.0 | 89.7 | 39.8 | 70.7 | 62.7 | 62.5 | 90.1 | 67.5 | 59.4 | 97.8 | 63.6 |  |
| 50x64 | 623.3M | 1066.8 | 83.9 | 75.9 | 85.1 | 59.9 | 22.7 | 29.8 | 62.7 | 53.1 | 48.0 | 31.0 | 76.1 | 91.1 | 47.9 | 73.9 | 78.9 | 53.9 | 93.6 | 71.0 | 63.9 | 98.3 | 66.0 |  |
| CLIP-ViT | B-32 | 151.3M | 30.1 | 83.7 | 59.7 | 89.8 | 64.2 | 23.2 | 17.2 | 53.0 | 44.0 | 37.4 | 19.7 | 66.5 | 82.6 | 32.6 | 63.3 | 38.1 | 62.3 | 87.5 | 58.6 | 54.2 | 97.1 | 61.3 |
| B-16 | 149.6M | 85.5 | 86.6 | 64.6 | 90.8 | 66.9 | 21.2 | 22.8 | 55.3 | 45.1 | 44.1 | 24.4 | 71.4 | 87.9 | 43.3 | 68.3 | 59.5 | 50.7 | 89.1 | 60.5 | 58.3 | 98.2 | 63.7 |  |
| L-14 | 427.6M | 364.6 | 85.9 | 77.8 | 95.6 | 75.8 | 19.4 | 31.9 | 62.1 | 55.3 | 47.2 | 31.7 | 79.1 | 92.3 | 50.6 | 75.5 | 73.3 | 52.0 | 93.6 | 68.9 | 64.6 | 99.4 | 66.7 |  |
| L-14-336 | 429.9M | 858.0 | 86.1 | 79.4 | 95.0 | 74.4 | 20.0 | 34.5 | 63.0 | 56.4 | 45.0 | 33.2 | 78.3 | 93.1 | 52.4 | 76.6 | 75.2 | 60.7 | 93.8 | 70.6 | 63.7 | 99.4 | 67.7 |  |
| Oracle | RN |  |  | 90.2 | 91.3 | 96.5 | 76.8 | 56.2 | 41.2 | 82.5 | 73.5 | 81.9 | 52.3 | 86.0 | 96.1 | 65.2 | 84.9 | 90.2 | 87.0 | 98.1 | 95.3 | 82.3 | 99.3 | 82.7 |
| ViT |  |  | 91.9 | 89.9 | 98.5 | 87.1 | 34.4 | 44.0 | 78.9 | 67.2 | 72.0 | 51.9 | 85.7 | 96.6 | 67.0 | 84.6 | 85.6 | 95.4 | 97.5 | 92.3 | 78.7 | 99.7 | 80.6 |  |
| All |  |  | 93.7 | 95.4 | 99.3 | 91.0 | 60.4 | 52.0 | 88.1 | 78.2 | 91.5 | 66.3 | 88.9 | 98.0 | 77.9 | 89.0 | 95.1 | 99.1 | 99.1 | 98.6 | 88.7 | 99.8 | 87.0 |  |

Table [B.1](#A2.T1) presents the results depicted in Figure [4](#S2.F4) from the main paper. It includes the number of parameters and GFLOPs/Image for each backbone, the performance of zero-shot CLIP (OpenAI) on the selected datasets, and the performance of the proposed Oracle method.

We observe that backbones with a similar number of parameters from different families can show significant differences in performance. For instance, ViT-B-32 outperforms ResNet-50x4 in the CIFAR10 and CIFAR100 benchmarks. However, this trend is not consistent across all datasets, as illustrated by the results on ImgNet-1k. Additionally, a higher number of parameters does not always correlate with better performance across different backbone families.

Notably, the Oracle method, which leverages all possible backbones, shows substantial improvements on several datasets: 43.5% on Eurosat, 36.0% on CLEVR, 25.1% on CUB, 21.8% on DTD, 16.6% on CIFAR100, and 16.0% on Cars. This clearly demonstrates the complementarity of each zero-shot CLIP backbone.

**Table B.2: Impact of artificial alterations of the input image on each backbone in terms of delta accuracy concerning the original performance**
|  | RN | ViT |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 50 | 50x4 | 50x16 | 50x64 | 101 | B-32 | B-16 | L-14 | L-14-336 |
| Resize | -3.81 | -3.47 | -1.90 | -1.25 | -4.05 | -4.41 | -2.43 | -2.82 | -2.41 |
| Flip | 0.12 | -0.36 | -0.36 | -0.55 | -0.67 | -0.22 | -0.12 | -0.80 | -1.01 |
| Grayscale | -6.36 | -7.37 | -5.98 | -7.01 | -7.52 | -8.94 | -8.70 | -7.73 | -6.99 |
| Color Jitter | -2.71 | -3.00 | -2.19 | -2.93 | -2.64 | -3.52 | -2.53 | -1.67 | -1.73 |
| Gaussian Blur | -5.12 | -3.72 | -1.96 | -2.06 | -4.49 | -3.17 | -3.83 | -3.21 | -1.80 |

In Table [B.2](#A2.T2), we present the results of the artificial image variations shown in Figure [6](#S2.F6). In this experiment, we selected a subset of each test set from the benchmarks and applied specific artificial image variations -i.e., Resize, Flip, Grayscale, Color Jitter, and Gaussian Blur- to evaluate the zero-shot classification performance on these “corrupted” images for each backbone.

The results indicate that ResNet-50 and ViT-B-16 are the most resilient to the Flip transformation, RN50x64 shows the highest resilience to Resize, and RN50x16 performs best for Grayscale. ViT-L-14 and L-14-336 exhibit the most robustness against Color Jitter and Gaussian Blur transformations. These findings suggest that each backbone has a unique ability to handle specific types of image variations.

## Appendix C Additional Baselines

To complete the experiments and establish a comprehensive baseline compared to other approaches, we introduce two non-parametric and one parametric more ensemble methods to our benchmarks:

#### Confidence ( Conf ):

This method utilizes the Shannon entropy to assess the confidence of each backbone in making a prediction. The backbone with the highest confidence for a given prediction is selected as the source of the final prediction.

#### Voting ( Vote T-1 and Vote T-3 )

: Voting combines different classifiers to predict the class label. In the Vote T-1 approach, we consider the top-1 prediction from each backbone, and the final prediction is determined by the label with the highest number of votes. In cases where multiple labels receive the same number of votes, we select the one with the highest probability. Additionally, we explore the Vote T-3 method, which seeks consensus among the backbones by considering the three most likely predictions from each backbone. These predictions are weighted based on their position within the top-3 list.

#### Calibrated confidence ( C-Conf )

For this approach, we first calibrate the probabilities of each backbone independently using temperature scaling. Then follow the procedure above and utilize the Shannon entropy of each backbone to select the one with the highest confidence

**Table C.1: Our results on ensembling the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques, and also compare to the best-performing single backbone (Single-Best). Mean, Max and Min $\Delta$ summarize the difference in performance when we compare it against the Single-Best backbone across datasets.**
|  | Caltech101 | Cars | CIFAR10 | CIFAR100 | CLEVR | Country211 | CUB | DTD | Eurosat | FGVC | Flowers | Food | GTSRB | ImgNet-1k | MNIST | PCam | Pets | SST2 | Resisc45 | STL10 | SUN397 | Mean $\Delta$ | Min $\Delta$ | Max $\Delta$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single-Best | 86.6 | 79.4 | 95.6 | 75.8 | 24.4 | 34.5 | 63.0 | 56.4 | 48.0 | 33.2 | 79.1 | 93.1 | 52.4 | 76.6 | 78.9 | 63.9 | 93.8 | 71.0 | 64.6 | 99.4 | 67.7 |  |  |  |
| Conf | 83.2 | 67.6 | 82.5 | 54.4 | 21.6 | 24.4 | 57.1 | 50.2 | 35.7 | 23.4 | 70.7 | 85.4 | 40.8 | 65.2 | 66.5 | 58.8 | 88.5 | 59.2 | 54.2 | 98.3 | 63.1 | -8.9 | -21.4 | -1.1 |
| Vote T-1 | 86.3 | 79.7 | 94.3 | 74.6 | 22.7 | 30.7 | 68.1 | 58.0 | 48.2 | 34.1 | 78.4 | 92.5 | 53.9 | 76.1 | 70.0 | 62.7 | 93.9 | 71.7 | 67.3 | 98.8 | 70.0 | -0.3 | -9.0 | 5.1 |
| Vote T-3 | 86.4 | 81.2 | 94.9 | 77.0 | 22.5 | 32.7 | 69.3 | 59.4 | 50.7 | 35.0 | 78.9 | 93.0 | 55.9 | 76.8 | 74.0 | - | 94.1 | - | 68.1 | 98.8 | 70.4 | 0.9 | -4.9 | 6.3 |
| Log-Avg | 86.9 | 80.9 | 94.9 | 75.0 | 23.6 | 31.3 | 69.8 | 57.7 | 49.4 | 35.3 | 77.9 | 92.9 | 54.8 | 76.6 | 74.8 | 62.3 | 94.2 | 72.6 | 68.3 | 98.8 | 70.9 | 0.5 | -4.1 | 6.7 |
| C-Log-Avg | 86.6 | 81.2 | 94.9 | 77.3 | 23.8 | 33.1 | 69.9 | 58.8 | 51.6 | 36.3 | 78.9 | 92.9 | 48.1 | 76.8 | 60.9 | 63.3 | 94.3 | 73.3 | 68.1 | 98.9 | 70.4 | 0.1 | -18.0 | 6.9 |
| C-Conf | 88.9 | 79.7 | 88.9 | 74.1 | 14.5 | 33.2 | 65.7 | 58.8 | 53.4 | 32.7 | 78.9 | 92.5 | 44.3 | 75.9 | 57.2 | 51.2 | 93.3 | 56.7 | 64.5 | 98.0 | 67.7 | -3.2 | -21.7 | 5.4 |
| GAC | 87.4 | 81.9 | 95.4 | 78.2 | 24.3 | 34.8 | 70.0 | 58.5 | 51.9 | 36.7 | 79.0 | 93.8 | 54.2 | 78.0 | 84.7 | 61.3 | 94.8 | 72.6 | 68.8 | 98.8 | 71.6 | 1.9 | -2.6 | 6.9 |
| Ours | 91.1 | 84.1 | 96.7 | 79.6 | 54.4 | 36.13 | 73.6 | 63.2 | 87.1 | 38.2 | 82.1 | 94.3 | 70.5 | 78.8 | 93.9 | 84.9 | 95.4 | 77.3 | 78.3 | 99.5 | 74.2 | 9.3 | 0.1 | 39.1 |

Table [C.1](#A3.T1) presents the baselines’ results in the combination of zero-shot CLIP backbones.
In non-parametric baselines, consistently with the presented in the main paper, leveraging the confidence of each backbone in its predictions consistently fails to enhance the overall performance beyond that of the best backbone.
The lack of calibrated probabilities in the Conf contributes to overconfidence in some backbones, resulting in performance degradation when combined.
Calibrating the confidence (C-Conf) leads to improvements, although the performance still falls short of matching the best backbone, except for DTD, Caltech101, Eurosat and Cars.
This trend persists in the Log-Avg approach, where averaging performance across backbones does not effectively exploit their complementarity between prediction, yielding an average delta accuracy of 0.1%. Notably, the Log-Avg approach shows substantial improvement for the CUB dataset compared to the best backbone. When the backbones are calibrated C-Log-Avg, the Log-Avg approach enhances its performances across all datasets except GTSRB, MNIST, Resisc45, and SUN397 when compared to the non-calibrated version. Intriguingly, conventional ensemble techniques such as Vote T-1 and Vote T-3 prove ineffective in providing a significant boost to prediction accuracy beyond that of the best backbone.

In employing parametric methods, we utilize the entire training set of the target datasets. Our proposed approach NLC demonstrates a noteworthy capability to enhance the performance of the best backbone, achieving a substantial improvement of up to 35.1% in the case of the Eurosat dataset. On average, our method exhibits a commendable improvement of 8.1% over the Single-Best backbone’s performance across the evaluated datasets.

## Appendix D NLC Using Linear classifiers of CLIP Backbones

**Table D.1: Linear classifier accuracy across multiple datasets and employing different backbones, showcasing the performance improvement achieved by the linear classifier compared to the zero-shot version. Also, the combination of Oracle of linear classifiers. The final three columns present the delta statistics with respect to the zero-shot version of each backbone. In the case of Oracle, the delta with respect to the best linear probing.**
|  |  | Caltech101 | Cars | CIFAR10 | CIFAR100 | CLEVR | Country211 | CUB | DTD | Eurosat | FGVC | Flowers | Food | GTSRB | ImgNet-1k | MNIST | PCam | Pets | SST2 | Resisc45 | STL10 | SUN397 | Mean $\Delta$ | Min $\Delta$ | Max $\Delta$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ResNet | 50 | 93.8 | 77.5 | 88.5 | 68.8 | 61.3 | 17.9 | 65.6 | 69.0 | 95.0 | 39.9 | 90.9 | 81.7 | 84.5 | 70.3 | 97.4 | 78.3 | 85.5 | 71.3 | 89.0 | 97.1 | 74.9 | 23.5 | -0.3 | 66.8 |
| 101 | 95.5 | 80.9 | 90.9 | 72.6 | 61.8 | 20.1 | 70.7 | 69.7 | 94.5 | 41.9 | 92.7 | 84.5 | 83.5 | 72.4 | 97.4 | 75.5 | 89.1 | 66.1 | 90.7 | 98.1 | 77.1 | 22.1 | 1.3 | 68.1 |  |
| 50x4 | 96.3 | 85.2 | 90.3 | 71.7 | 53.8 | 22.0 | 76.0 | 73.9 | 94.9 | 49.6 | 95.9 | 89.7 | 86.1 | 76.2 | 97.7 | 82.2 | 91.6 | 73.6 | 91.0 | 98.1 | 78.4 | 22.6 | 1.5 | 67.5 |  |
| 50x16 | 97.1 | 88.0 | 91.5 | 73.9 | 61.0 | 26.4 | 81.5 | 76.7 | 95.5 | 55.7 | 96.2 | 91.8 | 88.6 | 80.4 | 98.3 | 74.8 | 93.2 | 78.1 | 92.7 | 98.9 | 80.6 | 20.5 | 1.2 | 54.3 |  |
| 50x64 | 97.2 | 90.4 | 93.8 | 77.0 | 31.5 | 33.2 | 84.9 | 79.1 | 95.6 | 61.2 | 97.9 | 93.1 | 91.1 | 83.1 | 98.5 | 76.4 | 94.1 | 80.3 | 93.7 | 99.3 | 82.1 | 17.5 | 0.5 | 47.7 |  |
| ViT | B-32 | 95.8 | 76.8 | 94.8 | 77.6 | 59.1 | 18.8 | 70.9 | 71.8 | 95.1 | 41.3 | 92.6 | 83.2 | 85.6 | 73.0 | 98.0 | 77.6 | 87.7 | 69.9 | 90.1 | 98.5 | 74.6 | 20.8 | 0.2 | 59.9 |
| B-16 | 96.9 | 83.2 | 95.7 | 79.4 | 61.7 | 23.4 | 76.5 | 74.9 | 95.5 | 49.2 | 94.6 | 88.1 | 87.6 | 76.6 | 98.2 | 82.4 | 91.7 | 71.4 | 91.9 | 99.2 | 77.8 | 20.1 | 0.2 | 51.4 |  |
| L-14 | 97.6 | 89.1 | 98.0 | 85.5 | 68.9 | 32.2 | 83.3 | 78.3 | 97.1 | 60.9 | 98.4 | 92.4 | 92.6 | 82.2 | 98.8 | 75.5 | 94.6 | 79.4 | 94.6 | 99.8 | 81.4 | 18.2 | 0.1 | 49.9 |  |
| L-14-336 | 97.7 | 90.6 | 97.7 | 85.4 | 67.1 | 35.9 | 86.3 | 78.8 | 96.9 | 63.9 | 98.5 | 94.7 | 92.4 | 84.0 | 98.8 | 78.7 | 94.6 | 80.5 | 95.1 | 99.7 | 82.5 | 18.2 | 0.3 | 51.9 |  |
| Oracle | RN | 99.1 | 96.1 | 98.2 | 91.4 | 91.9 | 48.2 | 93.0 | 89.5 | 99.0 | 76.1 | 99.1 | 97.6 | 96.3 | 88.9 | 99.8 | 92.6 | 97.6 | 94.9 | 98.5 | 99.8 | 92.4 | 8.1 | 0.5 | 28.7 |
| ViT | 99.0 | 95.7 | 99.5 | 94.1 | 91.4 | 49.4 | 92.8 | 88.6 | 99.2 | 78.1 | 99.3 | 97.6 | 96.9 | 89.1 | 99.7 | 92.7 | 97.7 | 92.6 | 98.5 | 99.9 | 91.5 | 6.5 | 0.1 | 22.5 |  |
| All | 99.4 | 97.6 | 99.7 | 96.7 | 97.2 | 59.5 | 95.6 | 92.7 | 99.6 | 84.6 | 99.6 | 98.8 | 98.2 | 91.8 | 99.9 | 96.0 | 98.7 | 96.9 | 99.5 | 100.0 | 95.0 | 9.1 | 0.2 | 28.3 |  |

**Table D.2: Our results on combining the LinearProbe CLIP predictions with different backbones, which we group into non-parametric and parametric techniques. Mean, Max and Min $\Delta$ summarize the difference in performance across datasets.**
|  | Caltech101 | Cars | CIFAR10 | CIFAR100 | CLEVR | Country211 | CUB | DTD | Eurosat | FGVC | Flowers | Food | GTSRB | ImgNet-1k | MNIST | PCam | Pets | SST2 | Resisc45 | STL10 | SUN397 | Mean $\Delta$ | Min $\Delta$ | Max $\Delta$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single-Best | 97.7 | 90.6 | 98.0 | 85.5 | 68.9 | 35.9 | 86.3 | 79.1 | 97.1 | 63.9 | 98.5 | 94.7 | 92.6 | 84.0 | 98.8 | 82.4 | 94.6 | 80.5 | 95.1 | 99.8 | 82.5 | - | - | - |
| Vote T-1 | 98.2 | 91.3 | 97.2 | 87.1 | 71.0 | 37.4 | 86.7 | 81.3 | 97.6 | 63.8 | 98.1 | 95.0 | 93.3 | 82.6 | 99.2 | 80.4 | 94.8 | 80.5 | 95.8 | 99.6 | 85.0 | 0.5 | -2.0 | 2.5 |
| Vote T-3 | 98.2 | 91.4 | 97.2 | 87.0 | 72.0 | 37.4 | 86.5 | 80.9 | 97.7 | 64.0 | 98.1 | 95.0 | 93.4 | 83.4 | 99.2 | 80.4 | 94.7 | 80.5 | 95.9 | 99.6 | 85.0 | 0.5 | -2.0 | 3.2 |
| Conf | 96.5 | 86.0 | 91.5 | 73.7 | 57.3 | 24.1 | 80.7 | 73.5 | 96.5 | 52.9 | 94.8 | 87.9 | 87.9 | 80.0 | 97.8 | 79.6 | 91.4 | 73.6 | 91.2 | 98.7 | 79.4 | -5.3 | -11.8 | -0.5 |
| Log-Avg | 98.0 | 91.5 | 98.4 | 89.0 | 73.8 | 40.5 | 87.3 | 82.0 | 98.0 | 64.7 | 98.6 | 95.3 | 94.3 | 84.6 | 99.3 | 80.8 | 95.1 | 81.3 | 96.3 | 99.8 | 85.5 | 1.3 | -1.5 | 5.0 |
| C-Conf | 97.5 | 91.3 | 96.4 | 85.0 | 62.7 | 35.2 | 87.3 | 80.1 | 96.4 | 62.9 | 98.2 | 95.0 | 91.9 | 84.6 | 98.9 | 71.9 | 94.3 | 61.9 | 94.7 | 99.3 | 83.0 | -1.8 | -18.6 | 1.0 |
| C-Log-Avg | 98.3 | 91.9 | 97.2 | 87.2 | 73.2 | 38.3 | 87.5 | 81.3 | 97.7 | 65.2 | 98.2 | 95.2 | 93.4 | 83.3 | 99.2 | 80.6 | 94.9 | 80.5 | 96.0 | 99.6 | 85.0 | 0.8 | -1.7 | 4.4 |
| GAC | 98.1 | 92.3 | 98.2 | 88.6 | 73.3 | 40.7 | 88.3 | 82.0 | 51.9 | 65.3 | 98.6 | 95.2 | 94.3 | 84.8 | 99.2 | 82.6 | 95.1 | 81.1 | 96.3 | 99.8 | 71.6 | -1.4 | -45.2 | 4.8 |
| MoE | 96.5 | 90.5 | 97.6 | 87.1 | 89.6 | 41.2 | 84.8 | 72.7 | 96.9 | 43.0 | 96.1 | 94.9 | 93.4 | 93.4 | 98.8 | 82.3 | 93.4 | 79.1 | 94.5 | 98.8 | 85.0 | 0.2 | -20.9 | 20.8 |
| NLC | 98.3 | 91.8 | 98.3 | 88.8 | 77.0 | 40.8 | 87.2 | 79.4 | 97.8 | 66.4 | 98.6 | 95.4 | 94.4 | 84.7 | 99.3 | 83.4 | 95.0 | 80.9 | 96.3 | 99.8 | 85.9 | 1.6 | 0.0 | 8.1 |

The results of adapting CLIP using a linear classifier are shown in Table [D.1](#A4.T1). RN50 demonstrates the most significant improvement, averaging an impressive 23.5% increase across all datasets, surpassing even ViT-B-32. However, RN50 still falls short of outperforming other backbones overall. ViT-L-14 and ViT-L-14-336 continue to be the top performers across all datasets, even after linear probing.

A notable observation is the superior performance of the Oracle of linear probes compared to any individual backbone, highlighting the potential benefits of combining multiple backbones. Although the performance gap between the Oracle and the best linear probe is consistently observed across datasets, it is less pronounced than in the zero-shot scenario, indicating a smaller margin for improvement.

## Appendix E Robustness Under Distribution Shift of ImgNet-1k

**Table E.1: Detailed ImageNet robustness performance. IN is used to abbreviate for ImageNet**
|  | IN | IN-V2 | IN-A | IN-R | IN-Sketch |
| --- | --- | --- | --- | --- | --- |
| Best Backbone | 76.6 | 70.3 | 77.6 | 89.0 | 60.9 |
| NNC (Train on IN) | 78.8 | 72.4 | 75.6 | 89.3 | 61.8 |

To assess how well the combination of backbones performs under varying conditions, we test its robustness using natural distribution shifts in the ImageNet dataset. We evaluate its performance on four datasets representing different distribution shifts: ImageNet-V2 [pmlr-v97-recht19a], ImageNet Adversarial [hendrycks2021nae], ImageNet Rendition [hendrycks2021many] and ImageNet Sketch [wang2009learning]. Specifically, we employ a NLC trained on the ZeroShot CLIP backbones using the original ImgNet-1k data [deng2009imagenet]. This allows us to determine whether the learned combination of backbones can maintain its performance across these distribution shift datasets.
In Table [E.1](#A5.T1), we observe that the learned combination of backbones, denoted as NLC, enhances performance in 3 out of 4 selected benchmarks, with improvements ranging from 0.3% to 2.1%. However, in the case of ImageNet Adversarial, the performance of the combination of backbones appears to suffer, possibly due to a more complex decision boundary.

**Table E.2: Classification accuracy of models under few-shot settings.**
| Few-shot | 1 | 2 | 4 | 8 | 16 |
| --- | --- | --- | --- | --- | --- |
| Linear-probe CLIP [radford2021learning] | 22.2 | 31.9 | 41.2 | 49.5 | 56.1 |
| CoOP[zhou2022learning] | 47.6 | 50.9 | 56.2 | 59.9 | 63.0 |
| CLIP-Adapter[gao2021clip] | 61.2 | 61.5 | 61.8 | 62.7 | 63.6 |
| Tip-Adapter[zhang2021tip] | 60.7 | 61.0 | 61.0 | 61.5 | 62.0 |
| Tip-Adapter-F[zhang2021tip] | 61.3 | 61.7 | 62.5 | 64.0 | 65.5 |
| NLC | 78.2 | 78.1 | 78.2 | 78.3 | 78.4 |

## Appendix F The Combination of NLC with Tip-Adapter

We integrate the Tip-Adapter [zhang2021tip] with our NLC ensembling mechanism and conduct experiments on Eurosat, FGVC, DTD, and Pets datasets. Initially, we apply Tip-Adapter and Tip-Adapter-F independently on each backbone, following their protocol with 1, 2, 4, 8, and 16 shots. Subsequently, we employ our ensembling mechanism to fuse the adapted backbones. For training NLC, we utilize the validation set used by Tip-Adapter to combine their logits with CLIP linearly.

Table [E.3](#A5.T3), we show that current few-shot adapter methods also have performance differences with various backbones, and still, our NLC surpassed the best backbone reported by their method.

Table [F.2](#A6.T2), [F.2](#A6.T2), [F.4](#A6.T4), and [F.4](#A6.T4) show the zero-shot performance of the CLIP used by Tip-Adapter. It also shows the Tip-Adapter and Tip-Adapter-F performance on each backbone and when we combine all the Tip-Adapter versions and backbones with NLC. Across the four datasets used for this experiment, NLC improve the performance of each version of Tip-Adapter. Notably, NLC for Eurosat obtained an improvement of up to 15% with respect to the best Tip-Adapter backbone (L-14-336) using 1 shot.

**Table F.1: Tip-Adapter and Tip-Adapter-F fused with our NLC and applied to Eurosat dataset**
| ZeroShot |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ResNet | ViT |  |  |  |  |  |  |  |
| 50 | 50x4 | 50x16 | 50x64 | 101 | B-32 | B-16 | L-14 | L-14-336 |
| 37.5 | 32.0 | 40.3 | 49.4 | 32.5 | 45.2 | 47.6 | 58.1 | 63.5 |

**Table F.3: Tip-Adapter and Tip-Adapter-F fused with our NLC and applied to DTD dataset**
| ZeroShot |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ResNet | ViT |  |  |  |  |  |  |  |
| 50 | 50x4 | 50x16 | 50x64 | 101 | B-32 | B-16 | L-14 | L-14-336 |
| 42.1 | 50.8 | 50.3 | 48.4 | 38.4 | 43.7 | 44.6 | 53.1 | 54.0 |

## Appendix G Training NLC Under Limited Samples

**Table G.1: Ablation of NLC performance when changing the number of samples used to train. We use NLC($n$) to denote NLC with $n$ samples per class. and showcase the improvement in performance compared with Single-Best backbone in a zero-shot setting.**
|  | Caltech101 | Cars | CIFAR10 | CIFAR100 | CLEVR | Country211 | CUB | DTD | Eurosat | FGVC | Flowers | Food | GTSRB | ImgNet-1k | MNIST | PCam | Pets | SST2 | Resisc45 | STL10 | SUN397 | Mean $\Delta$ | Min $\Delta$ | Max $\Delta$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single-Best | 86.6 | 79.4 | 95.6 | 75.8 | 24.4 | 34.5 | 63.0 | 56.4 | 48.0 | 33.2 | 79.1 | 93.1 | 52.4 | 76.6 | 78.9 | 63.9 | 93.8 | 71.0 | 64.6 | 99.4 | 67.7 |  |  |  |
| NNC(1) | 87.3 | 82.3 | 93.6 | 78.0 | 25.1 | 36.0 | 71.0 | 59.3 | 55.0 | 37.1 | 79.4 | 93.6 | 54.8 | 77.6 | 83.3 | 62.5 | 94.3 | 73.8 | 70.0 | 98.9 | 71.1 | 2.2 | -2.0 | 8.0 |
| NNC(2) | 87.3 | 81.4 | 93.0 | 78.2 | 25.1 | 33.2 | 70.0 | 59.9 | 52.3 | 37.2 | 79.5 | 93.5 | 55.3 | 77.4 | 84.4 | 62.5 | 94.5 | 73.1 | 70.2 | 98.9 | 71.1 | 1.9 | -2.6 | 7.0 |
| NNC(4) | 87.1 | 81.3 | 95.3 | 75.9 | 26.9 | 35.4 | 70.2 | 59.9 | 54.0 | 35.9 | 78.2 | 93.1 | 55.2 | 78.1 | 81.6 | 62.7 | 94.5 | 72.7 | 68.8 | 99.1 | 71.2 | 1.9 | -1.3 | 7.2 |
| NNC(8) | 87.1 | 81.6 | 95.6 | 76.2 | 26.9 | 36.1 | 70.3 | 58.1 | 54.0 | 37.0 | 80.3 | 93.1 | 54.9 | 78.3 | 82.5 | 67.0 | 94.2 | 72.4 | 68.9 | 99.5 | 71.2 | 2.3 | 0.0 | 7.3 |
| NNC(16) | 87.1 | 82.8 | 95.8 | 77.2 | 27.0 | 36.2 | 71.3 | 60.4 | 53.5 | 37.0 | 80.4 | 93.5 | 55.2 | 78.4 | 83.3 | 69.6 | 94.2 | 73.6 | 69.6 | 99.4 | 72.2 | 2.9 | 0.0 | 8.3 |

In our exploration of the effectiveness of the NLC approach, we extend our analysis to a scenario where we limit the samples to combine the zero-shot CLIPs. This experiment allows us to assess the adaptability and performance of our proposed method under limited training data conditions. Table [G.1](#A7.T1) presents the performance of NLC by means of a limited number of samples. Although the performance of NLC overall improves when it has more data available to combine the backbones, in most cases, just using one sample NLC(1) per class is enough to improve its performance. Notably, there is a stop in performance degradation when we use more than 8 samples NLC(8) in all the benchmarks.

## Appendix H Learned Temperature Values

Figure [H.13](#A8.F13) and [H.23](#A8.F23) present the distribution of the temperature values using a box plot for the NLC method, normalized by their maximum value. Notably, there is a dominance in the temperature values for the best backbones in each family, i.e., RN50x64 and ViT-L-14 or ViT-L-14-336. Particularly prominent in STL10, Pets, DTD, FGVCand Flowers, making these backbones especially relevant for NLC. Interestingly, it is observed that the most weighted backbone within the ResNet family is not consistently ResNet-101, despite its deeper architecture. This observation is evident in datasets such as Pets, CUB, FGVC and SUN397, where the mean value of the temperature corresponding to ResNet-50 surpasses that of ResNet-101.

Furthermore, the distribution of the temperature values across backbones for datasets such as CLEVR, Eurosat, and GTSRB is more uniform compared to other datasets. This suggests that the NLC method is effectively leveraging the strengths of each backbone to arrive at accurate labels for each sample, resulting in a more balanced distribution of weights across different backbones, complementing our analysis on Fig. [3](#S1.F3).

Figure: Figure H.1: Caltech
Refer to caption: /html/2405.17139/assets/x8.png

Figure: Figure H.14: GTSRB
Refer to caption: /html/2405.17139/assets/x20.png

## Appendix I Overlap Diagrams For Other Datasets.

In this section, we present the linear Venn Diagrams for each of the other datasets used in the experiment section Caltech (Figure [I.24](#A9.F24)),
Cars (Figure [I.25](#A9.F25)),
CUB (Figure [I.30](#A9.F30)),
CIFAR10 (Figure [I.26](#A9.F26)),
CIFAR100 (Figure [I.27](#A9.F27)),
CLEVR (Figure [I.28](#A9.F28)),
Country211 (Figure [I.29](#A9.F29)),
CUB (Figure [I.30](#A9.F30)), DTD (Figure [I.31](#A9.F31)),
Eurosat(Figure [I.32](#A9.F32)),
FGVC (Figure [I.33](#A9.F33)),
Flowers (Figure [I.34](#A9.F34)),
Food (Figure [I.35](#A9.F35)),
GTSRB (Figure [I.36](#A9.F36)),
MNIST (Figure [I.37](#A9.F37)),
PCam (Figure [I.38](#A9.F38)),
Pets (Figure [I.39](#A9.F39)),
RenderedSST2 (Figure [I.40](#A9.F40)),
Resisc45 (Figure [I.41](#A9.F41)),
STL10 (Figure [I.42](#A9.F42)), and
SUN397 (Figure [I.43](#A9.F43)).
We can see that in each dataset, the CLIP backbones present possible complementarities that could be exploited.

Figure: Figure I.24: Caltech101 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x29.png

Figure: Figure I.25: Overlap diagrams for the Cars dataset with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x32.png

Figure: Figure I.26: Cifar10 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x35.png

Figure: Figure I.27: Cifar100 Venn diagrams with the correct prediction of each backbone. The Top part of the Venn diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Venn diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x38.png

Figure: Figure I.28: Clever Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x41.png

Figure: Figure I.29: Country211 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x44.png

Figure: Figure I.30: Cub Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x47.png

Figure: Figure I.31: Dtd Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x50.png

Figure: Figure I.32: Eurosat Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x53.png

Figure: Figure I.33: FGVC Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x56.png

Figure: Figure I.34: Flowers Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x59.png

Figure: Figure I.35: Food Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x62.png

Figure: Figure I.36: GTSRB Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x65.png

Figure: Figure I.37: MNIST Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x68.png

Figure: Figure I.38: PCam Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x71.png

Figure: Figure I.39: Pets Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x74.png

Figure: Figure I.40: RenderSST2 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x77.png

Figure: Figure I.41: Resisc45 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x80.png

Figure: Figure I.42: STL10 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x83.png

Figure: Figure I.43: SUN397 Overlap diagrams with the correct prediction of each backbone. The Top part of the Overlap diagram shows the number of backbones that are predicting correctly a set of images. Each column represents a set of image instances that are predicted correctly by some group of backbones. Each row in the diagram shows in colour the backbone that correctly predicts a certain set of image instances, in grey when the backbone is not correctly predicting those instances. The bottom part of the Overlap diagram shows the number of images in a certain set. The right part is the total amount of correctly predicted images per backbone.
Refer to caption: /html/2405.17139/assets/x86.png

## Appendix J Possible Combinations

Tables [J.1](#A10.T1), [J.2](#A10.T2), [J.3](#A10.T3), [J.4](#A10.T4), [J.5](#A10.T5), [J.6](#A10.T6), [J.7](#A10.T7), and [J.8](#A10.T8) present the results of possible combinations of backbones using the non-parametric and parametric approaches proposed in the paper. Notably, the performance of NLC consistently emerges as the best across various backbone combinations and datasets when compared to other methods.

Notably, instances exist where the combination of specific backbones yields a more substantial performance boost than utilizing all backbones together. For instance, in the Pets dataset, combining ResNet 50, 101, and ViT-B-32 results in a delta improvement of 2.37%, surpassing the 0.99% improvement achieved by using the five backbones selected for this experiment. This phenomenon is consistent across datasets with different backbone combinations. In Cars, there is a boost of 5.71% when combining ResNet-101 and ViT-B-32, compared to the 2.55% boost when using all five different backbones. While the best delta improvement among backbones may not necessarily come from combining all backbones, the best overall accuracy is consistently obtained when using the combination of all backbones.

Table: Table J.1: Our results on Pets dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.2: Our results on Cars dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.3: Our results on CUB dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.4: Our results on DTD dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.5: Our results on FGVC dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.6: Our results on Food dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.7: Our results on Flowers dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

Table: Table J.8: Our results on ImgNet-1k dataset for all the possible combinations of combining the zero-shot predictions of CLIP backbones, which we group intro non-parametric and parametric techniques. Also, the best-performing single backbone (Single-Best) and the Oracle performance. We present, for each combination of backbones, the improvement , constancy $-$ and deterioration of accuracy performance for each method when we compare it against the Single-Best backbone. Mean, Max, and Min $\Delta$ summarize the difference in performance across methods and backbone combinations.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.1: How is the training data for the ensemble module curated and how is it split from other data? Please provide more details.

1.2: When the training data for the ensemble module is only available in some classes, does it generalize well to novel classes? Also, when training with only some fixed prompts, does it overfit these prompts?

1.3: Continue to 1.2, considering more complex concepts (eg, [ARO](https://github.com/mertyg/vision-language-models-are-bows) or [SugarCrepe](https://arxiv.org/abs/2306.14610) datasets), can the ensemble module correctly rerank CLIP models on novel compositions of concepts?

2.1: When trained on the same dataset, different backbones in the same series (eg, R-50/101/50x4, ViT-S/B/L) make different mistakes. Does this finding only apply to CLIP or also apply to other methods (eg, vanilla supervised learning)?

2.2: Is diversity always preferred when selecting models in a per-sample way, no matter whether using CLIP or not?

3.1: When inferencing with the proposed method over a dataset, does the user need to load all CLIP models to the device? How much memory is needed compared to using single models, and how much inference time is needed?

3.2: Can the authors also provide a comparison in computation and time cost with other ensemble methods (eg, SL, MoE)?

Minor: L815: OpenCLIP is a project developed by LAION. The models mainly considered in this paper are trained by OpenAI, and the corresponding dataset is called WIT-400M.
soundness: 3
strengths: - This paper presents a finding that the logits of CLIP models are very diverse, and the more diverse their logits are, the higher performance can be reached by their ensemble.
- This paper presents a comprehensive study on the diversity and properties of CLIP's logits and possible factors behind them.
- The results produced by the ensemble model are considerably higher than single models. When compared with other ensemble methods, there is also some improvement.
- The paper is clearly written and is easy to understand.
summary: This paper starts with the finding that existing CLIP models show diverse behaviours (error distributions) on classification tasks, even if they are pre-trained on the same dataset (WIT-400M). This indicates potential for model ensembling. The authors then propose a learnable ensemble module that adjusts the logits of each model with a learned temperature value conditioned on the input image, and sums up the logits of all models for the final prediction. This can also be combined with an existing cascade strategy for better efficiency. Experiments show consistent improvements over the best single backbones.
weaknesses: - The setting of training the ensemble module might harm generalizability and more clarifications are needed. It requires training data for each class regarding which model is better for each evaluation dataset, which is ad-hoc and remains questionable whether it harms CLIP's ability to generalize to unseen classes, concepts, or compositions.
- It is not very surprising that more diverse models enable better ensembles, especially when the oracle model is picked per-sample, this adds a lot of flexibility compared to ensembling without data conditions.
- The method itself is related to one prior method (Super Learner), and the main improvement is adding image conditions. But when compared with SL, the performance improvement is not big.
- From a practical viewpoint, how much additional inference-time computation is required? I see the results of FLOPS, and it would be better also to report GPU memory and inference time.

More detailed classifications are elaborated in the questions section.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section above.
soundness: 3
strengths: 1. Training large-scale backbone requires high computational cost. Therefore, thinking about how to combine existing pretrained backbones is a valuable research topic. Ensemble is a reasonable way to do so.
2. Empirical results are extensive for image classification task including several datasets.
3. Instead of making ensemble directly on logit from different backbone versions, it proposes a learnable module based on temperature scaling.
4. Overall, the paper writing is in a good shape and easy to follow.
summary: This paper focuses on vision-language backbones, especially two-tower CLIP like network structure. It evaluates the classification tasks across several datasets. The main idea is to propose an ensemble strategy in logic space by adaptively learning the temperature across different versions of pretrained CLIP backbones. Empirical results are based on a large group of image classification datasets. It also provides detailed analysis for a better intuition.
weaknesses: 1. More evaluation tasks are valuable to be considered to further support the paper statement. Even if limited with CLIP backbone, adding more retrieval tasks will be supportive.
2. Ensemble on logit is still relatively not novel enough as a research work. Compared with previous ensemble, it still relies on the output of each individual model and make the late fusion.
3. Such ensemble increases the performance but requires much more test time, and it also requires to conduct a post-learning process. The overall pipeline is not that straightforward.
4. It looks like the proposed pipeline cannot obtain improvement by adding more training shots, more analysis and intuition will be helpful.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - **(Q1)**: As mentioned above, here is the question raised earlier: looking at the results and in particular the difference in performance between Fig. 8. and Fig. 9. I might be wrong but can the decrease of outperformance from 9.1% to 1.6% be linked to the additional supervision signal intrinsic to the proposed NLC method? Is this what we see here? And if yes, could the results in Fig. 8 be biased?

- **(Q2)**: why is the MoE missing in Fig.8? I assume that the linear heads from the experiment of Fig.9 are serving as MoE?

- **(Q3)**: on page 6, lines 298, with one-layer MLP, do you mean a MLP with one hidden layer?
soundness: 3
strengths: - **(S1)**: this paper presents an in-depth analysis of different CLIP-trained vision backbones with respect to different dimensions such as few-shot image classification, complementarity, robustness, and diversity.

- **(S2)**: this paper proposes a novel method to combine the strengths and weaknesses of available CLIP-trained vision backbones able to outperform regular ensembling methods on a large amount of datasets.

- **(S3)**: this paper is easy to read and follow given the systematic build-up of the paper.
summary: This work investigates the differences across various CLIP-trained vision backbones with respect to classification performance and robustness, and proposes an adaptive combination of pretrained CLIP backbones, which is able to increase performance beyond regular enabling.

I very much appreciate the author's systematic approach: they first show that there exist (large) differences between (zero-shot) classification performance of individual CLIP-trained vision backbones. Then they drive an oracle performance upper-bound and investigate  complementarity of different backbones concluding that different backbones classify different subsets of images differently. This motivates the authors to propose an adaptive ensembling approach called Neural Logit Controller (NLC) to combine individual backbones to one prediction.

The proposed NLC is based on network calibration using learned temperature scaling. To do so, the method needs to be informed by a supervised signal given by a single labeled example per class, as the authors claim.

Experimental results on a large collection of datasets show that the proposed approach is able to outperform state-of-the-art few-shot methods. For some datasets by large (up to 39.1), on average by 9.1% over the best performing individual backbone and better than baseline ensembling methods. However, outperformance is much smaller when data-specific linear heads are trained (=1.6%). Additional results show a promising combination of Tip-Adapter with the proposed approach.

As mentioned before, I very much appreciate the systematic evaluation of the growing number of available vision foundation models. Having so many foundation models available, renders the question how *foundational* these models are. So maybe the research community should invest more time in the investigation and how to harness these foundation models. To do so, we would need to understand their differences, strengths and weaknesses and then derive from that propose a novel aggregation, ensembling or merging methods. ...and this is what this paper is doing.

I also very much appreciate the large number of experiments and evaluations done by the authors to support this work - including the additional results in the appendix. One is just one thing, looking at the results and in particular the difference in performance between Fig. 8. and Fig. 9. I might be wrong but can the decrease of outperformance from 9.1% to 1.6% be linked to the additional supervision signal intrinsic to the proposed NLC method? Is this what we see here? And if yes, could the results in Fig. 8 be biased towards the supervision part of NLC? I will raise it again in the question section.


----------------------reply to rebuttal----------------------


I would like the authors for the information provided in the rebuttal. I went through the paper again and look in particular at the appendix and other reviewer / author discussions and given that I would like to increase my rating for this submission. I think this work contributes to the research community with its detailed evaluation and proposed ideas.
weaknesses: - **(W1)**: Maybe the large out-performance of the proposed method from Fig.7 is linked to the injection of a supervised signal and therefore the comparison against regular ensembles might be biased. I have raised this in **(Q1)**.

- **(W2)**: I would love to see more details about the injection of the supervision signal given the one labeled example per class. I am not sure if this is what I can find in appendix G and Table G.1?

- **(W3)**: not really a weakness but some minor things:
- - page 3, line 134: train a pair of vision and text encoders φ_l and φ_v <- I think the first is the language and the second is the vision encoder.
  - Fig 4: it is not clear which setup corresponds to the dashed lines.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What is the standard deviation of the accuracy when using different subsets of data for training the methods in few-shot setting? This is important to understand the robustness of the method when using different training samples.
soundness: 3
strengths: 1. The paper shows that smaller models have certain samples that they can predict correctly, and larger models can give incorrect predictions on these samples. This is an interesting observation, and highlights the need for careful thought when scaling models.

2. The proposed NLC is simple but effective, and can be trained in a data-efficient manner as demonstrated in Section 4.2 and 4.3.
summary: This paper analyzes the prediction diversity for CLIP models from OpenCLIP by analyzing the overlap between the sets of correctly predicted images from the test set of ImageNet1k via zero-shot classification. The paper finds that model size increases accuracy, but there exist images that are correctly predicted by smaller models (within same family as well). To combine the models, the paper proposes a method to combine logits of different backbones by scaling the logits using a learned temperature value per backbone. Temperature scaling is implemented usign a neural network called Neural Logit Controller (NLC) which takes the concatenated features of all the backbones and predicts the temperature. NLC is trained using a standard cross-entropy objective, and can be trained using only one image per class. The method boosts the zero-shot and linear probing performance on a number of downstream tasks, and the performance boost correlates with prediction diversity.
weaknesses: 1. The authors claim on line 200 that the results are non-obvious is not entirely true. Prior work has shown similar results across vision and language domains, albeit on different models [1, 2, 3]. [1] have shown prediction diversity on a range of models trained in a supervised setting and propose a method for combining models using distillation. They show that weaker models are more specialized in certain subsets of data. While the metric for measuring diversity is different, they study the same core problem. [2] show that finetuning models on different domains and then averaging model weights can improve out-of-distribution performance (this assumes the same model size, but studies a similar problem of different models being good at different subsets of data).

2. **Incomplete comparisons in few-shot experiments (Section 4.3)**:
- The results in Section 4.3 (Comparison with Few-Shot Adapter Methods) are incomplete: Figure 10 and Table 1, which show the zero-shot performance of TiP-Adapter applied to individial backbones and NLC does not show the complete picture - since NLC is an ensemble method, a proper comparison would be other ensemble methods. Similar is the case when NLC is combined with Tip-Adapter. In principle, any ensemble method can be combined with Tip-Adapter, so a proper comparison would be with other ensemble methods applied in a similar manner as NLC. Without these results, the comparisons are incomplete. 

3. **Number of training samples**:
- Point 2 is especially more important considering that few-shot experiments in Section 4.3 are the only experiments which demonstrate the data efficiency of NLC, which is claimed to be a major contribution by the authors. From Section 4.1: "The non-parametric ones are “static” while the parametric ones use the training split of each target dataset to adjust the ensemble adaptively." implies that NLC uses the entire training set for calibration. For few-shot experiments, if this data efficiency is applicable to other ensemble methods as well, this cannot be attributed to NLC, but would be a general property of ensemble methods.
- From linear probing experiments in Section 4.2, Figure 9 shows that with a similar training budget, NLC does not provide a significant improvement over Super Learner (SL). Mean increase of 1.6 is the same, and SL also does not lead to any degradation over the single best model. The max increase seems to be a noisy metric to track, since MoE can show a maximum 20.8 increase but also shows degradation, indicating the max increase is really dataset specific.

4. The paper claims that the method can be combined with Cascade, but does not provide the trainiing details. The paper also does not compare the performance of the proposed method with the original Cascade method. Minor: The paper also misses a citation for a cascade method [4].

5. Although the paper mentions this in the limitations, fusing models at training stage has been done in [1] in supervised classification settings, hence the additional inference overhead of NLC is a major practical limitation. Furthermore, while the paper shows the potential for applying the method to other tasks like image retrieval, the paper does not actually study this using NLC, making the impact of the method limited.

**References**:

[1] Karsten Roth, Lukas Thede, A. Sophia Koepke, Oriol Vinyals, Olivier J. Hénaff, Zeynep Akata: Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model. ICLR 2024

[2] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, David Lopez-Paz: Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization. ICML 2023

[3] Ruiqi Zhong, Dhruba Ghosh, Dan Klein, Jacob Steinhardt: Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level. ACL/IJCNLP (Findings) 2021: 3813-3827

[4] Neeraj Varshney, Chitta Baral: Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems. EMNLP 2022: 11007-11021

Meta-review: Summary
The paper investigates different CLIP models in terms of the model size and classification accuracy. The key finding of this work is that while overall accuracy is correlated with model size, there are certain samples the smaller models classify accurately vs. the larger models. The authors propose a Neural Logit Controller (NLC) to combine different predictions from different CLIP models. The approach is evaluated on multiple different datasets.

Strengths
1. The key finding in this paper of smaller models helping larger models is worth reiterating especially in the context of the common ML "wisdom" of larger models are better. The solution (NLC) is simple and effective and is likely going to benefit the community.
2. Good experimental setup. The authors analyzed a variety of different OpenCLIP models.
3. The paper is well written and easy to follow.

Weaknesses
1. The primary task studied in this work is image classification. It would make the paper stronger if the authors studied a more diverse range of tasks.
2. NLC requires late fusion and thus the compute cost scales with the number of models being ensembled

Justification
The paper received a majority positive reviews. The paper is technically strong, simple, studies a relevant problem and makes a good contribution to the field.

Decision: Accept (Poster)