Asymmetric Duos: Sidekicks Improve Uncertainty
Authors: Tim G. Zhou
Evan Shelhamer
Geoff Pleiss
Venue: NeurIPS 2025 spotlight ()
Keywords: Uncertainty Quantification
Ensembles
Deep Learning
Transfer Learning
Primary area: deep_learning

Abstract: The go-to strategy to apply deep networks in settings where uncertainty informs decisions—ensembling multiple training runs with random initializations—is ill-suited for the extremely large-scale models and practical fine-tuning workflows of today. We introduce a new cost-effective strategy for improving the uncertainty quantification and downstream decisions of a large model (e.g. a fine-tuned ViT-B): coupling it with a less accurate but much smaller "sidekick" (e.g. a fine-tuned ResNet-34) with a fraction of the computational cost. We propose aggregating the predictions of this *Asymmetric Duo* by simple learned weighted averaging. Surprisingly, despite their inherent asymmetry, the sidekick model almost never harms the performance of the larger model. In fact, across five image classification benchmarks, and a variety of model architectures and training schemes (including soups), Asymmetric Duos significantly improve accuracy, uncertainty quantification, and selective classification metrics with only ${\sim}10-20$% more computation. Code is available at: https://github.com/timgzhou/asymmetric-duos

Asymmetric Duos: Sidekicks Improve Uncertainty
Tim G. Zhou 1.2+
Evan Shelhamer' 1.2
Geoff Pleiss 1.2
University of British Columbia ? Vector Institute
Abstract
The go-to strategy to apply deep networks in settings where uncertainty informs decisions-ennembllng multiple training runs with random initializations-is ill- suited for the extremely large-scale models and practical fine-tuning workflows of today. We introduce new cost-effective strategy for improving the uncertainty quantification and downstream decisions of a large model (e.g. fine-tuned ViT- B): coupling it with a less accurate but much smaller "sidekick" (e.g. a fine-tuned ResNet-34) with fraction of the computational cost. We propose aggregating the predictions of this Asymmetric Duo by simple learned weighted averaging. Surprisingly, despite their inherent asymmetry. the sidekick model almost never harms the performance of the larger model. In fact, across five image classification benchmarks and variety of model architectures and training schemes (including soups), Asymmetric Duos significantly improve accuracy. uncertainty quantifica- tion, and selective classification metrics with only 10 20% more computation. Code is available at: http:///tttbb..ooommmmm///////et
Introduction
Deep ensembles 22 have endured de facto method in settings that require uncertainty quan- tification (UQ) for safety or downstream decision making 4 g 31.. When models are trained from scratch with different random seeds, this simple strategy yields strong performance across UQ
and decision decision-making benchmarks (22] 39). provides robustness under distribution shifts [42]. and
requires no modification to model training methodology.
However, as the modern machine learning paradigm increasingly revolves around larger and larger pre-trained models fine-tuned for specific downstream tasks [12] e.g.J. deep ensembles become in- creasingly ineffective and impractical. First, unlike training models from scratch, fine-tuning from a pre-trained model is much less sensitive hyperparameters and training randomness. While ensem- bling multiple fine-tuning runs can yield performance increases, it pales in comparison to the gains obtained when ensembling
sccassc models [19] 49 Second, and more critically, the growing size of modern models makes ensembling more expensive than ever. Even with moderate- sized models like ResNet-50 (24), training and inference costs were concern
26 with large architectures such as ViT-H [10]. these costs become even more prohibitive. Deploying a single one of these models can approach the limit of practical resources. Ensembling with double, triple, or larger multiples of the computation may be out of the question.
The smallest deep ensemble requires two models, which with standard approaches needs 2x the computation for both training and inference. In this paper, we reduce this cost to a fractional amount (e.g. 10 20%) while still obtaining meaningful improvements accuracy, uncertainty quantifica- tion, and downstream decision making. Our Asymmetric Duo framework pairs a base model with smaller "sidekick" to make better predictions together. For instance, we can pair larger ViT-B with a smaller ResNet-34, while fine-tuning each separately, to improve over either alone. As the
tgzhouucs, ubc. ca
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Single Model
Asymumetri Duo compute
Deep Enzembble
Duos (ours): IND
Duos (oursh: OOD
Baseline
compute
mox compute
ACTROC
20%
AUBIC
AUTROC
ATraC
20%
Fume
emal
DPS
loeur
Ace
And
Ace
NLL
NL. Caltach 296
Weighted Average
Simple Average
ImageNet
iWildCam
Figure 1: Overview of Asymmetric Duos. (Left) Schematic of Duos vs. single models and deep ensembles. (Right) Gains from Duos where the sidekick adds only 10%-20% more FLOPs depend- ing on the choice of models. Asymmetric Duos improve upon their base models across accuracy, uncertainty quantification, and selective classification metrics on in-distribution (IND) and out-of- distribution (OOD) data (see Sec. for experiment details), We plot relative improvements toward perfect scores; shading marks the standard deviation across the choice of base and sidekick models.
sidekick model requires fraction of the compute at training and inference time, it only requires minimal computational overhead over the base model, which enables practical use.
Intuitively, an Asymmetric Duo might not help or might even hurt. The smaller model's accuracy can might expect that combining predictions from both models could degrade
be much lower, and performance. Yet surprisingly, we find that Asymmetric Duos consistently improve performance, even when the base model and sidekick are extremely imbalanced (i.e. the sidekick is 10x smaller and 80% as accurate as the base model). Across datasets and model families, Asymmetric Duos yield up to 10- 15% improvements in accuracy, uncertainty quantification, and selective classification metrics, while only adding 10-20% of the computational cost of the base model.
The Asymmetric Duo framework is straightforward to implement: given working base model, the user adds a second smaller model (by say simply downloading it from a "model zoo") and fine-tuning it in the same way the base model. Moreover, Asymmetric Duos complemen- tary to other compute-efficient model combinations, like Model Soups [60] and apply broadly by incorporating these methods base sidekick models. Our experiments show that Duos im- prove the uncertainty quantification of both uncertainty-biin pipelines (like standard training) and uncertainty-aware pipelines (like soups), and so offer practical drop-in addition existing models.
2 Background
We focus on deep networks fine-tuned for a k-way classification task in some input domain X. However, the key ideas of our Duos could be extended to regression tasks or other problem settings.
We denote deep networks as mappings of the form f X R& where output logits s(X) are mapped to the k-class probability simplex via the softmax function o(Z) = [e Σ 2
The deep network's 's class prediction for an input X is the index of the largest logit (i.c. Y()) = arg max; [f [X]]). Its predictive uncertainty can be quantified via its associated softmax probability:
un)(f(X) 1 (f
(We note that alternative measures of predictive uncertainty, such the entropy of the categori- cal distribution defined by (f))). are common in other works [2 29] We explore entropy uncertainty in Appendix C and find no meaningful difference with respect to our main results.)
2.1 Deep Ensembles
A deep ensemble 29 refers m independently-trainn trained deep networks f1..... typically of identical architecture and training procedure. At test time, each input X is passed through each network and the resulting logits are averaged to produce an aggregate prediction f(X) == 1 > Ja(X) from which class prediction and uncertainty estimate can be derived. The ran- domness of initialization and shuffling of training samples is usually sufficient produce diverse logits that, when averaged together, produce more accurate class predictions and better calibrated

---

uncertainty estimates than any of the component models in isolation (29 42 Deep ensembles have and have thus been adopted across
also been shown be robust to distribution shifts 32
42
safety-critical and decision-making applications 4 9
31
As stated in the introduction, deep ensembles are often impractical with the modern transfer learning setup [41 49 60). The primary downside is computational cost. It is common for practitioners to fine-tune large pre-trained models, typically downloaded from "model zoo", and training or de- ploying multiple of these large models can be prohibitively expensive 77]. In addition, member mod- els fine-tuned from a shared pre-trained model tend to settle within the same loss basin, preventing the necessary diversity to produce meaningful improvements [40]. Consequently, deep ensembles
are often restricted settings with moderate-sized models trained from scratch [29 42 49
2.2 Alternative Methods
Many methods aim to replicate the benefits of ensembles without the computational overhead.
Model soups [60] average the parameters of independently fine-tuned models, rather than their predictions, yielding a "weight-space" ensemble with the same inference cost as a single model. This method takes advantage of the fact that models fine-tuned from an identical base model often settle into the same pre-trained loss basin, and thus the loss is approximately locally convex with respect to the parameters. This parameter averaging often yields competitive performance with the prediction averaging of ensembling when the component models are fine-tuned, though the gain is less than ensembling
models.
Shallow ensembles [30 reduce the multiple model training and inference costs of deep ensembles by instead making multiple predictions from a single model. Specifically, the last layer (or the last & layers) of a deep network are replaced with m copies that each receive the same input. The outputs of the "heads" are averaged together and learned jointly. Since only the last layer is duplicated, the computation and parameters added by a shallow ensemble are only a fraction of that of the original deep network. However, the benefits provided by shallow ensembles are weaker and less consistent than those of deep ensembles. In fact, single model baselines sometimes outperform them on selective classification tasks and robustness under distribution shifts (39]
Other methods. Implicit ensembles aim to produce multiple diverse networks at inference that are all derived from single training run. These networks derived through Monte Carlo sampling (c.g. when combined with Dropout [33), checkpoints from the training process (26). or predic- tions made early layers [1. While these approaches have lower training costs than standard deep ensembles, they retain the same inference costs and typically are incompatible with fine-tuning. Weight-sharing ensembles aim to embed multiple models within a single architecture that can be trained simultaneously [1] 53 58 e.g.]. While some methods incur lower inference-time costs 23 they typically require from-scratch training and bespoke architectures and/or training tech- niques. Non-ensemble methods often modify architectures, training, or inference; while they can be effective, these modifications may complicate practical use. See [5 44 for thorough surveys.
3 Asymmetric Duos
We introduce Asymmetric Duos to augment large base model fiarge with small sidekick model fsmall, which are both independently pre-trained and fine-tuned, for improving inference at little computational cost. Duos allow many choices of base fiarge and sidekick Ssmall- It is practical to choose a base that could be deployed by itself for adequate predictive performance and aggregate it with a much smaller sidekick, even if the sidekick cannot satisfactorily address the task on its own.
We contrast our pairing of asymmetric models with deep ensembles. Even the smallest deep ensem- ble with
2 models doubles the training and inference cost. To sidestep this cost, we instead look to combining models of different sizes. At first glance, this strategy may appear to have limits; too much imbalance may produce worse predictions than the base model alone. However, through a carefully designed aggregation strategy, we show that our Asymmetric Duos can improve perfor- mance even when the fsmall uses an order of magnitude less computation than faagee
The Asymmetric Duo framework learning and regularization techniques. This flexibility offers two key advantages. First, Duos can
and compatible with current transfer-
3

---

more readily adjust the amount of computation, compared to standard ensembling, by extending to combinations of models with vastly different sizes, which relevant given the wide span of pre- trained models available today. Second, Duos only add small sidekick model to a larger base model, and so the additional computation merely a fraction of that of the larger model.
3.1 Aggregating Predictions from Asymmetric Models
In standard deep ensembles, all models are inherently "equal" in that they only differ in randomiza- tion. By contrast, the two networks in Asymmetric Duo explicitly differ in size and predictive performance. For example, an Asymmetric Duo might combine a ConvNeXt-Ban (89 million pa- rameters, 15 billion FLOPs, 84.1% Top-1 accuracy on ImageNetIK] [35 and RegNeXt-1.6GE (9 million parameters. 1.6 billion FLOPs, 79.7% Top-1 accuracy on ImageNet [6). While this asymmetry achieves computational efficiency, the larger model is more likely to be correct than the other. Aggregating predictions with the equal-weighted logit average of deep ensembles could therefore be detrimental.
While many mechanisms could account for the asymmetry between fsmall and fharpe when aggre- gating predictions, propose simple weighted average of their logits. Specifically, we include two temperature parameters, Tunge and Tmmlll to weight the logits:
S (X) Stagge (X)r Thrree + SXman(X) - Tmmall-
The Duo class predictions and uncertainty measure are based on these weighted logit averages:
Duo (X) arg max[fuuo(X]]] (X)],, unc(fuwo (X))
Tuning the temperatures. We tune Tiage and Tsmall to minimize the negative log likelihood on the same set used for hyperparameter selection during fine-tuning. As there are only two parameters, the learned weights will likely generalize even from small reused validation set. With faage and fsuaall validation outputs saved, this tuning takes only seconds with any standard optimizer. Note that this procedure closely matches temperature scaling for single-model) calibration (21.
This temperature tuning automatically guards against defective Duos. If femall too inaccurate, then
the Duo can effectively revert to ftarge through the weighting Targe = 1, Tsmall = O. Conversely, any non-trin aoiviaivial weighting (i.e. Tsmall >> 0) implies that the Asymmetric Duo provides benefit over flarge since it improves negative log likelihood on the validation set.
Ablations. To understand how our proposed asymmetric Duo affects performance on downstream tasks, our experiments and analyses consider the following ablations:
1. Unweighted Duos equally weight the predictions of both models without regard for their asymmetry, similar to deep ensembles. It is equivalent to setting Thage Temaall 0.5: This ablation assesses the importance a weighted average for aggregating logits.
2. UQ Only Duos only use Duo predictions for the uncertainty estimate, using the base model's class prediction:
Duo = Viagge = arg max (Jarge X));,
une() = (o (X) Ximai)))))
This ablation isolates the effects of Asymmetric Duos on correctness prediction and se- lective classification tasks, demonstrating whether any benefits to performance can be at- tributed to better uncertainty quantification versus more accurate class predictions.
3.2 Measuring the Computational Cost of Asymmetric Duos
The cost of a Duo versus single model is dictated by the degree of asymmetry in size. How to mea- sure size non-trivial, because no one property fully captures a deep network's 's size or complexity. Parameter count, network depth, floating point operations per forward pass (FLOPs), and through-
put each reflect different aspects of model capacity and computational cost 25 52 (6). To focus on
practical use measure computation. We profile FL.OPs due to their consistency across hardware Section 4 and we profile throughput as measure of real- world time efficiency Appendix Asymmetric duos deliver efficient improvement for either measure of computational cost.

---

ImageNot Acc
ImageNet et V2 Acc
IWildCam P1
:WildCam OOD F1
Caltech 256 Acc
80
30
40
60
25
70
20
60
50
30
15
70
10
10
0.1
FLOPs (10")
FLOPs (10")
(10P)
FLOPs (10P)
FLOPs (10*)
ShuffleNat
EfficiontNet
Not
Conv NeXt
VIT
Parameter Count (100)
RegNet
40
80
120
160
Figure 2: Model Size Correlates with Accuracy. Performance improves with higher FLOPs and parameter counts across many models multiple benchmarks. Our Duos allow adjust size: combining a bigger flarge with littler fsmall adds just a little computation, or more, as we choose.
We measure asymmetry as the relative increase in FLOPs over using Siage isolation:
FLOPs Balance % Computation Increase
(Sarge))
4 Experiment Setup
We evaluate Asymmetric Duos in the context of image classification, measuring their effects on ac- curacy, uncertainty quantification, and downstream decision-making maaknng. To demonstrate their "out-of- the-box" nature, we focus on Duos that combine models from popular pre-trained models available from the torchvision
and timm
59 "model zoos."
Datasets. We benchmark ImageNet S for large-scale evaluation, and Caltech 256 20 and iWildCam (3] for measuring transfer to new domains. accuracy and uncertainty quantifica- tion tend to degrade under distribution shifts 42), we further evaluate with ImageNet V2 48 and iWildCam-OOe 27 to test robustness under such shifts.
Models. For fiarge architecture and parameters, consider pre-trained models that 1. consis- tently achieve high performance on fine-tuning benchmarks [e.g. [8] and 2. are commonly used in practice for various computer vision tasks. We therefore experiment with four strong flagge that cover convolution and attention architectures: a ConvNeXt-Base (35) pre-trained on ImageNet, a Swin -ase 34] pre-trained on ImageNet, an EflicientNet-
[53] pre- ure-trained the larger- scale ImageNet21K, and a CLIP VIT-B 10 45) pre-trained on the larger still LAION-2B (50). These span standard scale (ImageNet) to extremely large scale (LAION). For fsmall models, consider smaller architectures from torchvision and timm 24 B3 46 52 [4]] 61 62) (see Ap- pendix A for complete list). Within a given model family (e.g. ResNet), choose models of varying size (e.g. ResNer-34, ResNet-50, ResNet-101, etc.) to produce Duos with various levels of asymmetry.
Fine- Fine-Tuning and Evaluation. For our downstream datasets evaluations, we fine-tune our models following the LP-FT recipe (28) with standard unweighted Cross-Entroppy Loss, AdamW (36] opti- mizer, linear warm-up with cosine annealing learning rate scheduler, and perform hyperparameter tuning for all models over learning rate and weight decay on withheld validation set.
We calibrate all fine-tuned models by temperature scaling (21), minimizing the negative log likeli- hood the validation set using L-BFGS, to improve baseline uncertainty guantification.
The Size-Acuuracy Relationship of Large and Small Models. Figure 2 summarizes the class prediction accuracy of all the pre-trained and fine-tuned models that use to construct Asymmetric Duos. We report accuracy for ImageNet, ImageNet V2, and Caltech 256 and Macro Score for (WildCam2000-WI.. WILDS account for the significant class imbalances present [27].
Consistent with prior work [8 38 we find strong correlation between model size (as measured by FLOPs/paramete counts) and in-distribution (InD) and out-of-distributiio (OOD) accuracy. In
5

---

ImageNet Acc
ImageNet V2 Acc
1WildCam
HWlldCam ODD F1 T
Caltach 256 Acc
51
86
74
94
48
36
85
92
45
VIT-B Clip
ConvNoxt-
Emn VV
Swin-V2B
0.25
0:25 0.50 0,75
0.50
0.50 0,75 FLOPs Balance
0,75
PLOPPs Balance
OLSPs Balance
FLOPSs Balance
FLOPs Balance
Duo: Ours
Duo: Unweighted
Base Model
Figure 3: Class Prediction (Accuracy and F1 f) as a function of FLOPs balance. (Balance corresponds to cost of a single flarge model; corresponds to the cost of a 2 ensemble.) Asymmetric Duos almost always increase accuracy over single flarge even for Duos that combine flarge with a famall that 1/10th the size. The learned temperature weighting is crucial achieve this performance; if the predictions of Starge and ssmall averaged with equally weight (Duo: Unweighted) then imbalanced Duos may have significantly worse performance than single models.
the next section, we investigate what degree combining pairs of models into Asymmetric Duos allow us to practically traverse the size/performance tradeoff.
Experiment Results
Here present the performance of Asymmetric Duos, measured by accuracy, uncertainty quan- tification, and selective classification performance, as a function of computation (i.e. Duo FLOPs balance). For each benchmark, form Asymmetric Duos using all possible combinations of flarge and fsmall architectures outlined in Section 4 For visual clarity, this section only contains results for single fiarge each dataset. (We choose different fiarge architectures for each dataset for diversity of results within this section.) We note that the results in this section are representative of all ftarge architectures (see Appendix G for the full set of results).
5.1 Duos as Classifiers - Accuracy Macro F1 Score
To determine if the addition of fsmall can enhance class prediction, we compare the accuracy of Asymmetric Duos to that of a single fiarge model in isolation. We report top-1 accuracy (ImageNet V2 and Caltech 256) and Macro Fl score (iWildCam) in Figure against FLOPs balance.
Overall, we find that Asymmetric Duos can yield significant improvements in accuracy/F1 over flarge- We note that the smallest Duos do not provide an accuracy advantage, and occasionally produce minor performance drops. This result is expected, and arguably the lack of major per- formance degradations is surprising given the sigmificantly lower accuracy of the smallest fsmall models. However, after the FL.OPs balance reaches 10%, which many settings may be negligible additional computation, Duos provide consistent accuracy improvements across all datasets.
We find that the proposed temperature weighting scheme is necessary for these improvements in curacy. In Appendix D we report the learned temperatures of all Duos, finding larger fiarge weights when famall smaller, with more even weights as balance increases. Our ablation further confirms this hypothesis. We observe that Duos that equally average the flarge and femall predictions (Duo: Unweighted) significantly underperform flarge when the balance near zero.
In Appendix G we also report results for probabilistic measures of accuracy; namely Brier Score and negative log likelihood. The results for these metries follow similar trends.
5.2 Predictive Uncertainty Separability Correctness Prediction AUROC
Predictive uncertainty estimates are often judged by their calibration, as measured by Expected Calibration Error (ECE) [e.g.21 42). Because we apply temperature scaling throughout our experi- ments, the fiarge models are largely calibrated to begin with, and so Asymmetric Duos provide little improvement. (See Appendix G for results.) We argue that this is not failure of Duos but rather success of temperature scaling.

---

ImageNot AUROC T
ImageNet V2 AUROC
WiléCam AUROC
IWildCam OOD AUROC
Caltech 256 AUROC
88
86
90
96
87
84
87
92
94
VITBB Clip
82
ConvNoxt-B
Emd 1ote--22
Swin
0.25 0.75 PLOPss Balance
0:25 0.50 0,75
0.50
0.50 0,75
0.25 0,75
PLOPs Balance
PLOPSs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duo: UQ Ouly
Base Model
Figure 4: Correctness prediction as measured by AUROC (t), which captures the separability of correct and incorrect predictions by uncertainty. In almost all cases, Asymmetrie Duos achieve higher AUROC than corresponding fiarge models in isolation, even when using a negligible amount of additional computation. We confirm that this increase cannot be attributed to accuracy improve- ments alone: our ablation (Duo: UQ only) that uses the Duo's uncertainty measure to separate the flarge class prediction produces comparable AUROC increase.
We instead choose to measure the quality of Duo uncertainty estimates based on their correlation with the correctness of class predictions. Intuitively, a good uncertainty estimate should be able to separate correct versus incorrect class predictions. To that end, we report correctness prediction Area Under the Receiver Operating Characteristic curve (AUROC) for Duos and for single fiarge as a function of FLOPs balance. Randomly assigned uncertainty measure would yield an AUROC score of 0.5, while a perfectly separable uncertainty measure would yield 1.0.
In Figure 4 we find that Asymmetric Duos consistently yield higher AUROC than flarge. Surpris- ingly, observe performance improvements even when the FLOPs balance is near This finding suggests that AUROC can be increased without much additional computation. Performance does not correlate with additional computation (i.e. higher FLOPs balance), but all Duos provide benefit.
Our ablation demonstrates that these AUROC improvements cannot solely be attributed to the in- creased accuracy afforded by Duos. Recall that the UQ Only Duo variant uses the Duo's uncertainty estimate while retaining the class prediction of fharge- Any AUROC benefits must therefore be at- tributed improvements in the uncertainty; as accuracy remains unchanged. We observe that UQ Only Duos obtain significantly better AUROC score compared to their base models, showing that the sidekick model is effective at re-ranking uncertainties for the class prediction of fiarge. confirming that Duos provide meaningful improvements in uncertainty separability in addition to accuracy.
5.3 Duos as selective classifiers AURC
To study the effect of Duo uncertainty estimates downstream decision-making tasks, evaluate Asymmetric Duos in the context of seleetive classification. Selective classification allows a model to abstain from prediction when the uncertainty is high, deferring uncertain cases to experts in effective human-in-the-loowe systems [16]. The proportion of the test set with uncertainty below a given threshold is referred domain-speciffe loss, which our case is the 0-1 loss analogous to classification error rate. As the uncertainty threshold decreases, the model becomes more cautious, abstaining from more inputs and reducing coverage while aiming to improve accuracy on this smaller covered subset. We emphasize that high-quality uncertainty estimates are necessary for selective classifiers to work in practice.
coverage, and performance on this subset is measured using the
We evaluate selective classification performance using the Area Under the Risk Coverage (AURC) metric [17]. AURC accumulates eтror rates at all coverage levels to measure model's overall poten- tial as a selective classifier. As with correctness prediction, the AURC metric conflates accuracy and uncertainty quantification,
different predictors achieve different accuracy at full coverage (entire test set accuracy). To that end, we evaluate our UQ Only Duo ablation in addition to the proposed approach to bypass this potential confound.
In Figure 5 we observe that Duos provide substantial boost to AURC at all computation levels, even when the FLOPs balance is fractional. Analogous to our correctness prediction results, UQ Only Duos also achieve improved AURC scores at low FLOPSs balance, demonstrating that these improvements cannot be attributed to accuracy alone. This is a direct result of our findings on
7

---

ImageNet AURC
IniegeNet V2 AURC
WWildCam AURC
IWidCCam COD AURC
Caltech 256 AURC
3.8
WTB Clip
Non
1.0
SwIBV2
5.0
3.8
0.8
3.1
9.0
4.0
0.6
0.25 0.50 0.75 FLOPs Balance
0.50 0.75
0.25 0.50 0.75
0.75
0.25 0.50 0.75
FLOPs Bolance
FLOPs Balance
FLOTs Balance
FLOPs Balance
Dac: Ours
Duo: UQ Only
Model
Figure 5: Selective classification performance measured by AURC (4), which averages the еπоо across classification coverage levels/abstaining rates. Duos significantly improve this metric while adding as little as 10% additional computation. As with our correctness prediction results (Figure 4) our UQ Only ablation confirms that these improvements cannot be solcly attributed to increases accuracy.
ImageNet SAC@98
ImageNet V2 SAC898
iWilCCam SAC@98
iWildCam 0OD SAC@90
Caltech 256 SAC899
60
57
40
90
VIT-3 Clip
0.0
ConvNext-
0.0
EebctenNNe//2-11
BificienNeeV22-1
Swin V2.0
0.25 0.75 PLOPs Balance
0.25
0.50 0.75
0.25
0.50
0.25 0.50 0.75
0.25 0.50 0.75
PLOPs Balance
PLOPs Balance
FLOPs Balance
PLOPs Balance
UO Only
Base
Figure 6: Selective classification performance measured by SAC (f), which measures the clas- sification coverage under a fixed accuracy constraint. Duos achieve significantly higher coverages levels of computation. As with our correctness prediction results (Figure 4 our UQ Only ablation confirms that these improvements cannot be solely attributed to increases in accuracy.
correctness prediction with AUROC. Together, these results also confirm that smaller models can effectively re-rank the uncertainties larger models. contrast, many prior methods for selective classification sacrifice non-selective predictive power [14), resulting in unfavorable trade-off's in practical metrics such as AURC.
5.4 Duos as selective classifiers - Selective Accuracy Constraint
To further evaluate Duos in the selective classification context, we report the Selective Accuracy Constraint (SAC) metric for all Duos and fharge models. SAC is practical evaluation metric for risk-sensitive applications with a guarantee. measures maximum coverage a predictor can achieve while maintaining a fixed target accuracy, making it well-suited for settings where relia- bility is critical. Prior work in selective classification has shown that leveraging softmax response uncertainty allows deep networks achieve high accuracy subset of confidently predicted data by abstaining on uncertain cases [14 [6]. Here we show duos boost coverage at the ambitious accuracy levels of 98% across all of our benchmarked datasets.
In 6i we observe that our proposed Asymmetric Duos achieve higher coverage at minimal FLOPs balances. This finding complements our previous results with the AURC metric. Furthermore, the UQ Only ablation almost always match our proposed method, further implying that the improve- ments in coverage at high accuracy levels stems from re-ranked uncertainties rather than accuracy increases. Altogether, these results suggest that Asymmetric Duos are a practical choice high-risk decision-making applications.
5.5 Soup with a (Side)kick: Combining Model Soups and Duos
We show the Asymmetric Duo framework is agnostic to model type, compatible with other methods, and compounds uncertainty quantification improvements by making a duo with a model soup (60] For this experiment, we fine-tuned sixteen ConvNeXt Base models on Caltech 256 starting from

---

96.0
Regilarr
86
0.85
Acmuracy
94
ConoeN Base
AUROC
14.55
AURC-
Cano Nex--Na-PPuo
SAce99
81
Rogular ColviNott Base
0.60
Saae Segunell--eeee
93.5 -
0.55
80
Reguder ConviN
0.25 0.50 0.75
1.00
0.00
0.25
0.50
1.00
0.00
0.25
0.75
0.25
0.50 0.75 1.00 FLOUs Balance
FLOIS Balance
FLOPs Balance
FLOPs Balance
Dao: Curs
Dao: UQ Only
Duo: Unwelghted
Figure 7: Asymmetric Duos are compatible with Soups. model soup (red line) achieves better performance than a standard single model (grey line) across accuracy, AURC, and SAC@98. Our Duos provide further improvement by incorporating the soup as fiargo
the same torchvision pre-trained model using different training hyperparameters. We then mixed greedy model soup for use as faarg in our Asymmetric Duo. In Figure 7 we observe that Duos are compatible with soups and other methods of making better base and sidekick models.
5.6 Comparisons between Asymmetrie Duos and Deep Ensembles
We show that Asymmetric Duos (AD) can match the performance of classic deep ensembles (DE) with significantly lower compute budget. We construct ensembles of ResNet50s (RN50) trained on the ImageNet dataset following the "pool-then-calibrate" strategy from [47). We compare against ADs that pair RN50 (faagge) with MNASNet0.75 75 (MN.75), ShuffleNet V2 2.0x (SN2.0x), and EfficientNet-B2 (ENB2) models, yielding FLOPss Balances (FB) of 0.05, 0.14, and 0.27, respec- tively.
Table 1: Base-model-connrrolle comparison between Asymmetric Duos and deep ensembles.
Method
Balancet,
Acc
Brier,
NLL
ECET
AUROC1
AURC]
RN50 (Temp Scaled)
0
79.82=#0551
2.90+00.05
84,06=3.38
3.39+1.39
86.62+0.49
5.72-0.38
[AD] RN50+MN.75
5
79.88=0042
2.87+00.5
80.45+2.17
3.25+00.11
87.11+0.29
5.42-0.19
[AD] RN50+SN2.06 Ox
14
80.32.10.22
2.8110.04
78.9311.78
2.62:10.66
87.00-1.227
5.34=0.17
[AD] RNS0+ENB2
27
81.87
2.60
71.47:0844
2.80
87.15:10.26
4.80=0.1
[DE] RNSO (m=2)
100
81.12.10.29
2.72.10.04
77.12.1.90
3.5411.33
87.1910.30
4.99_0.24 0.24
[DE] RNS0 (m=5)
400
81.97+0.13
2.61+0.01
73.16+0.90
3.79++0.76
87.62+0.11
1.48-0.00
Deep Ensembles (e.g., m = RN50s) across most metrics, despite using 4x fewer FLOPs. We attribute this to architectural diversity from heterogeneity [19] and Duos' simple but effective temperature-basse aggregation. If we allow Asymmetric Duos to match the computational budget of m 5 Deep Ensembles, then Duos become even more advantageous. See Appendix B for FLOPs-controlled comparison.
Table shows that Asymmetric Duos quickly approach performance of
6 Discussion
In this work, we proposed the Asymmetric Duos framework, which yield consistent improvements across a range of predictive performance. uncertainty quantification, and decision making metrics. While these improvements may also be achievable through deep ensembles, Asymmetric Duos offer a substantially more efficient alternative, requiring only fraction of the additional computation cost of their larger member model. Moreover, the framework inherently compatible with fine-tuning workflows, making it practical and scalable strategy in modern deployment scenarios.
used similar construction of heterogeneous m = 2 deep ensembles to study the effect of predictive diversity induced by models pre-trained on different datasets. Our work differs mainly by aggregating deep network pairs of vastly different sizes as opposed to different architectures with similar accuracy, diving deeper into a smaller member model's effects on Duo's uncertainty, and of course our fine-tuning the pairs on the same data.
Methods related Duos in construction. Gontijo-Lopes

---

Rahaman and Thiery [47] studied the combination of temperature scaling [21] and deep ensembles 29 and found that deep ensembles constructed by gool-then-calibre or gies outperform naive deep ensembles and those following a calibrate-then-poo strategy proper scoring rules. Our proposed Asymmetric Duos follow a similar construction order to a pool-with- calibration strategy, but fundamentally differ in motivation. As shown with our Unweighted Duos ablation, the asymmetry between flarge and fsmall makes a pool-with-cald with-callibbate strategy necessary in preventing degradation in accuracy. Whereas Rahaman and Thiery 477 focuses on such construc- tions' effects on calibration, we put unique emphasis on uncertainty quantification metrics like correctness prediction and selective classification.
strate-
Limitations and future directions. Our experiments are limited to the image classification task. While our findings show strong performance and computational efficiency on total of five test sets, it remains unexplored how Asymmetrie Duos perform in other tasks and modalities, such as image segmentation, regression, and natural language processing. Exploring these extensions could reveal broader applicability of the framework.
This work serves as an initial step toward understanding the aggregation strategies of deep networks of different sizes and capacities. While we focused on small set of interpretable aggregation strategies for both point prediction and uncertainty quantification, there remains a large space of unexplored methods. Notably, our finding that simple validation- ensembles from models with vastly different accuracies is itself a surprising and underexplored result. Future work may benefit from exploring other learning-baseed adaptive weighting schemes for member aggregation, and may even call them Dynamic Duos.
weighting can yield effective
Acknowledgements
GP and ES supported by Canada CIFAR AI Chairs. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC: RGPIN-2002-066405). Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
We thank Mathias Lécuyer for his helpful review and feedback on the experiments and exposition.
References
[1] Javier Antoran, James Allingham, and José Miguel Hermández-Lobato. Depth uncertainty in neural networks. In Advances in Neural Information Processing Systems, volume 33, pages 10620-10634, 2020.
[2] Luis Basora, Arthur Viens, Manuel Arias Chao, and Xavier Olive. A benchmark uncertainty quantification for deep learning prognostics. Reliability Engineering & System Safery, 253: 110513, 2025.
[3] Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset, 2020.
[4] Jesús Carrete, Hadrián Montes-Campos, Ralf Wanzenböck, Esther Heid, and Georg KH Mad- sen. Deep ensembles vs committees for uncertainty estimation in neural-network force fields: Comparison and application to active learning. The Journal of Chemical Physics, 158(20), 2023.
[5] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical au- tomated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition workshops, pages 702-703, 2020.
[6] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Mathiis Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances in neural information processing systems, 34:20089-20103, 2021.
[7] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Sealing vision transformers to billion parameters. International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
10

---

[8] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[9] James M Dolezal, Andrew Srisuwananukom, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Brittany Cody, Aaron Mansfield, Sagar Rakshit, Radhika Bansal, Melanie C Bois, al. Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology. Nature communications, 13(1):6572, 2022.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[11] Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 1 factors. In International Conference on Machine Learming, pages 2782-2792, 2020.
[12] Alex Fang, Simon Kornblith, and Ludwig Schmidt. Does progress on imagenet transfer real- world datasets? In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 25050-25080, 2023.
[13] Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050- 1059. PMLR, 2016.
[14] Ido Galil, Mohammed Dabbah, and Ran EI- Yaniv. What can we learn from the selective pre- diction and uncertainty estimation performance of 523 imagenet classifier?? In The Eleventh International Conference on Learning Representations, 2023.
[15] Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1): 1513-1589, 2023.
[16] Yonatan Geifman and Ran El- Yaniv. Selective classification for deep neural networks. In Advances in neural information processing systems, volume 30, 2017.
[17] Yonatan Geifman, Guy Uziel, and Ran El- Yaniv. Bias-reduced uncertainty estimation for deep neural classifiers. In International Conference on Learning Representations, 2019.
[18) Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks. Ad- vances in Neural Information Processing Systems, 36:29343-29371, 2023.
[19] Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations, 2022.
[20] Gregory Griffin, Alex Holub, Pietro Perona, al. Caltech-256 object category dataset. Tech- nical report, Technical Report 7694, California Institute of Technology Pasadena, 2007.
[21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. ICML., 2017.
[22] Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon. Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 318-319, 2020.
[23] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingho Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2021.
11

---

[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- lion, pages 770-778, 2016.
[25] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: (744:0:770044661 2017.
[26] Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q Weinberger. Snapshot ensembles: Train I, get m for free. International Conference on Learning Repre- sentations, 2017.
[27] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference ma- chine learning, pages 5637-5664. PMLR, 2021.
[28] Ananya Kumar. Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning distort pretrained features and underperform distiiiiiiiittiinn.. In International Confer- ence on Learning Representations, 2022.
[29] Balaji Lakshminarayanaan, Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. In Advances in neural information pro- cessing systems, volume 30, 2017.
[30] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 15t::11111000 11.6614, 2015.
[31] Xiao Li, H Eric Tseng, Anouck Girard, and Ilya Kolmanovsky. Autonomous driving with perception uncertainties: Deep-ensemble based adaptive cruise control. arXiv preprint 1557:7 2024.
[32] Yinghao Li, Lingkai Kong, Yuanqi Du, Yue Yu, Yuchen Zhuang, Wenhao Mu, and Chao Zhang. MUBen: Benchmarking the uncertainty of molecular representation models. In NeurlPS 2023 Al for Science Workshop, 2023.
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceed- ings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[34] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12009-12019 2022.
[35] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976- 1986. 2022.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[37] Vision maintainers and contributors. Torchvision: Pytorch's computer vision library. wwtp://ooooooocciiiiiioooooomo 2016.
[38] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. Interna- tional conference on machine learning, pages 7721-7735. PMLR, 2021.
[39] Bálint Mucsányi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disen- tanglement: Specialized uncertainties for specialized tasks. Advances neural information processing systems, 37:50972-510388 2024.
12

---

[40] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. 2020.
[41] Neyshabur, Sedghi, and C Zhang. What being transferred in transfer learning? In Advances in Neural Information Processing Systems, 2020.
[42] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's s uncertainty? evaluating pre- dictive uncertainty under dataset shift. In Advances in neural information processing systems, volume 32, 2019.
[43] A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.007003, 2019.
[44] Andrea Pugnana, Lorenzo Perini, Jesse Davis, and Salvatore Ruggieri. Deep neural network benchmarks for selective classification. Journal of Data-centric Machine Learning Research, 2024.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[46] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428 10436, 2020.
[47] Rahul Rahaman and Alexandre H Thiery. Uncertainty quantification and deep ensembles. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021.
pages
[48] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages $389-54001 PMLR, 2019.
[49] Ildus Sadrtdinov, Dmitrii Pozdeev, Dmitry P. Vetrov, and Ekaterina Lobacheva. To stay or not to stay in the pre-train basin: Insights on ensembling in transfer learning. Thirty-seventh Conference Neural Information Processing Systems, 2023.
[50]
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[51] Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Bazan Clement Emile Marcel Raoul, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational learning is effective for large deep networks. In Ruslan Salakbutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 44665- 44686. PMLR, 21-27 Jul 2024.
[52] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-611 PMLR, 2019.
[53] Mingxing Tan and Quoe I.e. Efficientnetv2: Smaller models and faster training. In Interna- tional conference on machine learning, pages 10096-10106. PMLR, 2021.
[54] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceed- ings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2820- 2828, 2019.
13

---

[55] Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.0744 2022.
[56] Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7907-7917, 2023.
[57] Kaizheng Wang, Fabio Cuzzolin, Keivan Shariatmadar, David Moens, Hans Hallez, et al. Credal deep ensembles for uncertainty quantification. Advances in Neural Information Pro- cessing Systems, 37:79540-79572, 2024.
[58] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
[59] Ross
Wightman. Pytorch image models.
huttps://github.com www.wwww....o/wwwgtttttt..t
mntooch-imagg-mode 2019.
[60] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre. Rebecca Roclofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without in- creasing inference time. In International conference on machine learning, pages 23965-23999 PMLR, 2022.
[61] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings the vision and pattern recognition, pages 1492-1500, 2017.
the IEEE conference on computer
[62] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 0844-6856, 2018.
14

---

A Experimental Details
A.1 Model Architectures
We select popular, recent, and performative pre-trained models from torchvision [7] and timm
[59 to construct our Duos, to make sure Duos are compatible with various recent architectures.
From torchvision, we experiment with EfficientNet [52]. EfficientNet V2 [53] MNASNet (54),
ConvNeXt [35], ResNet (24), ResNeXt [6]], RegNet [46], ShuffleNet- V2, Swin 622, and Swin (V1
& V2)
33 34 From timm, we gel an Efficientnet-V22 pre-trained on ImageNet 21k g and
ViT-B- 16 10
pre-trained on LAION 2B 500
A.2
Training Procedure
For all our fine-tuned experiments we follow the I.P-FT recipe from Kumar 28 we first train only the final classification head (also known Linear Probing, or LP) for a few epochs (8 for both Caltech 256 and iWildCam) using cross-entropy loss and AdamW, then unfreeze the entire network and Fine-Tune (FT) for more epochs (16 on Caltech 256, 12 on iWildCam) under the same loss and optimizer.
Our FT procedure employs a sequential schedule (linear warm-up followed by cosine annealing), and we perform hyper-pare parameter search over learning rate in [1 x 10-6 66 3.3 x 10-41 and weight decay in [1 x 10 80-8_,1 1 x 10 10-,, picking the best trial by validation score. We use batch size of 128 for Caltech 256 and 16 for iWildCam. All models are trained on NVIDIA L40S GPUs. Random Augmentation [S is used to augment training samples during the FT phase. The best LP checkpoint initializes FT, and the top FT model by validation performance carried forward into our Duo evaluations.
A.3 Datasets
Here provide descriptions of each dataset used and how train-val-test splits chosen in our experiment.
ImageNet S contains 1,000 classes and over million training samples. All the pretrained models used in our experiment have been trained on ImageNet as a last step. Since ImageNet never released the official test set, rescarchers split the official validation set into val and test splits. Since we don't 't need to fine-tune models on ImageNet for our experiment, our only usage of a validation set is to temperature scale single models and to tune temperature weightings for Asymmetric Duos. We use only 5% of the official test split as our validation set to show how data-efficient the temperature- weighting step is.
ImageNet V2 48 is a careful reproduction of the original ImageNet, but nevertheless showed model performance degradation, indicating distribution shifts. Since this dataset is used solely for evaluation, the entire dataset is treated as an OOD test set for ImageNet models and Duos.
iWildCam (OOD) 3 271 is camera-trap dataset with 203,029 training images containing 182 classes of animals. The IND and OOD are determined by distinctive camera traps, since different locations vary largely in vegetation and backgrounds, resulting in poor generalization to new camera traps. We use the official IND val split for iWildCam, which contains 7315 images captured by the same set of camera traps as the training set.
Caltech 256 20 contains 30,607 images spanning 256 object classes and clutter class. We use random 1 15% split of the dataset as val and another 15% as test. The same validation set was used for both hyper-tuning and temperature tuning.
A.4 Greedy Soup Setup
For our Greedy Soup experiments, we independently fine-tune 16 ConvNeXt-Baan -tae models initial- ized from the same backbone with different hyperparameters over the same search space our regu- lar tuning procedure on Caltech 256 starting from the same pre-trained weights from torchvist issiiiin and greedily average their weights to improve performance on the validation set following the pro- cedure outlined by Wortsman al. (60) The greedy soup Figure 7 averages three of the sixteen trials.
15

---

Modele: CELOPs Throughput Corssaatio
Asymmittic Don Throughout Slawdown ve FLO% Balance
80
fexr Archilecture
Theoughut (mmpllecee
10r
Throushout flwddoosh
ettermiiee:
Canohleds
Badban
Raghia
40
103
Arehttacture Pamily
BosNet
UtricteetNet
StrutseNet
20
MNNASt
Swith
RegNett
Vm
102
10
lct
107
Figure 8: Comparison of throughput versus FLOPs as measures of computational cost/model size for single models (left) and CongNeXX-Large--asee Duos (right). both cases we find large correlation between the two measures.
A.5 Deep Ensemble Setup
All m = x DEs in Section 5.6 and Appendix Dla are constructed by randomly selecting 32 subsets of size from 10 independently trained ResNet-50 checkpoints from timm and torchvision, following the sppol-then-calibrate strategy of Rahaman and Thiery [47]. Because our temperature- -easre aggregation strategy for Duos naturally incorporates calibration, this setup enables fair comparison on NLL, Brier, and ECE metrics.
All ADS used for comparison in Section 5.6 are formed by pairing one ResNet-50 checkpoint as flage with a sidekick famall drawn from torchvision.
B Alternative Measure of Computation (Throughput)
As discussed in Section 3.2 deep networks have several reasonable characterizations of size/computational cost, including parameter counts, FLOPs, depth, and throughput. We explore FLOPs in the main text due to its consistency across hardware and the intuitive interpretation of the FLOPs balance the additional computation cost percentage of the larger member model's computation cost. However, FLOOPs fail to consider architectures' parallelization differences, which play factor in model's speed during deployment. Here we show how our FLOPs balance notion would translate into throughput, on single NVIDIA V100 Volta GPU. Figure & shows clear linear correlation between throughput and FLOPs, with small variations among architecture families. For single models, the coefficient of determination for both metrics is 0.938; for Duos it is 0.795. This result implies that the fractional additional FLOPs repeatedly emphasized our main text translate directly to wall-clock-time efficiency.
C Alternative Measures of Uncertainty-Enterr
Throughout the paper els and Duos. (Eq. [ Here we demonstrate that our results are robust to this choice. We replicate main results from Sectionn5whh tive distribution, another common measure used in practice. Figure 9 shows AUROC, AURC, and SAC@9) performance under entropy-based uncertainty quantification. We observe similar perfor- mance trends to our softmax response-based results in Figures and
softmax response to quantify the predictive uncertainty of single mod-
instead measure uncertainty via the entropy of the predic-
D Analysis of Temperature Weightings
we plot the ratio of the temperatures Tharee/ T Tsmall for Asymmetric Duos of various FLOPs balances for all farge models on Caltech 256 to study how temperatures are assigned for fine-tuned models. A ratio > 1 implies that famall does not contribute to the Duo prediction and that the Duo functions like a single model. ratio that closer to implies that fsmall and fiarge both meaningfully contribute to the Duo and that we should expect higher performance than from
In Figure 10
16

---

NmageNet AUROC
ImageNet V2 AUROC
WiléCam AUROC
IWildCam OOD AUROC
Caltech 256 AUROC
86
91
98
90
96
87
84
87
90
83
84
VIT-B Clip
ConvNext-
Emn estNNet-2-1
92
Swin-V28
82
0.25
0:25
0.50 0,75
0.50
0.50 0.75 FLOPs Balance
0,75
PLOPss Balance
PLOPs Balance
FLOPSs Balance
FLOPs Balance
Duo: Ours
Duo: UQ Ouly
Base Model
ImtgeNot AURC
Tes V2 AURC
IWildCam AURC
iWildCam COD AURC
Caltech 256 AURC
3.8
VI- Clip
Swin
7.0
1.0
3.6
5.0
6.0
0.8
3.4
9.0
4.0
0.25 0.50
0.75
0.25 0.75
FLOPs Balance
PLOPs Balance
FLOPs Balance
PLOPs Balance
PLOPs Balance
Duo: Ours
Due: UO Only
Base Model
ImageNet SAC
ImageNet SAC898
IWildCam SACg98
IWildCam SACE98
Caltach SAC@98
20
40
57
90
54
10
54
48
0.0
Clip
OoneNest-3
ErTicientNee332111
Swin-V2.8
51
42
0
0.25
0.50
0.75
0.25 0.50 0.75
0.25 0.50 0.75
0
0.55 0.50 0.75
0.25
0.50 0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs
Dao: Ours
Dao: UQ Only
Base Model
Figure 9: Using entropy to measure uncertainty yields similar results on Asymmetric Duos' correct- ness prediction and selective classification metrics.
Asymme Learned Weighting on caltech256
30
Relative Impro inprovement on Accuracy
Temperature Ratio TangeT Toal
ConvNext-B B
Tempera Ralio Toeeee TraegeTemaa Tomal
Swin-V2-B V2-B
25
6
EfficientNet.V2..
20
5
WTT-B Clip
4
Equal weighting
15
Higher Tra (more weight fange]
-10
2
2-
1
Higher small (more wrightttn famst)
0.0
0.2
0.4
0.6
0.8
0.00
0.25
0.50
0.75
FLOPs Balance
FLOPs Balance
Figure 10: (Left) Ratio of learned temperatures Tiarge/ Tlarge/Temall Tamall for Asymmetric Duos on the Caltech dataset. With small FLOPs balances, flarge is weighted up to 7x more than famall. As the FLOPs balance increases, the weighting becomes more even. (Right) When FLOPs balances is close to 0, Asymmetric Duos significantly up- up-weight flarge to prevent performance degradation; as FLOPs bal- ances increase, Asymmetric Duos learn to assign more weights to ssmall for higher improvements.
what furrg achieves in isolation. As expected, the temperature for faarge is larger than that of famall for small FLOPs balances. As the balance grows, the Duo members weighted more equally.
17

---

E Asymmetric Duos versus Deep Ensembles
Section 5.6 compares Asymmetric Duos and pool-then-calibrat DEs with the same base model, finding that Duos nearly match the performance of ensembles with a fraction of compute. In this section, we compare ADs and DEs under fixed computational budget, which allows the Duo to use a much larger flarge than the DE base model.
Table 2: FLOPs-controlled comparison between Asymmetric Duos and deep ensembles. AD(m,m++) denotes Duos with total compute falling between and m+11 times the FLOPs of a base ResNet-50 (RN50) model. RN50 AD(1.2) includes only Duos with RN50 base models for Dase-controlled comparison against DEs.
Method
FL.OPs(G)
Acct
Brien]
NLJL
ECF.
AUROCT
AURCJ
RN50 (temp-scaled)
79.82=0.51
2.90=0.05
84.06=3.38
3.39=1.39
86.62+0.49
5.72+0.38
RN50 AD(1,2)
[4.1.822
81.5+0.91
2.7=0.12
73.023.87
2.620.95
86.8+0.28
5.020.32
RN50 DE(m=2)
8.2
81.08=0.29
2.72+0.04
77.19=1.66
3.40=1.32
87,18+0,31
5.01#0.24
AD(4.5)
(16.4.20.5)
84,8+1.2
2.2+0.1
59,4=5,4
2.920.7
86,9+0.7
3,9+0.4
RN50 DF(m=5)
20.5
11.95+0.15
2.6140.01
72.77=0.84
3.94=0.73
87.5640.11 13
4,53+0. 13
AD[7.8)
(28.7,32.8)
86.3+0.2
2.0+0.0
53.0+0.9
3.0+0.4
87,340.3
3.4+0.1
RN50 DE(m=8)
32.8
82. 19+0.00
2.57+0.01
71.78=0.41
2.93=0.17
87.64+0.04
4.39+0.02
Table 2 shows that ADs (e.g., AD(4, 5)) considerably outperform DEs with similar test-time FLOPs cost (e.g.. RN50 DE with y 5) most metrics. Notably, while DE performance tends to saturate as m increases, ADs continue to improve thanks the stronger base model at higher FL.PSs budgets. This suggests simple and effective strategy: when deploying Asymmetric Duos, choose the most performant fiarge that your budget allows to maximize both accuracy and uncertainty quantification metrics. The setups on DEs are identical to Table 2 and is described Appendix A
F Evaluating Asymmetrie Duos against scalable Bayesian Methods
Bayesian approaches have been prominent for uncertainty quantification tasks, at times performing competitively against deep ensembles. However, similar to deep ensembles, many Bayesian methods still struggle in scalability, making them hard to compare with methods like Asymmetric Duos. Here evaluate Asymmetric Duos with practical Bayesian methods, Improved Variational Online Newton (IVON) [51] and Laplace Approximation [6].
F. Improved Variational Online Newton (IVON)
The recently proposed IVON [1 was the first variational Bayesian objective optimizer that matched Adam (36) Accuracy when applied to large-scale training, in both crom-scratch and fine-tuning settings. We benchmark our Asymmetric Duos against a ResNet50 fine-tuned with IVON optimizer on Caltech256.
Table Base-model-controolld comparison between Asymmetric Duos and IVON.
Method
Acct
Brier,
NLL.
ECE,
AUROCT
AURCI
Bascline ResNet50
87.72
7.60
67.90
7.56
91.81
1.92
Temp-Scaled ResNet50
87.72
6.88
49.54
1.22
91.44
2.01
IVON (deterministic)
87.98
45.33
1.89
91.92
1.83
IVON (2 samples)
87.23+0.33
7.21#0.09
49.44+0.58
1.59=0.26
91.33+0.27
2.09#0.06
IVON (4 samples)
87.98+0.17
6.9120.05
47.08+0.37
2.60#0.22
91.4020.27
1.8920.05
IVON (8 samples)
88. 18.1940.16
6.77+0.02
45.88+0.22
3. 3.10=0.20
91.65+0.28
1.81=0.02
IVON (16 samples)
88.4120.17
6.7220.01
45.5220.14
3.3620.19
91.4720.29
1.7820.01
IVON (32 samples)
88.43+0.13
6.68+0.02
15.17=0.15
3.46+0.15
91.58+0.10
1.77=0.02
Asymmetric Duos (1,2)
89.75#1.17
5.80=0.60
40.73+4.56
0.97+0.28
91.58+0.53
1.52+0.18
Table shows that IVON works well out the box, outperforming standard AdamW with minimal
training overhead. Its sampling-based inference further improved over the deterministic version,
18

---

though like Deep Ensembles, it introduces an test-time cost with diminishing returns. In contrast, Asymmetric Duos built on ResNet-50 achieve better performance across all metries with only fractional increase in compute. Furthermore, similar to the compatibility shown between soup and Asymmetric Duos in Section 5.5, IVON, as great plug-and-use uncertainty-aware optimizer, would fit nicely with our Asymmetric Duos framework.
F.2 Laplace Approximation
Laplace Approximation(LA) another practical and competitive Bayesian method, approximating the posterior with Gaussian distribution centered at any regularly trained model [6]. We bench- mark our Asymmetric Duos against Laplace Approximation on Swin-V2- V2-S trained on Caltech256, as models with larger final feature dimensions are computationally infeasible for LA given our hard- ware limits. For the Laplace Approximation method, we follow the original authors and use a kron hessian structure to approximate the posterior of the last-layer of the model. We limit FLOPs Bal- ance (FB) to be 0.1 FB 0.3 for Duos to emphasize its computational efficiency.
Table 4: Base-model-controllcs comparison between Asymmetric Duos and Laplace Approxi- mation (LA).
Method
Acc1
F11
Brier].
NLL
ECE|
AUROC1
AURC|
SAC@981
Swin-V2-S baseline
91.7
90.5
5.3
49.1
5.5
93.8
1.0
85,3
Swin-V2-S temp-scaled
91.7
90.5
4.7
33.3
1.5
93.6
1.0
85.6
Swin-V2-S LA
92.2
91.4
4.9
41.6
4.9
93.4
0.9
86.5
Asymmetric Duos
93.020.3
92.2=0.3
4.0:0.22
26.9=1.3
1.1:0.2
93.4=0.2
0.8+0.0
87.920.7
Table shows that although Laplace Approximation slightly improve upon baseline on metrics such as AUROC and AURC, falls short in absolute performance gains compared to Asymmetric Duos across most metrics.
19

---

G Complete Experimental Results
Here we present full results for all 4 faarge models across their benchmarked datasets.
CLIP VTT-B Pretrained on ImageNet 21K benchmarked as flarge on all five datasets.
NmageNet
ImageNet
iWildCam
iWidCam
Caltech
86
76
76
85
75-
74
72
95
VIT-B Clip
74
VIFR Clip
WTF8 Clip
VIFa Clip
VTT-B Clip
0.25
0.75
0.25
0.75
0.50
0.50 0.75
0.25 0.75
FLOPS Balance
FLOPss Balance
FLOPs Balanoe
FLOPs Balanoe
FLOPs Balance
Unweighted
Model
Ace scaled by 100 for legibility
Image Not F1
Image Not V2 F1
IWildCam
IWildCam ODD Fl
Caltoch 256 F1
77
go
76
32.
85
42
26
74
VIFI
WTF Clp
39
WVF Clip
24
VIF- Clip
WI-B Cip
0:25
0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duc: Ours
Duc: Unweighted
Base Madel
Pt. scaled 100 for legibiliy.
ImageNet NLL J
ImageNet V2 NLL
IWilCaam NLL
IWilCCam OOD NLL
Callech 256 NLL
6.5
WF3 Glip
VIEB Glip
14
VIFa Clip
14
VIF Clip
VIEB Clp
11
6.0
2.0
5.5
10
10
10
1.8
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
FLOPs Balance
FLOPs Palance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Dao: Ours
Duo: Unweighted
Base Madel
scaled legibiii....
ImageNet
ImageNet
[WildCam ECE
WildCCam
Caltech 256
WF3 Chp
VLFB Clip
VI-3 Clip
VIF
VIFB Cup
4.0
15-
20
3.0
6.D
10
5.0
1.0
1.0
0.25
0.50 0.75
0.25 0.50 0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25 0.50 0.75
FLOPs
FLOPs
Balance
Dao: Ours
Duo: Unweighted
Base Model
by for legibilty
ImageNet Brier
ImageNet Beier
WWlldCam Clip
WildCam
21
38
33
220
225
30
36
22
200
27
34
180
175
24
0.25 0.50 0.75
0.25
0.75
0.50 FLOPs Balance
0.50
0.75
0.25
0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duo: Unweighted
Model
Beier scaled by 100000 for bepibilty..
20

---

ImageNet AUROC
ImageNet V2 AUROC
:WilCCam AUROC
IWildCam OOD AUROC
Caltech 256 AUROC
88
86
90
87
85
94
VIT-B Clip
VIB Cup
86
WTB Clip
VIT-3 QUi
VIT-B Clip
0.25
0:25
0.50
0.50 0.75
0,75
PLOPPs Balance
PLOPs Balance
PLOPSs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duo: UQ Ouly
Base Model
AURDC scated 100 for leghbiity
ImtgeNot AURC
Tet V2 AURC
IWildCam AURC
iWildCam COD AURC
Caltech 256 AURC
VI- Clip
VIBB Clip 8.0
ViFa Clip
VIEa Clip
VIEB Clip
3.6
0.5
8.0
344
0.4
7.5
0.25 0.50
0.25
FLOPs Balance
PLOPs Balance
FLOPs Balance
PLOPs Balance
PLOPs Balance
Duo: Ours
Due: UO Only
Base Model
AUIIC woaled legibilt..
ImageNet SAC
ImageNet SAC898
IWildCam SACg98
IWildCam SACB98
Caltach SAC@98
60
20
40
57
10
20
52
54
0.0
Clip
Clip
VTT-8 Clipp
Clip
0.25 0.50
0.75
0.25 0.50 0.75
0.25
0.50
0.75
0.25
0.50 0.75
0.25 0.50 0.75
FLOPss Balance
FLOPs Balance
FLOPs
Dao: Ours
Dao: UQ Only
Base Model
Jeptbility
Figure 11: Accuracy, F1, NLL, ECE, Brier, AUROC, AURC, SAC @98 for all flarge
ViT-B Duos.
21

---

Swiin-2-BB Pretrained on ImageNet1K is benchmarked as flarge on all five datasets.
ImageNet Acc
ImageNet Acc
iWilCCam Ace
IWLdCam OOD Acc
Callech 256 Acc
78
76
76
94
84
74
76
92
72
SwinAr2-B
SwinA/2-B
Swin-V2.B
Swin-V2-8
Swin-V2BB
0,75
Fl OPs Balance
Ft OPs Balance
OL. Balanoe
FLOPs Balanoe
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
Act scaled 100 for legibilng
ImageNet P1
ImageNet V2 F1
IWildCam F1
iWildCam 0DD Fl 1
Caltech 256 F1
86
76-
48
36
94
33
74
45
92
30
Swink/2-B
Swin--2-B
42
Swin-V2-B
90
82
27
0.75 FLOPs Balance
0.25
0.50
0:25 0.50 0.75
0.75 FLOPs Balance
0.25 0.50
0.25
0.50
0.75
0.25 0.50 0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
scaled
ImageNot
ImageNet
IWildCam
IWildCam
Caltach 256
Swin-2-B
13
SwimAV2-
Swin-V2-B
14
Swin
Vwin
7.0
12
12
3.0
6.0
2.0
0.25
0.50
0.75
0.25
0.50 0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
FLOPs
FLOPs
FLOPs
Duo: Ours
Duo: Unweighted
Base Model
scaled by 10 Eoelegibilyy
ImageNat
ImageNet
Caltoch
Swin&V--1
4.0
10-
10
15
3.0
7.5
10-
2.0
2.0
5.0
1.0
0.0
0.25
0.50 0.75
0.50
0.50 0.75
0.25
0.50
0.25
0.75
FLOPPs Balanos
FLOP Balance
FLOPPs
PL.PPs Balance
FLOPPs Ralance
Unweighted
BCB scated by 100 for legibiliyy
ImageNet Brier 1
ImageNet V2 Brier
WWidCam Brier
WildCaam OOD Brier
Caltoch 256 Brier
Swin-/2-R
42
Swin4/2.8
210
Swin-V2-88220
Swin-V2.8
Swin-V2-R
24
195
200
40
180
36
180
30
20
0.25 0.75 FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
Beter scaled by 100000 for logibiiiy..
22

---

ImageNet AUROC
ImageNet V2 AUROC
:WileCam AUROC
IWildCam OOD AUROC
Caltech 256 AUROC
92
86
91
86
94
86
84
92
Swin-/2BB
Swin
Swin-V2BB
Swinyy2-
Swin-22
0:25 0.50
0.50 0.75
0,75
PL.PPs Balance
PLOPs Balance
PLOPSs Balance
FLOPs Balance
FLOPs Balance
Doo: Ours
Duo: UQ Ouly
Base Model
AURDC scated 100 for leghbiity
ImageNot AURC
TeaageNet V2 AURC
(WildCam AURC
iWildCam COD AURC
Caltech 256 AURC
Swin&/2-B
Swin-V2-8
Swin
SwinsV2-8
4.5
6.5
4.0
9.0
5.5
0.6
3.5
8.0
0.25 0.50
0.25
PLOPs Balance
PLOPs Balance
FLOPs Balance
PLOPs Balance
PLOPs Balance
Duo: Ours
Due: UO Only
Base Model
AUIIC woaled legibiiy.
ImageNet SACEBR
ImageNet SAC898
|WildCam SACg98
IWildCam SACB98
Caltach SAC@98
20
48
10
40
87
0.0
Swin-V2-R
SwinsV2-5
SwisW2.8
Swin-V2-8
0.25 0.50
0.75
0.25 0.50 0.75
0.25 0.50 0.75
0.25 0.50 0.75
0.25
0.50
0.75
FLOPs Balance
FLOPs Balance
FLOPs
Dao: Ours
Dao: UQ Only
Base Model
legibility
Figure 12: Accuracy, FI, NLL, ECE, Brier, AUROC, AURC, SAC@98 for all flarge Swin-V2--B Duos.
23

---

L Prefrained on LAION-2B is benchmarked as flarge on all five datasets.
ImageNet Acc
ImageNet Acc
iWilCCam Ace
iWIdCam OOD Ace
Callech 256 Acc
87
78
80
86
78
85
84
74
E
76
76
Emd NoN---122
95
0:25 0,75
P1 OPs Balance
Ft OPs Balance
OL. Balanos
FLOPs Balanoe
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
Act scaled 100 for leghbiing
ImageNet Net P1
ImageNet V2 F1
WildCCmm F1
iWildCam ODD F1 1
Caltech 256 F1
87
78
51
39
96
86-
76
48
85
95
84
45
33
Officiest
0.25
0.50 0.75
0:25 0.50 0.75
0.25 0.50 0.75
0 0.25 0.50 0.75
0.25 0.50 0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
scaled Jepibility
ImageNot
ImageNet
[WildCam Ne
IWildCam LibxoesiNetV2-
Caltach 256 Vefeien
14
11
12
2.0
10-
10
10
9.0
8.0
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50 0.75
0.25 0.50 0.75
FLOPs
FLOPs
FLOPs
Balance
Duo: Ours
Duo: Unweighted
Base Model
scaled by 10 Bocleiiiilyy
ImageNot
ImageNet
IWildCam Neiieee
Caltoch
B.D
Net-vent
30
EfficiectNe
6.0
20
2.0
2.0
4.0
5.0
10
1.0
0.25 0.50 0.75
0.50
0.25 0.50 0.75
0.25
0.50
0.75
0.25
0.75
FLOPPs Balanos
FLOP Balance
FLOPPs Balanos
OLOPss Balanca
FLOPPs Ralance
Unweighted
BCB scaled by 100 for legibiliyy
ImageNet Brier 1
ImageNet V2 Brier
WWidCCmm Brier
WildCaam COD Brier
Caltoch 256 Brier
32
200
36
210
28
180
34
180
24
20
160
32
150
0.25
0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
Beter scaled by 100000 for logibiiiy..
24

---

NmtgeNet AUROC
ImageNet V2 AUROC
WileCam AUROC
IWildCam OOD AUROC
Caltech 256 AUROC
88
86
86
94
EffcienNNaVV21
Ednd EimeeesNNet--2-1
0:25 0.50
0.50
0.50 0,75
0.75
PLOPSs Balance
PLOPs Balance
FLOPSs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Doo: UQ Ouly
Base Model
AURDC scaled 100 for legibiltt
NmtgeNot AURC
TraggNNet V2 AURC
(WildCam AURC
iWildCam COD AURC
Caltech 256 AURC
N
Not
5.5
4.0
8.5
7.0
6.0
3.5
0.3
7.5
5.0
0.3
0.50
0.25
FLOPs Balance
PLOPs Balance
FLOPs Balance
PLOPs Balance
PLOPs Balance
Duo: Ours
Due: UO Only
Base Model
AUIIC woaled legibilt..
ImageNet SAC
ImageNet SAC898
IWildCam SAC@98
IWildCam SACB98
Caltach SAC@98
60
20
56
84
10
10
80
40
20
56
32
0.0
EmicieetNet--22
0
0.25
0.50
0.75
0.25 0.50 0.75
0.25
0.50
0.75
025 0.50 0.75
0.25 0.50 0.75
FLOPs Balance
FLOPs Balance
FLOPs
Dao: Ours
Dao: UQ Only
Base Model
Jegtbillly
Figure 13: Accuracy, FI, NLL, ECE, Brier, EfficientNet V2-L Duos.
AUROC, AURC, SAC@98 for all flarge
25

---

Cen--ae Pretrained on ImageNet-1K is benchmarked as flarge on all five datasets.
ImageNet Acc
ImageNet Acc 7
iWldCam Acc
iWidCam OOD Ace
Caltech 256 Acc
77
84
74
78
77-
76
93
75
82
ConcNext-B
ConvNoxt-B
ConviNuxtB
ConvNout-B
92
C
0,75
F1 OPs Balance
FI OPs Balance
OLOPPs Balanos
FLOPs Balanoe
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
Act scaled 100 for legibiliy
ImageNet Net P1
ImageNet V2 F1
WWldCaam F1
iWildCam ODD F1
Caltech 256 F1
74
84
93
45
83
72
ConoNect-B
ConvNect-H
Cont
ConvNest-i
0.25
0.50
0.75
0.25 0.50 0.75
0.25 0.50 0.75
0 0.25 0.50 0.75
0.25 0.50 0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duc: Ours
Duc: Unweighted
Base Model
scaled
ImageNot
ImageNet
IWildCam
IWildCam
Caltach 256
ConvNext-B
Conviiext-i
CamyNesg-
GonvNext-
7.5
13
12
7.0
6.5
2.5
6.0
11
0.25
0.50
0.75
0.25
0.50 0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50 0.75
FLOPs
FLOPs
FLOPs
Balance
Duo: Ours
Duo: Unweighted
Base Model
scaled by 10 Eocleiiiilyy
ImageNot
ImageNet
Caltoch ConvNest-Bl
4.0
CapvNest-Bi
10
Noat-Nattti
10
ConvNext-8i
ComeNNxt-
3.0
3.0
8.0
10
6.0
2.0
4.0
0.25
0.50 0.75
0.50
0.25
0.50 0.75
0.25
0.50 0.75
0.75 FLOPPs Ralance
0.25
FL.PPs Balanos
FLOP Balance
FLOPss
Balanca
Unweighted
BCB scaled by 100 for legibiliyy
ImageNNtt Brier 1
ImageNet V2 Brier
WWidCam Brier
WiilCCam OOD Brier
Caltoch 256 Brier
CameWect-R
42
ConsNext-8
CanvNect-8
ConvNext-8)
ConvNest-R
26
40
200
180
24
180
165
0.25 0.75
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duc: Unweighted
Base Model
Beter scaled by 100000 for logibiiiy..
26

---

NmageNet AUROC
ImageNet V2 AUROC
WWilCCam AUROC
IWildCam OOD AUROC
Caltech 256 AUROC
86
86
91
84
85
ConvNext-B
82
ConvNoxt-
89
ConvNext-B
ConvNiett B
ConvNect-8
0:25 0.50
0.50
0.50 0,75
0,75
PL.PPs Balance
PLOPs Balance
FLOPSs Balance
FLOPs Balance
FLOPs Balance
Duo: Ours
Duo: UQ Ouly
Base Model
AURDC scaled 100 for legibiity
ImageNot AURC
Tet V2 AURC
(WildCam AURC
iWildCam COD AURC
Caltech 256 AURC
4.8
ConvNest-B
8
GonvNext-8
5.5
Next-
BonyNeat-B
0.7
4.5
6.0
4.2
5.0
0.6
4.0
5.0
0.25 0.50
0.25
0.25
FLOPs Balance
PLOPs Balance
FLOPs Balance
PLOPs Balance
PLOPs Balance
Duo: Ours
Due: UO Only
Base Model
AUIIC woaled legibllty.
ImageNet SAC
ImageNet SAC898
|WildCam SACg98
IWildCam SACB9R
Caltach SAC@98
45
20
10
54
15
ComoNoxt-R
0.0
QoneNest-3
Conv
GutvNeatt8
ConvNeet-8
0
0.25
0.50
0.75
0.25 0.50 0.75
0.25 0.50 0.75
0
0.55 0.50 0.75
0.25 0.50 0.75
FLOPs Balance
FLOPs Balance
FLOPs
Dao: Ours
Dao: UQ Only
Base Model
legtbillly
Figure 14: Accuracy, FI, NLL, ECE, Brier, ConvNeXt-Baan Duos.
AUROC, AURC, SAC@98 for all flarge
27

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our main contribution is the development of computationally efficient
Asymmetric Duos method that improves upon single models across wide selections of metrics using only fractional additional computation. This claim was made clear in the abstract, and supported by Figure and Section 5
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of our work is discussed in the Section 6 and specifically the paragraph on Limitations and Future Directions.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work is an empirical study that doesn't include novel theoretical results.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: Yes]
Justification: Details found in Section Jand Appendix A are sufficient for reproduction of our main experimental results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All pre-trained backbones used in our experiments are publically available, and are described Appendix A Full experimental code would be made available.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: We provide detailed descriptions training procedure Section 4 and Ap- pendix A
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our scatterplot results disspayy all Duos, so no error bar needed. Our high-level result in Figure displays clear error bar to show significance.
8. Experiments compute resources
28

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Compute resources are made clear whenever necessary (e.g. for the tempera- ture weights tuning step described in Section and the throughput experiment described in Appendix B).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification: No ethical concerns.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method aims improve computational efficiency and effectiveness in deep networks' predictive power and uncertainty quantification quality. We do not perceive any societal impacts beyond those of standard deep networks.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method aims improve reliability of model uncertainty and has more risk of being misused than existing methods like deep ensembles.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All pre-trained models used in our paper are open-source and publicly avail- able.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Not applicable.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
29

---

Answer: [NA]
Justification: Not applicable.
16. Declaration LLM usage
Does the paper describe the usage of LLMs if is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in any important, original, or non-standard capacity.
30


Reviews:
[{"review_id": "PgUGH1UJIK", "reviewer": "Reviewer_72FX", "summary": "The paper presents \"Asymmetric Duo,\" a novel efficiency-focused approach to uncertainty estimation that challenges conventional ensembling methods. Rather than the computationally prohibitive practice of training multiple large models, the researchers demonstrate how pairing a heavyweight primary model with a lightweight companion model yields remarkable benefits. By implementing a learned weighting mechanism to blend predictions from these mismatched architectures (such as combining a ViT-B with a much smaller ResNet-34), the system achieves enhanced performance across multiple dimensions. Testing across five image classification datasets revealed that this asymmetric pairing not only preserves but actually improves the primary model's accuracy and uncertainty calibration while requiring only fractional additional computational resources. Particularly noteworthy is that this performance boost occurs despite the significant capability gap between the models, working effectively even when applied to sophisticated techniques like model soups. This represents a practical breakthrough for deploying uncertainty-aware deep learning in resource-constrained environments.", "questions": "- Are the large and small models trained on the same set of data? If so, what would happen if the training data is not available to the smaller model? \n- The paper only considered a duo. Does adding more small models further improve the performance?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n- The paper tackles an important problem. \n- The idea of combining a large model is simple yet smart. \n- The results that combining a large model prediction with a small one can lead to enhanced classification accuracy is pleasantly surprising to me. \n- Extensive experiments were conducted to validate the claims. \n\n## Weaknesses:\n- Though empirically supported, the proposed method is not backed up by any theoretical foundations. It would be better if there are interesting theoretical insights to further support the proposed method. \n- Although most of the Duos lead to performance improvements, the relative improvements can vary given the same FLOPs Balance. It is not immediately clear to me on how we can select the optimal model/configuration to pair up with the large model.", "was_revised": true, "final_justification": "I have read the authors rebuttal and also the other reviewers' feedback. Despite several weaknesses, I think the findings of the paper is interesting. I am inclined towards accepting the paper.", "rebuttal": "# Author Response to Reviewer 72FX\n\nWe thank the reviewer for the thoughtful review and insightful questions.\n\n---\n\n**[W1] Regarding theoretical foundations**  \nWe appreciate this feedback. While we focused on empirical validation given the practical nature of our contribution, we could easily establish several straightforward theoretical results.\n\n*Accuracy improvement guarantee*  \nIf we select temperature weights to maximize validation accuracy, Duos necessarily outperform the large model on validation data (the large model is simply a Duo with weights $[1,0]$). Given the low VC dimension of the space of possible temperature weightings ($\\leq 3$), standard learning theory provides $O\\left(\\sqrt{1/|\\mathrm{val\\_set}|}\\right)$ generalization bounds on the test accuracy improvement.\n\n*Ensemble equivalence*  \nRecent work [Dern2025, Abe2022] show large models behave like ensembles of smaller models. A direct application of Theorem 3.3 from Dern2025 proves that a Duo combining M×D and D-parameter random feature models is equivalent to an ensemble of $(M+1)$ D-parameter models—formalizing our intuition about why asymmetric combinations work.\n\nBoth proofs are approximately 10 lines and follow from standard techniques. We omitted them to avoid cluttering the paper, but we're happy to include whichever result you feel would most benefit readers. Would either theorem meaningfully strengthen the contribution in your view?\n\n---\n\n**[W2] Practical Selection of Duos**  \nWe thank the reviewer for this thoughtful point regarding the practicality of selecting the sidekick model in an Asymmetric Duo, a question that interests us as well. \n\nSimilar to choosing a backbone model for fine-tuning, it is indeed difficult to know beforehand which Duo will perform best on a specific downstream task. That said, some heuristic indicators have been proven useful. In particular, we have found that FLOPs balance and pre-training performance on a standard dataset like ImageNet are informative when choosing the sidekick model. FLOPs balance is free and reliable, and is aligned with the general \"larger models perform better\" intuition in modern ML. Lastly, as the review suggested, *most* Duos lead to performance improvements, so even though it's hard to predict the optimal choice of sidekick, it is easy to find good contenders.\n\n---\n\n**[Q1] Training Dataset**  \nYes, both the base model and the sidekick model in our experiments have been trained on the same downstream training set, though this is not mandatory. We can still make Duos even if a sidekick model has only seen partial or none of the entire training set, and this would be an interesting investigation for future work.\n\n---\n\n**[Q2] Adding more than one sidekick model**  \nWe thank the reviewer for pointing out this natural follow-up of our Duo framework.\n\nCombining a large base model with more than one sidekick can be considered as another approach to scale up the Asymmetric framework, and its efficiency/performance tradeoff is one that we find interesting as well. We conducted additional experiments using a Trio setup, aggregating a base model with two sidekicks using the same temperature weighting strategy described in Section 3.1, while keeping the total additional FLOPs within 20%. While Trios sometimes show slight improvements over Duos in AUROC and AURC, we also observe some regressions in Accuracy, Brier score, and NLL.  \n\nOverall, we find that Asymmetric Duos offer a more consistent and efficient tradeoff at low FLOPs budgets, making them the more practical choice in most settings.\n\n---\n\n**References**\n\n[Abe2022] Abe, Buchanan, Pleiss, Zemel, Cunningham. *Deep Ensembles Work, But Are They Necessary?* NeurIPS 2022.\n\n[Dern2025] Niclas Dern, John P. Cunningham, and Geoff Pleiss. *Theoretical limitations of ensembles in the age of overparameterization* ICML 2025."}, {"review_id": "PMpD0fgdbK", "reviewer": "Reviewer_x6b8", "summary": "The paper presents a combining strategy for a set of two learning models. The althors argue that the proposed model improve upon single models.", "questions": "- How does the method compare to dynamic or adaptive combination strategies commonly used in ensemble models?\n- How does the proposed method differ from a weighting scheme with weights adjusted via cross validation\n- The experiments do not compared the proposed model to state of the art models for the classsification tasks", "limitations": "The proposal is simple and thus requires an extesive experimental evaluation to support its claims. It that sense, the experimental procedure should be improved by compraing the proposed strategy to state of the art models.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The proposed method is simples but the paper lacks a more robust analysis (theorectical and/or computational) to ensure that the proposed combination scheme presents any advantage when compared to other popular combination strategies", "was_revised": true, "final_justification": "Due to the additional discussions and analyses presented during the revision stage, I have decided to update the scores assigned to the paper. The contribution is now clearer, and an extensive comparison with other relevant methods has been added", "rebuttal": "# Author Response to Reviewer x6b8\n\nWe thank the reviewer for the review.\n\n---\n\n**\\[Q1 & Q2] Aggregation Strategy Confusion**\n\nThere may be some confusion about our core contribution. While our method uses an aggregation approach that is indeed standard validation-based weighting, our core contribution is applying this standard technique to models of vastly different accuracies to achieve the performance improvements of ensembles at fractional computational costs. This finding is surprising given that practical wisdom and prior work [Opitz1999; Wood2023] suggest asymmetric model combinations should not work effectively. We have added this clarification to Section 6 (Limitations and Future Directions) to better highlight our contribution, and we hope this resolves the confusion regarding the aggregation strategy.\n\nThis work serves as an initial step toward understanding the aggregation strategies of deep networks of different sizes and capacities. While we focused on a small set of interpretable aggregation strategies for both point prediction and uncertainty quantification, there remains a large space of unexplored methods. Notably, our finding that simple validation-based weighting can yield effective ensembles from models with vastly different accuracies is itself a surprising and underexplored result.\n\n---\n\n**\\[Q3] Asymmetric Duos compared to SOTA methods**\n\nWe are not sure what SOTA methods the reviewer is referring to.\nWe built Asymmetric Duos directly off of popular and recent models that have been empirically demonstrated to be consistently best-performing through extensive evaluations by [Goldblum2023] on a suite of computer vision tasks.\nWe additionally compare and combine Asymmetric Duos to recent SOTA methods like model soup [Wortsman2022] and show compatibility in Section 5.5.\nWe would be happy to receive more specific feedback on which methods the reviewer is interested in seeing us benchmark against.\n\n---\n\n**References**\n\n[Goldblum2023] Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, et al. *Battle of the backbones: A large-scale comparison of pretrained models across computer vision tasks*. *Advances in Neural Information Processing Systems*, 36:29343–29371, 2023.\n\n[Opitz1999] Opitz, R. M., & Maclin, D. W. (1999). *Popular ensemble methods: An empirical study.* Journal of Artificial Intelligence Research, 11, 169–198.\n\n[Wortsman2022] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, *et al.* *Model soups: Averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.* In *International Conference on Machine Learning*, pages 23965–23998. PMLR, 2022.\n\n[Wood2023] Wood, D., Mu, T., Webb, A., Reeve, H. W. J., Luján, M., & Brown, G. (2023). *A Unified Theory of Diversity in Ensemble Learning.* Journal of Machine Learning Research, 24, 1–49."}, {"review_id": "vHoIz6SvLt", "reviewer": "Reviewer_ZFrX", "summary": "The paper introduces Asymmetric Duos, a practical and cost-effective method for uncertainty quantification (UQ) that leverages an ensemble of two models: a large model and a considerably smaller \"sidekick.\" The final prediction is obtained via a simple learned temperature-weighted aggregation of both outputs in logit space. The approach is evaluated in the image classification setting and is shown to improve predictive accuracy and UQ with negligible overhead, making it particularly attractive in large-scale or fine-tuning regimes.", "questions": "- **[Q1]** How do the learned temperatures behave in practice? Do they consistently emphasize a single model or does the dominance of one model balance out with FLOPS? Are there \"mode-collapses\"? Could the authors quantify how often the sidekick model meaningfully changes the prediction, e.g., via majority class swaps?\n\n\n- **[Q2]** How does the performance (accuracy/UQ) of Asymmetric Duos compare against a standard Deep Ensemble with 5–10 models on smaller datasets (and other scalable UQ methods)? See [W1].\n\n\n- **[Q3]**:  Do the authors see potential connections between the proposed Duos approach and Bayesian Deep Learning—particularly sampling-based inference—which also constructs ensembles of model instances to approximate the Bayesian model average (BMA)? Would you then expect a BMA composed of posterior samples from asymmetric architectures to perform well?\n\n\n$\\rightarrow$ If the authors can satisfactorily address the raised weaknesses and questions, in particular  W1/Q2, I would be inclined to raise my score to an accept.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- **[S1]** The approach is simple and practical, offering improvements in predictive accuracy and UQ with very low computational overhead.\n- **[S2]** Applicable in standard fine-tuning scenarios and scales well to large architectures.\n- **[S3]** The empirical evaluation is thorough within the considered setup, and the paper is clearly written and easy to follow.\n\n## Weaknesses\n\n**Major:**\n\n**[W1]** The study focuses exclusively on very large architectures. However, it is known from theory and practice that deep ensembles of moderate size (e.g., 5–10 models) can offer significant improvements and can themselves even be improved significantly (cf. [1,2,3]). It would be insightful to compare Asymmetric Duos against larger deep ensemble sizes - potentially even other UQ methods see W2 - on smaller-scale problems where these are computationally feasible. This would allow a rough assessment of how large the *cost-quality trade-off* is between DEs with usual ensemble sizes and Asymmetric Duos (and potentially other competing UQ methods). \n\n**Minor:**\n- **[W2]** No comparison is made with scalable approximate Bayesian methods like IVON [4] or Laplace Redux [5]. While not critical, such a comparison could better contextualize the proposed method in the UQ landscape.\n- **[W3]** The aggregation scheme is simplistic—fixed temperature scaling. The paper lacks theoretical development or empirical exploration of even simple data-dependent or more complex adaptive combination strategies. As the simple one seems to work already well this point is minor. A closer look at potentially persistent performance gaps with larger ensemble sizes (see W1) could help in assessing how much room for improvement from more evolved aggregations can be anticipated.\n- **[W4]** The discussion of individual results and metrics is occasionally redundant and could be streamlined for conciseness.\n## References\n\n- [1] Wang, Y., & Wang, X. (2025). Agree to Disagree: Demystifying Homogeneous Deep Ensembles through Distributional Equivalence. In *The Thirteenth International Conference on Learning Representations*.\n- [2] Wild, V. D., Ghalebikesabi, S., Sejdinovic, D., & Knoblauch, J. (2024). A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods. *Advances in Neural Information Processing Systems*, 36.\n- [3] Sommer, E., Robnik, J., Nozadze, G., Seljak, U., & Rügamer, D. (2025). Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks. In *The Thirteenth International Conference on Learning Representations*.\n- [4] Shen, Y., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Raoul, B. C. E. M., Yokota, R., Gurevych, I., Cremers, D., Khan, M. E., & Möllenhoff, T. (2024). Variational Learning is Effective for Large Deep Networks. In *Proceedings of the 41st International Conference on Machine Learning*, *Proceedings of Machine Learning Research*, 235:44665–44686. Available from: https://proceedings.mlr.press/v235/shen24b.html.\n- [5] Daxberger, E., Kristiadi, A., Immer, A., Eschenhagen, R., Bauer, M., & Hennig, P. (2021). Laplace Redux – Effortless Bayesian Deep Learning. In *35th Conference on Neural Information Processing Systems (NeurIPS 2021)*.", "was_revised": true, "final_justification": "I recommend acceptance. \n\nMy initial score was borderline, but my concerns have been, for the most part, addressed through a productive discussion and new experiments provided by the authors.\n\nThe rebuttal successfully resolved the main weaknesses regarding the experimental evaluation [W1, W2] by providing new, crucial comparisons against both standard Deep Ensembles and other scalable UQ methods. This new evidence significantly strengthens the paper's claims and contextualizes its contribution.\n\nMy final recommendation is contingent on the authors integrating all new results and the related discussions addressing the other raised weaknesses/questions into the final manuscript.", "rebuttal": "# Author Response to Reviewer ZFrX\n\n---\n\nWe thank the reviewer for the thoughtful review and the insightful suggestions.\n\n**[W1&Q2] Direct Comparison between Asymmetric Duos (AD) and Deep Ensembles (DE).**  \nWe thank the reviewer for highlighting this important comparison. Here we directly compare ADs and ResNet-50 DEs on ImageNet while controlling for total FLOPS. In short, while Deep Ensembles show diminishing returns as the number of members ($m$) increases, Duos continue to improve in accuracy and uncertainty quantification with higher FLOPS. This makes Duos more effective under high compute budgets. We provide the results and experimental details below:  \n\n| Method      | FLOPS(G)     | Acc↑       | Brier↓    | NLL↓       | ECE↓       | AUROC↑    | AURC↓    |\n|-------------|---------------|------------|-----------|------------|------------|-----------|-----------|\n| Base Model  | 4.1           | 79.8±0.5   | 3.3±0.5   | 102.6±14.1 | 16.8±9.6   | 85.0±1.9  | 6.4±0.8   |\n| AD[1,2)     | [4.1,8.2)    | 81.2±0.8   | 2.7±0.1   | 74.1±4.0   | 2.5±0.8    | 86.9±0.7  | 5.1±0.3   |\n| DE(m=2)     | 8.2           | 81.2±0.3   | 2.9±0.2   | 88.8±9.7   | 12.4±7.5   | 86.4±1.0  | 5.3±0.5   |\n| AD[2,3)     | [8.2,12.3)   | 82.3±0.6   | 2.5±0.1   | 70.2±3.1   | 3.0±0.8    | 87.1±0.7  | 4.7±0.3   |\n| DE(m=3)     | 12.3          | 81.7±0.2   | 2.8±0.2   | 83.5±9.4   | 9.4±7.9    | 86.8±0.8  | 4.9±0.4   |\n| ...         |               |            |           |            |            |           |           |\n| AD[4,5)     | [16.4,20.5)  | 84.8±1.2   | 2.2±0.1   | 59.4±5.4   | 2.9±0.7    | 86.9±0.7  | 3.9±0.4   |\n| DE(m=5)     | 20.5          | 82.0±0.2   | 2.7±0.2   | 79.0±7.7   | 6.9±6.5    | 87.1±0.6  | 4.6±0.3   |\n| ...         |               |            |           |            |            |           |           |\n| AD[7,8)     | [28.7,32.8)  | 86.3±0.2   | 2.0±0.0   | 53.0±0.9   | 3.0±0.4    | 87.3±0.3  | 3.4±0.1   |\n| DE(m=8)     | 32.8          | 82.3±0.1   | 2.6±0.1   | 76.8±3.8   | 5.2±3.7    | 87.3±0.2  | 4.5±0.1   |\n\n\nThis comparison highlights the superior scalability of Asymmetric Duos under FLOPS budgets. \nAD$[m,m+1)$ denotes Asymmetric Duos with total FLOPS within $m$ to $m+1$ times of that of a Base ResNet50 Model. Note that all metrics values were scaled by $10^{2}$ for better presentation, with the exception of Brier score which was scaled by $10^{4}$. DEs of sizes $m=2$ to $8$ are formed by 32 random subsets of 10 ResNet-50 checkpoints from timm; ADs compose a ResNet50/ResNet101/ResNet152/ConvNeXt-Base/ViT-B base model with a sidekick model with FLOPS Balances $\\geq0.1$. \n\nWe also repeated this analysis controlling for parameter count instead of FLOPS and observed similar trends. These results and their visualizations will be included in the Appendix in our revision. We hope these findings will spark broader interest in methods for both homogenous combination (ensembles) and heterogeneous combination (asymmetric Duos) and contribute to a better understanding of when and how to apply each approach effectively.\n\n---\n\n**[W2] Scalable Bayesian Methods.**  \nWe thank the reviewer for making thoughtful and specific suggestions on scalable bayesian methods.  \n\n**Improved Variational Online Newton (IVON)**\n\nWe benchmark Asymmetric Duos against a ResNet50 fine-tuned with IVON optimizer on Caltech256.  \n\n| Method                    | Acc↑       | Brier↓     | NLL↓       | ECE↓       | AUROC↑     | AURC↓     |\n|---------------------------|------------|------------|------------|------------|------------|-----------|\n| Baseline ResNet50         | 87.72      | 7.60       | 67.90      | 7.56       | 91.81      | 1.92      |\n| Temp-Scaled ResNet50      | 87.72      | 6.88       | 49.54      | 1.22       | 91.44      | 2.01      |\n| IVON (deterministic)      | 87.98      | 6.63       | 45.33      | 1.89       | 91.92      | 1.83      |\n| Asymmetric Duos [1,2)     | 89.75±1.17 | 5.80±0.60  | 40.73±4.56 | 0.97±0.28  | 91.58±0.53 | 1.52±0.18 |\n| IVON (2 samples)          | 87.23±0.33 | 7.21±0.09  | 49.44±0.58 | 1.59±0.26  | 91.33±0.27 | 2.09±0.06 |\n| IVON (4 samples)          | 87.98±0.17 | 6.91±0.05  | 47.08±0.37 | 2.60±0.22  | 91.40±0.27 | 1.89±0.05 |\n| IVON (8 samples)          | 88.19±0.16 | 6.77±0.02  | 45.88±0.22 | 3.10±0.20  | 91.65±0.28 | 1.81±0.02 |\n| IVON (16 samples)         | 88.41±0.17 | 6.72±0.01  | 45.52±0.14 | 3.36±0.19  | 91.47±0.29 | 1.78±0.01 |\n| IVON (32 samples)         | 88.43±0.13 | 6.68±0.02  | 45.17±0.15 | 3.46±0.15  | 91.58±0.10 | 1.77±0.02 |\n\n\nIVON works well out of the box, outperforming standard AdamW with minimal training overhead.\nIts sampling-based inference further improved over the deterministic version, though like Deep Ensembles, it introduces an $m\\times$ test-time cost with diminishing returns.\nIn contrast, Asymmetric Duos built on ResNet-50 achieve better performance across all metrics with only a fractional increase in compute.\nFurthermore, similar to the compatibility shown between soup and Asymmetric Duos in Section 5.5, IVON, as a great plug-and-use uncertainty-aware optimizer, would fit nicely with our Asymmetric Duos framework.  \n\n\n**Laplace Approximation**\n\nWe also compare Laplace Approximation (LA) with Asymmetric Duos. Due to LA's higher computational and memory usage when applied to architectures with high last-layer dimension, we perform this analysis on Caltech256 using a Swin-T model.\n\n| Method                       | Acc↑      | F1↑       | Brier↓   | NLL↓      | ECE↓     | AUROC↑   | AURC↓    | SAC@98↑   |\n|:-----------------------------|:---------|:---------|:--------|:---------|:--------|:-----------|:--------|:---------|\n| Swin-T baseline              | 90.2     | 89.2     | 5.7     | 41.3     | 4.0     | 92.8       | 1.3     | 80.3     |\n| Swin-T temp-scaled           | 90.2     | 89.2     | 5.5     | 37.5     | 1.7     | 92.9       | 1.3     | 79.8     |\n| Swin-T Laplace Approximation | 90.7     | 89.6     | 5.7     | 47.4     | 5.4     | 93.3       | 1.2     | 81.3     |\n| Swin-T Asymmetric Duos (<0.3 FLOPS Balance)              | 91.5±0.3 | 90.6±0.3 | 4.9±0.2 | 33.2±1.4 | 1.4±0.2 | 93.0±0.3   | 1.0±0.1 | 83.4±1.1 |\n\nWe observe that, similar to IVON, Laplace Approximation (last-layer-only & kron & probit as recommended) reliably improves over the base model. However, it falls short in absolute performance gains compared to Asymmetric Duos across most metrics. Notably, LA can also be incorporated into any Duo, further demonstrating the generality and practicality of the Asymmetric Duos framework.\n\n---\n\n**[W3] More Asymmetric Duos Aggregation.**  \nWe thank the reviewer for recognizing that the simple fixed temperature technique is effective. We argue this simplicity is a strength: if we select temperature weights to maximize validation accuracy, Duos necessarily outperform the large model on validation data (the large model is simply a Duo with weights [1,0]). Given the low VC dimension of the space of possible temperature weightings (≤3), standard learning theory provides O($\\sqrt{1/|\\mathrm{val_set}|}$) generalization bounds on accuracy improvement with small constants. We will include a derivation of this bound in revision (and we can provide details to the reviewers if needed, though it follows from standard learning theory). Second, the efficacy of such a simple approach is itself a notable empirical contribution. While established work suggests that ensembling models of different performance levels typically yields degradations [Opitz1999; Wood2023], our finding that asymmetric models can be effectively combined with simple temperature weighting should surprise the community. Again, we will include this discussion in the revision.\nWhile data-dependent mechanisms could yield further gains, they would sacrifice these theoretical guarantees and require additional training overhead. Nevertheless, we believe pursuing such \"Dynamic Duos\" is a fruitful direction for future work.\n\n---\n\n**[Q1] Temperature behaviors in practice.**  \nLearned temperatures tend to emphasize the base model in low-FLOPS-balance regimes and gradually increase the weight on the sidekick model as the FLOPs balance improves. Figure 10 in Appendix D illustrates this behavior for Asymmetric Duos, showing the ratio between the learned temperatures of the base and sidekick models as a function of FLOPs balance. This reveals a gradual shift in weighting toward the sidekick as its relative capacity increases.\n\n---\n\n**[Q3] Bayesian Model Averaging and Asymmetric Duos.**  \nThe connection between Duos and Bayesian model averaging is interesting. If one were to combine posterior samples from a large model and a small model, this could be interpreted as sampling from a hierarchical Bayesian model where the prior is a mixture over architecture types, effectively placing prior probability mass on both small and large model architectures. We agree that a Bayesian version/interpretation of Duos is an interesting avenue for future work that could make use of additional train-time or test-time computation beyond our efficient duos.  \n\n---\n\nWe thank the reviewer again for the thoughtful feedback. We hope our responses adequately address the concerns raised and would be happy to discuss further during the reviewer–author discussion period.\n\n---\n\n**References**\n\n[Opitz1999] Opitz, R. M., & Maclin, D. W. (1999). *Popular ensemble methods: An empirical study.* Journal of Artificial Intelligence Research, 11, 169–198.\n\n[Wood2023] Wood, D., Mu, T., Webb, A., Reeve, H. W. J., Luján, M., & Brown, G. (2023). *A Unified Theory of Diversity in Ensemble Learning.* Journal of Machine Learning Research, 24, 1–49."}, {"review_id": "GiQlMmQOWu", "reviewer": "Reviewer_VDfk", "summary": "The paper proposes to build an ensemble of n=2 models, especially when the second model is smaller than the first. The models are combined on the logit level by learning mixture weights (= temperatures) based on validation log likelihood. The authors find that these “Duos” give increased performance, both in terms of accuracy and uncertainty quantification, and that the second model does not have to be big for this to be true. Indeed, if the second model has about 25% of the FLOPs of the first model, the performance already saturates. This indicates that model ensembles can be more efficient than the popular default n=10 deep ensembles.", "questions": "See the request for other ensemble baselines.\n\nSince you mix the two models on the logit level (=unnormalized), it could be that one model generally has logit vectors with bigger norms than another, such that there is a confounding factor besides the mixture temperatures. I do not expect this to happen systematically, but could you check whether this is an issue, just in case?", "limitations": "The authors describe limitations and related works that are close to their results.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The paper is well written and section connect to each other in a logical manner\n* The paper pays attention to detail, such as reporting number of parameters + FLOPS + throughput to give a wholistic picture, adding ablations on metrics and models in the appendices, and making sure comparisons are fair (for example using the UQ-only Duo when comparing AURC)\n* The analysis is very deep in terms of metrics (accuracy, logl, Brier score, ECE, AUROC, AURC, SAC). All of these metrics are grouped nicely and put into perspective correctly, I am not missing anything here.\n* The paper is reproducible. The appendix contains an in-depth description of all hyperparameters and code that reproduces Figure 1-14 is provided.\n\n## Weaknesses\n\n### Major weaknesses (that influence my score)\n\n* The paper only compares duos to single models as a baseline. It would be interesting to compare to deep ensembles of different sizes, if not further modern ensembling approaches. Currently, the paper only supports the hypothesis “Yes, adding a second model improves performance and UQ of the duo, even if the second model is small.”, but comparing to ensembles could help judge if the improvement is already saturating to the performance of the ensemble (showing that deep ensembles are unnecessarily big), or how big the gap is. I would be happy to see experiments on this in the rebuttal. Regardless of whether the outcome is “positive” or not, I think it would strengthen the paper.\n* The novelty of the paper is incremental. In essence, the main message is “Adding a second model helps improve performance slightly, and this second model can be smaller (although bigger = better)”. Since ensembling is already well-known, and also n=2 ensembles were explored in literature, the methodological and insight contribution is limited, and it is also limited in scope to image classification, which is a topic that ages currently.\n\n### Other notes that did not influence my score and need no rebuttal but could help improve the camera-ready:\n\n* Caption of Figure 2, I would replace “littler” with “smaller”\n* “Randomly assigned uncertainty measure would yield an AUROC score of 0.5“ → might want to rewrite to either a measure / estimator that randomly assigns, or randomly assigned estimates. A randomly assigned measure is somewhat unorthodox\n* “We note that the AUROC metric couples accuracy and uncertainty quantification“ I do not believe this is true. The AUROC (Section 5.2) is independent of the marginal of the binary variable, which in your case is the accuracy. (the AURC in section 5.3 does conflate them)\n\n## Justification for score\n\nThe paper is exceptionally well executed both in experiments and in writing. However, its novelty is somewhat incremental. Since I generally support well-executed papers, even if not exactly novel, I recommend to accept the paper. However, I am looking forward to experiments with deep-ensemble and other ensembling baselines, which I believe are somewhat missing with the papers scope of showing that ensembles don’t need to be big. I am willing to increase my score if these experiments can be provided, whether the outcome is “negative” or “positive” for the papers message.", "was_revised": true, "final_justification": "The rebuttal has addressed the main point of critique (of me and other reviewers) by adding the deep ensemble results. As auch, I believe the paper is now well-rounded off and technically solid. Although not award-winning, I think the paper should be accepted, and because I believe rebuttal periods should be used to get out of borderline scores, I update to a score of 5.", "rebuttal": "# Author Response to Reviewer VDfk\n\nWe thank the reviewer for the thoughtful review and the constructive feedback. \n\n---\n\n**[W1] Direct Comparison between Asymmetric Duos (AD) and Deep Ensembles (DE).**  \nWe thank the reviewer for highlighting this important comparison. Here we directly compare ADs and ResNet-50 DEs on ImageNet while controlling for total FLOPS. In short, while Deep Ensembles show diminishing returns as the number of members ($m$) increases, Duos continue to improve in accuracy and uncertainty quantification with higher FLOPS. This makes Duos more effective under high compute budgets.\n\nWe provide the results and experimental details below:  \n\n| Method      | FLOPS(G)     | Acc↑       | Brier↓    | NLL↓       | ECE↓       | AUROC↑    | AURC↓    |\n|-------------|---------------|------------|-----------|------------|------------|-----------|-----------|\n| Base Model  | 4.1           | 79.8±0.5   | 3.3±0.5   | 102.6±14.1 | 16.8±9.6   | 85.0±1.9  | 6.4±0.8   |\n| AD[1,2)     | [4.1,8.2)    | 81.2±0.8   | 2.7±0.1   | 74.1±4.0   | 2.5±0.8    | 86.9±0.7  | 5.1±0.3   |\n| DE(m=2)     | 8.2           | 81.2±0.3   | 2.9±0.2   | 88.8±9.7   | 12.4±7.5   | 86.4±1.0  | 5.3±0.5   |\n| AD[2,3)     | [8.2,12.3)   | 82.3±0.6   | 2.5±0.1   | 70.2±3.1   | 3.0±0.8    | 87.1±0.7  | 4.7±0.3   |\n| DE(m=3)     | 12.3          | 81.7±0.2   | 2.8±0.2   | 83.5±9.4   | 9.4±7.9    | 86.8±0.8  | 4.9±0.4   |\n| ...         |               |            |           |            |            |           |           |\n| AD[4,5)     | [16.4,20.5)  | 84.8±1.2   | 2.2±0.1   | 59.4±5.4   | 2.9±0.7    | 86.9±0.7  | 3.9±0.4   |\n| DE(m=5)     | 20.5          | 82.0±0.2   | 2.7±0.2   | 79.0±7.7   | 6.9±6.5    | 87.1±0.6  | 4.6±0.3   |\n| ...         |               |            |           |            |            |           |           |\n| AD[7,8)     | [28.7,32.8)  | 86.3±0.2   | 2.0±0.0   | 53.0±0.9   | 3.0±0.4    | 87.3±0.3  | 3.4±0.1   |\n| DE(m=8)     | 32.8          | 82.3±0.1   | 2.6±0.1   | 76.8±3.8   | 5.2±3.7    | 87.3±0.2  | 4.5±0.1   |\n\n\nThis comparison highlights the superior scalability of Asymmetric Duos under FLOPS budgets. \nAD$[m,m+1)$ denotes Asymmetric Duos with total FLOPS within $m$ to $m+1$ times of that of a Base ResNet50 Model. Note that all metrics values were scaled by $10^{2}$ for better presentation, with the exception of Brier score which was scaled by $10^{4}$. DEs of sizes $m=2$ to $8$ are formed by 32 random subsets of 10 ResNet-50 checkpoints from timm; ADs compose a ResNet50/ResNet101/ResNet152/ConvNeXt-Base/ViT-B base model with a sidekick model with FLOPS Balances $\\geq0.1$.  \n\nWe also repeated this analysis controlling for parameter count instead of FLOPS, and observed similar trends. These results and their visualizations will be included in the Appendix in our revision. These results and their visualizations will be included in the Appendix in our revision. We hope our findings on Asymmetric Duos would spark broader interest in model combination methods, including homogeneous DEs, and this direct comparison would contribute to a better understanding of when and how to apply each approach effectively.\n\n---\n\n**[W2] Incremental Novelty.**  \nWe appreciate the reviewer's perspective, but note the empirical novelty of our heterogenous Duos, that can effectively combine two fine-tuned networks of vastly different sizes without joint fine-tuning. Asymmetric Duos are uniquely poised to be highly compatible with today's large pretrained models and fine-tuning workflow, and are able to \"interpolate\" the accuracy-FLOPs trade off by adding fractional additional computation w.r.t. to the base model, introducing finer control over the scaling laws at minimal computational cost.  \n\n---\n\n**[Q1] Scales of Un-normalized Logit Vectors.**  \nThis point on a potential scale mismatch of raw logits is easy to overlook, we thank the reviewer for the careful observation.  \n\nYou're right that naively averaging logits from heterogeneous models can introduce bias due to norm mismatches; an issue we also observed in the Unweighted (ablation) Duos. Our AD strategy specifically addresses this through temperature-based logit scaling. Since scaling logits by a temperature also scales their outputs' norm linearly, the learned temperatures implicitly adjust the scale between model outputs. For instance, on the ImageNet validation set, EfficientNet-V2-L (145M parameters) and EfficientNet-B0 (5M parameters) have average logit vector l2 norms of 24.50 and 33.07, respectively. Our learned temperatures (1.64 for V2-L and 0.86 for B0) adjust these to 40.21 and 28.31, effectively up-weighting the base model’s contribution.  \nWe observe similar results for all Duos on all datasets, and will include this discussion in the revision to alleviate concerns around this point.\n\n---\n\nWe thank the reviewer again for the thoughtful feedback. We hope our responses adequately address the concerns raised and would be happy to discuss further during the reviewer–author discussion period."}]

Author rebuttal:

Meta-review: This well-written and well-motivated paper proposes a new approach to uncertainty estimation of large-scale models that is computationally savvy. It has been evaluated by 4 knowledgeable reviewers who all agreed that it is acceptable for NeurIPS (3 straight accept scores, one marginal accept), despite several noted weaknesses (some of which the authors have managed to address in the rebuttal and discussion with the reviewers). The authors are strongly encouraged to carefully reflect on the feedback received in the final revision of the paper.
Decision: Accept (spotlight)