Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
Authors: Firas Gabetni
Giuseppe Curci
Andrea Pilzer
Subhankar Roy
Elisa Ricci
Gianni Franchi
Venue: ICLR 2026 Poster (2026)
Keywords: Uncertainty quantification;ensembling approaches;
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings. Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles, an efficient transformer-based ensemble that prunes attention heads to create diverse members and merges them via a new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Experiments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, in zero-shot classification on ImageNet-1k, our approach surpasses state of the art methods, even without requiring additional training.

Published as a conference paper at ICLR 2026
ENSEMBLING PRUNED ATTENTION HEADS FOR SAARRTAINTY-AAARE EFFICIENT TRANSFORMERS
Firas Gabetni'; Giuseppe Curci" Cur 2; Andrea Pilzer", Subhankar Roy", Elisa Ricei Gianni Franchi 1.6
U2IS, ENSTA, Institut Polytechnique de Paris 'University of Trento "NVIDIA
University of Bergamo, Italy
FFondazione Bruno Kessler (FBK) AMIAD, Pôle Recherche, Palaiseau
ABSTRACT
Uncertainty quantification (UQ) is essential for deploying deep neural networks in safety-critical settings, Although methods like Deep Ensembles achieve strong UQ performance, their high computational and memory costs hinder scalability to large models. We introduce Hydra Ensembles, efficient transformer- based ensemble that prunes attention heads to create diverse members and merges them via new multi-head attention with grouped fully-connected layers. This yields a compact model with inference speed close to a single network, matching or surpassing Deep Ensembles in UQ performance without retraining from scratch. We also provide an in-depth analysis of pruning, showing that naive approaches can harm calibration, whereas Hydra Ensembles preserves robust uncertainty. Ex- periments on image and text classification tasks, with various architectures, show consistent gains over Deep Ensembles. Remarkably, zero-shot classification ImageNet-Ik, our approach surpasses state of the art methods, even without requiring additional training.
INTRODUCTION
Deep neural networks (DNNs) excel in vision, language, and multimodal tasks, yet are prone to mak- ing overconfident errors (Hein et al. 2019)). This unreliability in predictions is especially concerning in safety-critical domains such as healthcare and autonomous driving, underscoring the importance of studying Uncertainty Quantification (UQ) in DNNs (Gawlikowski et al. 2023).
Several UQ approaches have been explored, including evidential methods (Sensoy et al. 2018), con- formal prediction (Mollaali et al., (025). Bayesian inference (Kendall & Gall (017). and ensemble methods (Lakshminarayanan et al. 2017 Laurent et al. (003)). Currently, the most reliable method for UQ Deep Ensembles (Lakshminarayanan et al.] (017)) 7) that aggregates predictions from multi- ple independently trained models. Although highly accurate, Deep Ensembles is extremely costly, as it requires multiple rounds of pre-training and fine-tuning, storing several checkpoints, and per- forming many number of forward passes the number of models, making it both slow and memory-intensivee especially for foundation models (e.g. CLIP (Radford et al. 2021). BERT (De- vlin et al., 2019)). Several efficient alternatives exist. such as MC Dropout (Gal & Ghahramani 2016), MIMO (Havasi et al. 2020), Batchen (Wen et al. 2020), and Packed Hnsembles (Lau-) rent al. 2023), which lower computational costs by reusing weights or sharing parameters. How- ever, they still require full pre-training, and scaling them to transformer-bast Dosovitskiy et al. 2020) large-scale foundation models computationally expensive.
(Vaswani et al. 2017
To still leverage the power of Deep Ensembles for UQ while reducing the inference cost, a naive approach is pruning or discarding unimportant weights of each constituent model. Contrary con- ventional wisdom, in this work we show that commonly used pruning methods (Molchanov et al.) 2019 He al.) 2020), despite their ability to preserve accuracy, can, in fact, harm calibration and lead to unreliable predictions. We prove theoretically and empirically under which conditions prun- ing degrades performance (predictive uncertainty). and consequently raise a pertinent question: How can we leverage ensembles of transformers for efficient and effective UQ?
Equul contribution.

---

Published as a conference paper at ICLR 2026
Motivated by this challenge, develop framework for ensembling pruned transformers for ef- ficient and effective UQ. which is amenable to large-scale models like CLIP and BERT. In detail, our framework, Hydra Ensembled" aims to preserve the diversity of multiple models, like Deep En- sembles, while maintaining the inference cost close to a single model. Unique to our approach, we generate diverse models via pruning attention heads from single pre-trained transformer model. These pruned subnetworks are then merged into a single model using Grouped Fully Connected (GFC) layers (Lafage al., 2025a). Importantly, unlike previous approaches (Liu al., 2021 Le et al., 2020), Hydra Ensembles does not require training each member from scratch, and can even operate without fine-tuning, offering computational advantages both at training and inference.
We evaluate our proposed Hydra Ensembles on three different classification tasks: image classi- fication with ViT (Dosovitskiy et al., 2020), text classification with BERT (Devlin et al. 2019). and zero-shot classification with OpenCLIP-Vil demonstrate that Hydra Ensembles is competitive to Deep Ensembles both in terms of accuracy and calibration metrics, while greatly reducing the inference costs. Notably, zero-shot ImageNet-IK
(Ilharco et al. 2021). Our experimental results
OOD benchmarks, Hydra Ensembles surpasses the state of the art ViLU (Laton et al. 2025 method (which requires training) by +1.3 AUROC, -3.5 FPR95, and +4 AUPR (Table
Our main contributions are: (i) We investigate the impact of pruning transformer-base models on UQ and demonstrate both theoretically and empirically that naive pruning can lead to poorly calibrated uncertainty (Section (.2). (ii) We introduce Hydra Ensembles, the first pruning framework specifically designed for UQ in transformer-
large-scale models (Section 4 (iii) We show that Hydra Ensembles delivers uncertainty estimates that are comparable to Deep Ensembles, while significantly reducing computational costs both during training and inference (Section(5).
2 RELATED WORK
Transformers and Uncertainty Quantification. Estimating epistemic uncertainty in DNN is chal- lenging. A common approach is to approximate the intractable posterior distribution over the model's weights. Bayesian methods such as Variational Inference (Graves) 201T Ranganath al., 2014) or Markov Chain Monte Carlo (MCMC) (Chen et al. 2014 Neal 2012 are theoretically sound but often too computationally expensive to scale to large transtormer models. Deep Ensem- bles (Lakshminarayanan et al. 2017) remain the gold standard for accuracy and calibration, but their cost grows linearly with the number of models, making them impractical for very large ar- chitectures. Lighter alternatives such as MC Dropout (Gal & Ghahramani] (0166). MIMO (Havasi) jet al. 2020), Batchensemblee (Wen et al. 2020), MaskEnsembles (Durasov et al. | (022)) and LoRA- Ensemble (Mühlematter et al.) [2024] Wang et al. 2023) reduce computation by sharing most of the backbone, but this limits the independence of ensemble members. Bocked-Ennsemble.. (Lau-) rent et al.] 2023) maintain stricter independence but reduce per-member representation capacity. For CLIP (Radford et a.. (021) and other Large vision-language models, post-hoc methods like Bayes VLM (Baumann al. (024) (Laplace approximation on the last layers) and ViLU (Lafon) ct al. 2025) (adding a lightweight error-prediction head) have been explored. While convenient,
post-hoc approaches generally yield weaker uncertainty estimates. Overall, ensemble-style methods provide stronger uncertainty but either require costly retraining or sacrifice diversity for efficiency. Hydra Ensembles offers a balance: preserves member diversity, avoids retraining, and achieves near single- model inference cost.
Network Pruning. Neural network pruning is widely used to reduce model size and computational cost while retaining predictive accuracy. Pruning removes less important weights to preserve accu- racy (Chauvin 1988 MMochanov et al.]20199) [2019 He et al., 2020 [un al. (025). but it often degrades robustness to noise and impairs uncertainty estimation (Liebenwein et al. 2022 Several prior works leverage pruning to construct ensembles, including unstructured pruning with complementary sub- networks (Whitaker & Whitley| (022), stochastic masking (W Whitaker & Whitley, 2024), and sparse training from scratch of pruned subnetworks (Liu et al.] |021] Le et al. (020). While these methods improve accuracy, they typically require retraining full models, rely slow iterative pruning. or use unstructured pruning that provides little to no speedup. In contrast, Hydra Ensembles lever-
In Greek mythology, the Hydra is serpent-like creature with multiple heads, and it is famously known for its regenerative ability: when one head is severed, two more grow in its place.
2

---

Published as a conference paper at ICLR 2026
ages pre-trained models and structured head-level pruning to build diverse subnetworks efficiently, avoiding retraining while improving accuracy and UQ.
ON PRUNING AND UNCERTAINTY QUANTIFICATION
In this section, we first introduce some preliminaries on UQ and pruning, and then present our initial theoretical result examining the effeet of pruning performance under noisy data.
3.1 PRELIMINARIES
Uncertainty Quantification. Consider a deep neural network (DNN) So(.) with parameters IRD trained on data D (x,, Yt }==e Uncertainty predictions usually split into two types (Hüllermeier & Waegeman 2021 (i) Aleatoric uncertainty: noise or ambiguity in the data,
(i) Epistemic uncertainty: limited knowledge of the model parameters.
2013). the prediction of DNN on input can be interpreted as condi- tional likelihood: fo(z) Ply 0. In practice, simple UQ technique is the MSP, maximum softmax probability (Hendrycks & Gimpel 2017). A more reliable method is Deep Ensembles, i.e. training several networks with the same architecture but different initializations:
Following Blundell et al.
Ply x))
1 M
Σ
Ply ((m) .).
Ensembles capture both types of uncertainty but are computationally expensive, since each network must trained fully. In this work, we explore pruning as a way to alleviate these costs while retaining the benefits of ensembles.
Pruning. Pruming reduces the number of weights DNN while keeping accuracy close to the original model. Formally, it seeks a smaller parameter set IRd (d < D) such that the loss L(-) of the DNN on dataset 'D is equal if we reduce the number of parameters:
(D (D)
Classical methods such as Optimal Brain Damage (LeCun et al. 1989) and Optimal Brain Sur- geon (Hassibi & Stork 1992) analyze the sensitivity weights using a Taylor expansion of the loss and prune the least important ones, please refer to A. for more details.
3.2 PRUNING UNDER NOISY DATA CONDITIONS
Pruning methods are effective on clean test datasets D as demonstrated by numerous studies on both structured (Kurtié al. 2023 K Kwon al. 2022 Park et al. (023) and unstructured (Kurtic
et al. 2022 Yao et al.) 2021 Xu et al., (022) pruning approaches. However, in the presence of noisy corrupted test dataset 2 pruning may degrade performance (Liebenwein et al., 2021).
Assumption (Clean vs. Noisy Datasets). Let D' denote a clean test dataset and D a noisy (or corrupted) test dataset. Define the loss gap between the two as
AL == Lp- (0) Lp: (0).
We assume that 0 minimizes Lo (training set) and LD: (clean test set), i.e.
VDD = 0, and VL
0,
and that pruning induces a small perturbation 50. Also we assume that the pruning perturbation 80 is aligned (non-negatively correlated) with the gradient of the noisy loss:
(o (0)) 50 2 0,
Proposition (Pruning is Worse under Noise), Suppose Assumption holds. Let H' and H denote the Hessians of the loss D' and Da respectively. H H' > 0, then the loss gap after pruning satisfies ( < AC + 80), i.e., pruning degrades performance more severely on the noisy dataset than the clean one.

---

Published as a conference paper at ICLR 2026
Clossifter
Gessifier
Classifier
Clossifier
Menge
Prune
Layer
Merged
Prune
Fuse
Multi-Hiedd Atttrttion
Fused Multi-Head Attention
Prune
N'x
Loyer Norm
x
Podtioonl
M
Enced rg
Pruned head
Maintained head Mcdel
Enceding
Toput Enhecdings
Mointained head Model
Trput Enbed
Mointoined head
Figure 1: Hydra Ensembles. We start from a single transformer backbone and prune its attention heads create multiple diverse subnetworks. These subnetworks are then combined at the head level into a Fused Multi-Head Attention (MHA), and then also merged at the MLP level, as described in Section 4.2 The pruned heads and MLPs can either be fine-tuned or kept frozen. Transformer heads are shown in matrix form for illustration only.
The proof of the proposition is present in Appendix A..2 suffer from stronger degradation of performance than unpruned models when evaluated on noisy data. Appendix B. cxamines whether the hypothesis required for Proposition [holds. Hence, such pruned networks cannot be directly relied upon for uncertainty quantification. This raises natural question: how can we leverage ensembles of pruned models a way that remains effective for uncertainty estimation?
This shows that classically pruned DNNs
A possible improvement is to finetune the pruned models, but for some foundation models this may expensive. A promising alternative comes from circuits (Olah et al. 2020), developed in mechanis- tie interpretability. A circuit is a subgraph of the network that performs a semantically meaningful subcomputation (c.g., feature or an attention head). Instead of removing weights blindly, one extracts subnetworks that preserve useful functionality. Recent methods, such as the Headmap al- gorithm (Wang et al., 2025), allow systematic extraction of circuits that remain stable under noise. In this paper we propose to build ensembles of such structured subnetworks, combining the benefits of pruning with robustness for UQ Our approach is described in the following Section.
In the supplementary material, we provide additional discussions (Appendix B that help deepen the understanding of our method. In particular, Appendix [B.2 explores the circuit-level representations involved in UQ. We show that certain attention heads are highly specialized for this task, and pruning them can be particularly harmful. This specialization also illustrated in Figure 4 making this section a unique contribution that we invite the reader to examine closely.
4 EFFICIENT ENSEMBLES WITH HYDRA ENSEMBLES
Hydra Ensembles (Fig. T is based on the idea of building an efficient ensemble of pruned Transform- ers. More specifically, we perform structured head pruning multiple copies of the same backbone (each pruned differently) and then merge the remaining attention heads into a single Transformer. This design preserves ensemble-like diversity and ensures that inference remains fast and efficient, since it does not require to run one forward pass for each model sequentially. In the following, we present the details of our approach. Subsection 4 describes how efficiently transform ensem- ble pruned models into a single Hydra Ensembles model, while Subsection 4.3 illlustraees how the individual pruned members of the ensemble are obtained.
4

---

Published as a conference paper at ICLR 2026
4.1 TRANSFORMER PRELIMINARIES.
A layer I of a pre-norm Transformer with L layers, hidden size d, H attention heads (head dim.
dx =d] d/H)) H) and sequence length T, processes input Xz, E RT'xd follows:
R1 = L
Yie- = Xe,e + (H.( X1 - Yee MLP(LN(Y..)) (1)
In multi-head attention (MHA), the input X₃ linearly projected into H sets of queries @(h),
keys K(h,, and values V(h), for each head h. The output of head h is computed as Z(h)
(K())))) V(O), and concatenates all heads. The Multi Layer Perceptron (MI.P) is
softmax vd. two-layer feed-forward network with nonlinearity a.
4.2 PROPOSED METHOD
We introduce Hydra Ensembles, an efficient ensemble method that prunes at the attention-head level.
Why prune attention heads? Each Trans- former layer has two main parts: MHA and MLP. MoE methods (Fedus et al. (2022) usually operate pruning on the MLPs, since they con- tain more parameters (e.g.. in ViT-B/16: MLP
4.7M vs. MHA 2.3M). In this paper in- stead, we propose to prune attention heads be- cause: (i) MoE already exploits MLP specializa- tion, pruning MLP will turn them less com- patible with MoE architectures; (ii) Head-level pruning is simpler for model merging, since adding heads is cheaper than merging full MI.s and better fits circuit extraction.
Dot-Prociact Attention
K
V
BFC (1r5
GFC (FC ane
BFC GFC (n
Hydra Ensembles Setup. Assume we have M pruned models ) m==1 all variants of the same original trained model fe. Each differs only in the set of surviving heads after pruning.
Figure 2: Illustration of Fused MHA.
For layer f, the input is the concatenation: X1,e= = [x x
(M) [
RTxx where
x(m) the input for model m. Our goal is to build a single Transformer that fuses these M models, performing ensemble inference single forward pass. This new transformer architecture consists of two principal components: the Fused MLP and the Fused MHA, which we describe below.
Merged MLP. At each MLP layer, we average the weights and biases across the M models:
w
Σ
T M
Σ b
())m)
Here, the superscript j the weights and biases indicates the j-th fully- colle-connected layer of the MLP. This produces a merged MLP (merge)
Fused MHA. The input at layer is first reshaped: X E MTx + X,, E ERTMd
Each pruned model retains its own of active heads, with total dimension de Hedk, where He denotes the number of heads remaining after pruning. For each model m, the projection matrices are:
W ((m) E Rdxx,
(-((m) E Rdxxe
WV E dRdx
Using Grouped Fully- Fully-Connected (GFC) layers (Xie et al. 2017: Laurent et al. 2023 Lafage et al., 2025a), queries, keys, and values are computed jointly:
Q: = GFC((E;; W(0) W
w,0
W.(M) E R TTMd
and similarly for Ke and Vz-
5

---

Published as a conference paper at ICLR 2026
The attention heads are then computed as:
A softmax 1
Vda Q²"h (K(()) RTXT
(2)
2(m (h) (1)) E RTdR.
(3)
Let Ze E Rx333 be the concatenation of all heads. The fused MHA is then:
MHA, (fuse)
(X.,)= GFC(Z;; WoW) wo(2) ..... w0(()) (...,,,((M))
RTTMd
where W Wo(m)) E Rd are the output projection weights. Please refer Fig for an illustration.
Final fused Transformer layer. The fused Transformer layer is therefore:
L = LN. (X,,)) E MTxd
(4)
V., = reshape (X.,) MHA (Fune)
(5)
PL. = LN! reshapee (Y.)) E IR MTTx
(6)
X1,41 = ressaeeehheaaae - MTxa (X.)) + MLP (mergo) (Pre) E E MTxd
In summary, the final fused Transformer block first normalizes the inputs from different models, then applies fused MHA reshaped representation where model-specific inputs are stacked along the depth dimension, reducing the token load for attention. The model-specific outputs are then reshaped back to the token dimension, passed through MLP with residual connections, producing the next-layer representation. This final layer with reshaping ensures that MHA (fuse) processes the same number of tokens as a standard MHA, while (werge) uses the same number of features, allowing it to be used without fine-tuning, This greatly improve efficiency (see Appendix B.4).
4.3 How TO CONSTRUCT ENSEMBLE MEMBERS
We have illustrated the procedure for merging multiple models into a single ensemble. We now describe our approach for creating the individual ensemble members, proposing two strategies de- pending on the available data.
Strategy 1:
No access to an uncertainty validation set. If no validation set is available, we propose to use a classical structured pruning with the Taylor method (Molchanoy et al. (019). This prunes the least important heads while keeping the architecture intact. This approach computationally efficient and well-suited for structured pruning at the head level, making it possible to remove the least important heads while keeping the overall model architecture. We denote this technique Hydra Ensembles (Taylor).
Strategy Access to an uncertainty validation set. If noisy or incertainty-oocusee validation data exists, we use circuits. In particular, the Headmap method (Wang et al. (025) identifies which heads matter most for uncertainty, and removes the rest. This strategy is more targeted than standard Taylor pruning, as it explicitly considers extracting circuits for given task. We denote this technique Hydra Ensembles (Circuit).
Fine-tuning or zero-shot usage? Pruned models can either be fine-tuned to recover accuracy, or used directly in a zero-shot way. Based on Proposition [ we argue that when pruning is performed without circuit-based strategies (e.g., using only Taylor pruning), it is generally preferable fine- tune the models. This is because zero-shot pruned models may lack robustness when applied to uncertainty quantification tasks.
When fine-tuning is chosen, each pruned model is trained independently on the same dataset. For example, in the case of ensemble with M members, obtain M distinct pruned models fg(m) These models are then fine-tuned on the same training dataset by solving the following optimiza- tion problem: g(m)+ = arg ming(o.) Lx(((m)) This procedure produces M specialized models. Appendix B.5 analyzes the diversity introduced by this fine-tuning strategy.
For supervised image and text classification, our method is fine-tuned; for zero-shot image classifi- cation, it is not. All experiments use M for ensembling, standard choice in Deep Ensembles.

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTAL RESULTS
We evaluate our method on three tasks: supervised image classification, zero-shot image classifica- tion, and text classification. For supervised tasks, models are fine-tuned; for zero-shot, they are not. All experiments use M = for ensembling, a standard choice, see e.g. (Lakshminarayanan et al.) 2017). The average number of heads per-layer is indicated in each section. Full implementation details of our experiments are in Appendix C We also provide a short study on cost/benefit of using more than 3 members in Hydra Ensembles in Appendix B.6
5.1 SUPERVISED IMAGE CLASSIFICATION
Datasets, metrics and architecture. We evaluate our approach considering both the ImageNet- IK (Russakovsky et al. 2015) and CIFAR-100 (Krizhevsky) (009) datasets. For ImageNet- IK, we use the standard train/ valiationo split, while for CIFAR-I00 we consider the official train/test split. We perform evaluation considering ViT-B/16 backbone (Dosovitskiy et al., 2020). Unless stated otherwise, VIT backbones are pre-trained on ImageNet-2IK (Ridnik et al. 2021) and fine- tuned on the target dataset. Following previous works (Lafage et al. 2025b). for In-Distribution (ID) evaluation, we report Top-1 Accuracy along with standard calibration metrics: Brier score, negative likeliloodd (NLL). expected calibration error (ECE) (Guo et al., (017). and adaptive ECE (aECE). For out-of cuttoo--distrbuttoon (OOD), we report AUROC, FPR95 and AUPR (Hendrycks & Gimpel 2017) and follow the OpenOOD benchmark (Yang et
2022) splits for both datasets.
Baselines. We compare several methods: (i) a single transformer architecture (SINGLE), acting as a reference; (ii) the state-of-the-art approach DEEP ENSEMBLES Lakshminarayanan et al. (017). built from independently pretrained and trained
RT ViT-B/16 models with different random seeds; (iii)
efficient ensemble methods such as BOKKED-ENSEMBLES (Laurent et al., 2023). BATCH ENSEM- BLES (Wen et al. (220) and MIMO (Havasi et al., 2020 (iv) methods based on parameter-efficient
adapters or dropout (LoRA ENSEMBLES (Mühlematter et al. 2024) and MC DROPOUT (Gal
& Ghabramani 2016)) (v) previous pruning baselines such as the unstructured pruning method
OBAF et al. 2025 the structured Taylor-based method in (Molchanov et al. 2019) (TAYLORR)
and CIRCAVG, which extracts circuits using OOD and accuracy losses (see Appendix We also test two variants of our method: HYDRA ENSEMBLES (TAYLOR) and HYDRA ENSEMBLES (CIRC). Details about post pruning fine-tuning Appendix D.2
Results. Table T shows the results on classification and OOD detection. On ImageNet- K, Hydra Ensembles surpasses mostly all baselines and narrow the gap with Deep Ensembles in terms of accuracy (-1.3%) while improving slightly average OOD performance (AUROC -0.8% and AUPR +0.4%). Compared with the original transformer model (SINGLE), adapter-based methods and other efficient ensembles, Hydra Ensembles achieves better robustness. A similar trend can be observed for CIFAR-100 dataset. Compared to Deep Ensembles, Hydra Ensembles shows its largest advantage in UQ consistently improving on average OOD metrics (AUROC +3.4%, FPR95 -2.2%, AUPR + 1.2%). Additional results including experiments on distribution shift and results reporting mean/std across multiple seeds for Hydra Ensembles are reported in Appendix
Beyond accuracy and robustness, we also examine the computational cost of Hydra Ensembles in Figure 3 Interestingly, in bfloat16, Hydra Ensembles achieves nearly the same inference cost as the single model, with a ratio Hydra Ensembles/ Siiglle of only 1.07x for both whole-test runtime and per-batch inference. By contrast, the Deep Ensembles approach is almost three times slower under the same conditions and have more than twice as many parameters (see also Appendix [B.4).
Although MLP fusion could in principle reduce ensemble diversity when applied post training, but in Hydra Ensembles this does not occur in practice because diversity is already assured by dif- ferent attention heads representations learned during separate member training, and observe no degradation in ID or OOD metrics (see Appendix B.7
2 Each ensemble member uses the same architecture (VIT-B/16) but diffferent random seeds, and is pretrained and fine-tuned separately, yielding diverse weights.
'OBA (Sun et al. 2025) prunes weights on the entire model. To match attention-heet pruning, we retain 89.1 % of parameters for ImageNet-1K K (8 heads per MHA block) and 94% for CIFAR-100 (se 10 heads per block).
7

---

Published as a conference paper at ICLR 2026
Table 1: Results on ImageNet-IK and CIFAR-100 datasets. Metrics evaluate accuracy, calibra- tion performance and OOD detection. Best in bold, second-best underlined.
IMAGENET-IK
CIFAR-100
Method
Heats Ace! Birker
N.L. FOE, aFCE
000 Ang
Ace?
Brier)
NEL. ECR, aFCE.
0O0 Ang
AUROU
FPH95)
AUPRI
AUROUS
FPR95]
AUPPI
SINGLE
12
80.07
0.27
0.31
0.11
0.81
55.40
50.25
00.91
92.15
0.11
0.25
0,017
0.005
35.40
40.27
93.55
DeEr EXSeMBLES
12
81.19
0.25
0.65
0.01
0.81
85.43
46.93
02.22
93.52
0.09
0.22
0.01
0.01
36.08
35.07
94.26
PAUKED ENSESILES
12
79.21
0.29
0.718
0.01
0.81
553.20
51.05
58.17
12
99163
0.83
0.31
0101
01188
36.19
35.54
M1.48
MIMO
12
80.99
0.27
0.72
0.01
0.81
83.63
52.64
59.14
92.62
0.10
0.23
0.009
0.008
88.08
37.00
95.15
ENSENILLS
12
80.53
0.31
0.006
86.27
53.66
84.74
ENSEMBLES
90.68
0.71
85.77
OBA
78.52
$4.90
85.85
OR
80.08
0.19
0.00
54.51
49.44
0.82
0.01
86. 7M
99.05
CIRCAVG
80.23
0.26
0.77
000
0.02
85.21
50:35
62.41
10
91.87
0.12
0.50
0.01
0.01
89.77
$7.79
95.31
IVDRA EANK (T (Tamin))
Sel
X1:30
0.76
0.15
0
0.81
85, 16
90:50
06.35
10c1
92.00
0.81
11208
01001
01:877
85.89
90.57
98.46
HYDRA ENs (CINc)
863
90.68
0.27
0.74
0.01
0.81
86.29
47.62
63.05
1063
92.11
0.12
0.28
0.005
0.006
99.63
36.44
95.17
Runtime per (ms)
Perameters (N)
MuctiplyAcos (G)
Deep
25.36
200
52.7
18:42
21.97
Hydra Ens
6.55
8.34
Singl:
6.15
87
17.6
5
10
15
20
25
0
50
100
150 Millions
200
0
40
50
GigaOps
Figure 3: Inference costs of Deep Ensembles, Single, and Hydra Ensembles with heads on ImageNet-Ie Ik. We report runtime per batch under BF16/FP32 (left), as well parameter count (middle) and multiply-adds (right).
5.2 SUPERVISED TEXT CLASSIFICATION
Datasets and architecture. We also test SST-2 (Socher et al. 2013). sentiment analysis bench- mark with binary labels, using a bert bass-aaaauuae -aceeeuucceeeddde model (Devlin et al. 2019) fine-tuned on the SST-2 training set (Socher et al. 2013).
Baselines. We consider similar baselines than in the image classification setting. However, for the Deep Ensembles baseline, avoid training three completely independent BERT (Devlin al., 2019 models from scratch to reduce computational cost. Instead, we fine-tune shared pre-trained BERT backbone and train three separate classifiers with different random seeds on top of (we denote it as DEEP ENSEMBLES (D)). More generally, in the supervised text classification setting, we only include those baselines from the image classification experiments that do not require re- training large models from scratch, due to the computational complexity of BERT. For all pruning approaches, we keep 6 heads per attention block. Please refer to Appendix D.2 for details on fine- tuning. We consider both variants of our method.
Results. Table ID accuracy, calibration, and OOD detection performance for different un- certainty methods. Deep Ensembles achieves the best overall performance but at high computational cost bostt three times a single model. while Hydra Ensembles in float16 they match single-model time. Compared to Deep Ensembles (D), classical lightweight approaches such MC Dropout and LoRA Ensembles are close in accuracy but show weaker OOD detection. Our proposed Hydra En- sembles performs par with it in terms of accuracy and calibration, while significantly improving OOD detection: AUROC (+2.8)), FPR95 (-7.6% and AUPR (+2.2%).
5.3 ZERO-SHOT IMAGE CLASSIFICATION
Datasets and architecture. We consider the OpenCLIP- OpenCLIP-ViT/B-22 model (Ilarco et al. 2021
and nine standard datasets covering diverse domains: ImageNet-Ik, CIFAR-100 and CIFAR 10, Food101 (Bossard et al. 2014). SUN397 (Xiao et al., 2010). Oxford Pets (Parkhi et al. DTD (Cimpoi et al. 2014), EuroSAT (Helber et al. 2018) and 101 (Fei-Fei et al.
2012)
2004
8

---

Published as a conference paper at ICLR 2026
Table 2: Results on text classification. Metrics evaluate accuracy, calibration performance and OOD detection. Best in bold, second- best underlined.
Method
Heads Ace:
Reier,
NLT.)
ECE)
sECR.
OOD Average
AUROC
FPR95
AUPRT
SINGLE:
92.55
0.12
0.27
0.05
0.04
70.16
70.62
$1.93
DEEP EXSEMBLES (D)
0.11
0.24
0.04
0.04
74.81
62.69
$4,9
MC DROPOUT
92.55
0.13
0.31
0.05
0.04
72:23
67.36
LoRA ENSIMBLES
12
92.89
0.12
0.28
0.05
0.04
70.83
68.7
82.27
TAYLOR
92:41
0.13
0.35
0.06
0.06
70.95
68.57
82.11
CBRCAVG
92.78
0.33
0.05
HYDRA-ENS ESS (TAYLOR)
663
0.12
0.29
0.05
0.004
71.85
62.59
82
HYDRA. ESS (CIRC)
0.34
77.6
Table 3: Results on zero-shot classification. Metrics evaluate accuracy, calibration performance and on the ImageNet-Ik OOD set. Best in bold, second-best underlined.
OOD Average
Train
Acct
Brier]
NLLŞ
ECE,
aECE!
AUROCI
FPR95;
AUPR
SINGLE
23,65
0.36
0.98
8.68
8:57
70.76
75.20
37.73
TEMP: SCALING
12
73.65
0.36
0.90
4,03
3,80
70.76
75.20
37.73
BAVESVLM
5.86
VILU
12
9.50
8,86
75.38
71.59
43.81
TAYLOR
60.75
0.53
13.76
13.67
33.64
CURCAVG
10
71.91
0.38
0.99
4,00
4.11
76.88
68.26
47.64
HYDRA (CIRC)
0.36
3.38
All datasets but SUN397 come from torchvision, whose source instead is Hugging Face. Fol- lowing Radford et al. (2021), we use "A photo of a {labe1}. as prompt template. See Appendix C.3 for additional details.
Baselines. We compare our method against recent UQ baselines which care developed based on C (i) BAYESVLM (Baumann et al.) 2024), which improves uncertainty estimation via Laplace approximation using subset of LAION (Schuhmann al. 202T]; (ii) VILU (Lafon) et al. (025), which trains an uncertainty predictor as dataset-specife binary classifier to distin- guish correct from incorrect predictions, thereby affecting only ECE and OOD performance; and (iii) TEMPERATURE SCALING (Guo et al., 2017). applied
small portion of the training set of ImageNet-11 1k following the setup described in Baumann et af. 2024). As in previous experiments, also report the results obtained with pruning- pruning-based approaches. For our method, we propose two setups: one with pruning only on the vision encoder, and another with both encoders pruned. More- over, we the effect of fine-tuning on a small subset of LAION-400M for the first one. Here we present the results with both encoders pruned. See Appendix D.3 for the details about the implemen- tation and Appendix 1pppendxxEE3.22tt65 for the results with the pruning only on the vision encoder in zero-shot and fine-tuned settings. Here, we only report the performance of the best-performing version of our method, Hydra Ensembles (Cire).
Results. Table 3 presents our results for classification averaged across the datasets, and for the ImageNet- Ik OOD set, following the same protocol as in the supervised setting. Disaggregated results and on the OOD of CIFARION 100, where Hydra Ensembles improves the baselines as well, are reported in Appendix E.3.I Hydra Ensembles achieves the best accuracy, ECE, aECE and the second-best NLL, all without any training. For OOD detection (see also disaggregated results in the Appendix), the only competitive baseline is ViLU, but our method surpasses it with substantially better AUROC (+1.3%), FPR95 (-3.5%) and AUPR (++4.0%).
CONCLUSIONS
In this work, we introduced Hydra Ensembles, structured pruning approach for building efficient transformer ensembles. By aggregating diverse pruned heads into a single model, Hydra Ensem- bles provides uncertainty estimates comparable to, and in some cases surpassing, Deep Ensembles,
Many UQ methods cannot he transferred directly CLIPP.

---

Published as a conference paper at ICLR 2026
while maintaining near-single-mode inference speed. Experiments both large-scale and small- scale image classification datasets, text classification, and zero-shot image classification demonstrate substantial improvements in OOD detection and calibration, occasionally even without retraining. We further show, both empirically and theoretically, that structured pruning of attention heads can meaningfully affect UQ, enhancing the separation between IID and OOD representations when heads are carefully selected, whereas naive or unstructured pruning may be detrimental. These results indi- cate that careful head selection preserves both accuracy and reliable uncertainty, positioning Hydra Ensembles as scalable, memory effireeefficieet, and robust framework for uncertainty quantification in modern deep learning models.
REPRODUCIBILITY STATEMENT
We have taken multiple steps to ensure the reproducibility our work. All experimental setups are described in detail in Scction including datasets, evaluation metrics, and architectures. Additional implementation details, such dataset splits, hyperparameter settings, pruning strategies, and fine- tuning protocols, are provided in the Appendix (see in particular Appendix C.I.5 C and (.2). For zero-shot experiments, we describe our CLIP baselines and pruning variants in Appendix C.3 and D.3 along with extended results. All theoretical assumptions and proofs are presented in the main text and appendix. We will release the complete source code, training scripts, and pretrained checkpoints upon publication to further facilitate reproducibility and extension of our results.
ACKNOWLEDGMENTS
This work was granted access the HPC resources of IDRIS under the allocation ADO1 AAD01001077 and the allocation ADOI 1016325 made by GENCL The authors also acknowledge the CINECA award under the ISCRA initiative for the availability of reg-pperorrance computing resources and support.
REFERENCES
Anton Baumann, Rui Li, Marcus Klasson, Santeri Mentu, Shyamgopal Karthik, Zeynep Akata, Arn Solin, and Martin Trapp. Post-hoe probabilistic vision-language models. CoRR, 2024.
Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out-of- distribution detection evaluation. arXiv preprint arXiv:2306.00826, 2023.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. nents with random forests. In European conference on computer vision, pp. 446-461. Springer, 2014.
discriminative compo-
Yves Chauvin. A back-propagation algorithm with optimal use of hidden units. Advances in neural information processing systems, 1, 1988.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference on machine learning, pp. 1683-1691. PMLR, 2014.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
10

---

Published as a conference paper at ICLR 2026
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Reprexentations, 2020.
Nikita Durasov, Timur Bagautdinov. Pierre Baque, and Pascal Fua. Masksembles for uncertainty estimation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 15539-13548, 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39) 2022.
Li Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178 IEEE, 2004.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, et al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513-1589 2023.
Alex Graves. Practical variational inference for neural networks. Advances in neural information processing systems, 24, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M Dai, and Dustin Tran. Training independent subnetworks for robust prediction. International Conference Learning Representations, 2020.
Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter prun- ing criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2009-2013 2020.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF conference computer vision and pattemn recognition, pp. 41-50, 2019.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE international geoscience and remote sensing symposium, pp. 204-207. IEBE, 2018.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: /2261, 2019a.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2019b. URL httpthtss////ppeeeeeee http:eiwwwweeeeeoee/////iio/ee
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of distribution examples in neural networks. In International Conference on Learning Representations, 2017.
11

---

Published as a conference paper at ICLR 2026
Rui Huang and Yixuan Li. Mos: Towards scaling out-of-distributon detection for large semantic space. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8710-8719, 2021.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Muchine learning, 11033)457-506, 2021.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? Advances in nearal information processing systems, 30, 2017.
Jaeyoung Kim, Kyuheon Jung, Dongbin Na, Sion Jang, Eunbin Park, and Sungchul Choi. Pseudo outlier exposure for out-of-distribution detection using pretrained transformers. arXiv preprint arXiv:2307.09455. 2023.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022.
Eldar Kurtić, Elias Frantar, and Dan Alistarh. Ziplm: Inference-aware structured pruning of lan- guage models. Advances in Neural Information Processing Systems, 36:65597-6556 2023.
Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gho- lami. A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101-24116,, 2022.
Gustaf Kylberg. Kylberg texture dataset v. 1.0. Centre for Image Analysis, Swedish University of Agricultural Sciences and .... 2011.
Adrien Lafage, Mathieux Barbier, Gianni Franchi, and David Filliat. Hierarchical light transformer ensembles for multimodal trajectory forecasting. In 2025 IEEE/CVF Winter Conference Ap- plications of Computer Vision (WACV), pp. 1682-1691. IEEE, 2025a.
Adrien Lafage, Olivier Laurent, Firas Gabetni, and Gianni Franchi. Torch-uncertainty: A deep learning framework for uncertainty quantification, 2025b.
Mare Lafon, Yannis Karmim, Julio Rolva-Rodríguez, Paul Couairon, Clément Rambour, Raphaël Sournier-Sniehotie Ismail Ben Ayed, Jose Dolz, and Nicolas Thome. Vilu: Learning vision- language uncertainties for failure prediction. arXiv preprint arXiv:2507.07620. 2025.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Mare Martinez, Andrei Bursuc, and Gianni Franchi. Packed-ensembles for efficient uncertainty estimation. ICLR, 2023.
Duong H Le, Trung-Nhan Vo, and Nam Thoai. Paying more attention to snapshots of iterative prun- ing: Improving model compression via ensemble distillation. arXiv preprint arXiv:2006. 1487, 2020.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yann Lecun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) (1(1)))2278-22222 2002.
12

---

Published as a conference paper at ICLR 2026
Lucas Liebenwein, Cenk Baykal, Brandon Carter, David Gifford, and Daniela Rus. Lost in pruning: The effects pruning neural networks beyond test accuracy. In Proceedings of the th MLSys Conference, San Jose, CA, USA, 2021.
Bo Liu, Liming Zhan, Zexin Lu, Yujie Feng, Lei Xue, and Xiao-Ming Wu. How good are Ilms out-of-distribution detection? arXiv preprint arXiv:2308. 10261, 2023.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with over- head for either training or testing: The all-round blessings of dynamic sparsity. arXiv preprint arXiv:2106. 14568, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning- In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12264-11272, 2019.
Amirhossein Mollaali, Christian Bolivar Moya, Amanda A Howard, Alexander Heinlein, Panos Stinis, and Guang Lin. Conformalized-kame Uncertainty quantification with coverage guar- antees for kolmogorov-arnold networks (kans) in scientific machine learning. arXiv preprint arXiv:2504. 15240, 2025.
Dominik Mühlematter, Michelle Halbheer, Alexander Becker, Dominik Narnhofer, Helge Aasen, Konrad Schindler, and Mehmet Ozgur Turkoglu. Lora-ensemble: Efficient uncertainty modelling for self-attention networks. arXiv preprint arXiv:2405.. 244X::33005/14388. 2024.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIS workshop on deep learning and unsupervised feature learning, number in 2011, pp. 7. Granada, 2011.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5(3):0000240001, 2020.
Seungcheol Park, Hojun Choi, and U Kang. Accurate retraining-free pruning for pretrained encoder- based language models. The Twelfth International Conference on Learning Representations, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pallern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8748-8763. PmLR, 2021.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, - 2021.
16
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. Artificial intelligence and statistics, pp. 814-822. PMLR, 2014.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Olga Russakovsky, Jia Deng, Zhiheng Huang, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):2112255 2015.
13

---

Published as a conference paper at ICLR 2026
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv: 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278-25222 2022.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classifica- tion uncertainty. Advances in neural information processing systems, 31, 2018.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, sentiment
and Christopher Potts. Recursive deep models for semantic compositionality treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-16422 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: //t::///wwww wwww ac web org/ wttp/////ngggg///on.oooo. 1170
Mingyuan Sun, Zheng Fang, Jiaxu Wang, Junjie Jiang, Delei Kong, Chenming Hu, Yuetong FANG, and Renjing Xu. Optimal brain apoptosis. In The Thirteenth International Conference on Learn- ing Representations, 2025. URL /ttp:s////peeeiieee net forum 7id=88rj66XXo0
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 1):1958-1970, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 074661 2018
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4930, 2022.
Xi Wang, Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine- tuning. arXiv preprini arXiv:2310.00033 2023.
Xuehao Wang, Liyuan Wang, Binghuai Lin, and Yu Zhang. Headmap: Locating and enhancing knowledge circuits in 1lms. In The Thirteenth International Conference on Learning Representa- fions, 2025.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. International Conference on Learning Representations, 2020.
Tim Whitaker and Darrell Whitley. Prune and tune ensembles: low-cost ensemble learning with sparse independent subnetworks. In Proceedings the AAAI conference on artificial intelligence, volume 36, pp. 8638-8646, 2022.
Tim Whitaker and Darrell Whitley. Stochastic subnetwork annealing: A regularization technique for fine tuning pruned subnetworks. arXiv preprint arXiv:2401.0888 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Toralla. Sun database: Lcrge-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and patlern recognition, pp. 3495-3492. IEFE, 2010.
14

---

Published as a conference paper at ICLR 2026
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR, 2017.
Runxin Xu, Fuli Luo, Chengyu Wang, Baobao Chang, Jun Huang, Songfang Huang, and Fei Huang. From dense sparse: Contrastive pruning for better pre-trained language model compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11547-11555, 2022.
Jingkang Yang. Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding. Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of- distribution detection. Advances in Neural Information Prucessing Systems, 35:32598-32611, 2022.
Zhewei Yao, Xiaoxia Wu, Linjian Ma, Sheng Shen, Kurt Keutzer, Michael W Mahoney, and Yuxiong He. Leap: Learnable pruning for transformer-based models. arXiv preprint arXiv:2105. 1iti::100..1666, 2021.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image image text tuning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 18123-18133, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6)) 2017.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 16816-16825, 2022.
15

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS SUPPPEEMENTAYY MATERIAL
On Pruning and Uncertainty quantification
17
A.1 Preliminaries
17
A.2 Pruning Under Noisy Data Conditions
17
Discussions
18
B.1
The influence of Zeros shot pruning and Uncertainty quantification
18
B.2
Studying the pruned heads and Uncertainty quantification
18
B.3
Effect of the number of Heads
20
B.4
Computational Cost of Hydra Ensembles
21
B.5
Studying the sources of diversity
23
B.6
Studying the cost/benefit of having more members
23
7
MLP fusion and Diversity
24
Details on Baseline Implementations
24
C.1
Supervised Image Classification
24
C.I.1
Stage I: Pre-training on ImageNet-21k
................................
25
C.1.2
Stage 2: Fine-tuning on ImageNet-Ik
25
C.1.3
Fine-tuning CIFAR-100
25
Packed Ensembles, Batch Ensemble, MIMO and Lora Ensemble
25
C.1.5
Image OOD evaluation details
26
Supervised Text Classification
26
Zero Shot Images Classification
26
D Details on Method Implementations
27
D.I Pruning techniques
27
D.I.I Taylor pruning
27
D.1.2 Circuit extraction and pruning
28
D.2 Post-pruning fine-tuning protocol.
30
D.3 Zero Shot Images Classification
31
Complementary Results
31
Supervised image classification
31
Supervised text classification
32
Zero Shot Images Classification
34
E.3.1 Exiensive Results
34
Effect of Pruning Only the Vision Encoder and Fine- Tuning
35
16

---

Published as a conference paper at ICLR 2026
A ON PRUNING AND UNCERTAINTY QUANTIFICATION
In this section we first recall the classical pruning framework, including its derivation via second- order loss approximations, set the stage for later results. We then extend this analysis noisy data conditions, illustrating why conventional pruning can be especially harmful when test distributions are corrupted. Together, these sections clarify both the foundations and the limitations of pruning, motivating our proposed ensemble-base approach.
A.1 PRELIMINARIES
Let us still consider the DNN fo(.). A pruning algorithm aims to find a subset of weights Ô E Rd with d < D such that (D(O) )) LD(0), so that the pruned model is lighter while maintaining accuracy on the test set: LD (0) = C (0).
Following the Optimal Brain Damage (OBD) method (LeCun et al. 1989). one studies the effect of small perturbation 50 on the loss via second-order Taylor expansion:
(0) A LD 50) (D( = (V))(O)) 80 + 50 (D) 50 o(1150 (8)
order
where H E R is the Hessian of the loss.
Similar to Optimal Brain Surgeon (OBS) (Hassibi & Stork 1992), the classical assumption that
the DNN is trained to a local minimum, so that the gradient term vanishes, and higher-order terms neglected. Hence, the pruning problem be formulated
min min 1/2 50 H00 s.t. e, 50 + 0g -0 60
where the constraint enforces elimination of the weight 0g. while the quadratic term controls the increase in loss.
A.2
PRUNING UNDER NOISY DATA CONDITIONS
This strategy is effective on a clean test dataset D', leading to a wide range of structured (Kurtić et al.
2023 K won et al. 2022 Park al. 2023) and unstructured pruming approaches (Kurtic al.)
Yao et al. |02 Xu et al. 2022). However, in the presence of a noisy or corrupted test dataset n pruning may degrade performance (Liebenwein al. 2021).
2022
Assumption (Clean vs. Noisy Datasets). Let D denote a clean test dataset and D a noisy (or corrupted) test dataset. Define the loss gap between the two
AC = CD (0) L (0).
We assume that minimizes Lp (training set) and Lp (clean test set), i.e.
VLp ( = 0, and VLD = 0,
and that pruning induces a small perturbation 80. Also we assume that the pruning perturbation 60 is aligned (non-negatively correlated) with the gradient of the noisy loss:
()) 80 2 0,
Proposition 1 (Pruning is Worse under Noise) Suppose Assumption I holds. Let H' and H" denote the Hessians of the loss on De and D respectively.
H H' > 0,
then the loss gap after pruning satisfies
AC < AC + 80),
i.e., pruning degrades performance severely the noisy dataset than the clean
17

---

Published as a conference paper at ICLR 2026
Proof. By the second-order Taylor expansion (cf. Eq. equation 3) the difference in loss gaps before and after pruning is
AC + 60) AC(O) =CCD= (0) (6).
Expanding both terms yields
(V (0) pp (0)) 50 + _0 HH" - H') 80 + o(1
Since VL (0) - 0 by Assumption 1, the first-order term vanishes. Thus,
AC + 60) LL (VLDx. 50 + 50 (HT H') 80 + o1160113)
Hf H' and (VLD= (0)) 80 2 the quadratic form is positive for any nonzero 80, which
implies
AC < AC 50)
and
SCp:(0) S 6CD (0).
showing that pruning amplifies the performance gap under noise.
This shows that classically pruned DNNs suffer from a stronger degradation of performance than unpruned models when evaluated on noisy data. Consequently, such pruned networks cannot be di- rectly relied upon for uncertainty quantification. This raises a natural question: how can we leverage ensembles of pruned models in way that remains effective for uncertainty estimation?
B DISCUSSIONS
B. THE INFLUENCE OF ZEROS SHOT PRUNING AND UNCERTAINTY QUANTIFICATION
Proposition states that pruning on uncertainty data is valid the following conditions hold: (VLo" (0)) 80 2 O, and H - H' > O, where the first condition requires that the pruning pertur- bation 80 aligned with the gradient, and the second condition requires that the difference between the Hessians is positive-definite
To verify these assumptions in practice, we apply the TAYLOR pruning strategy to a ViT-B/16 model, evaluating
ImageNet- IK (Deng et al. (2099) and CIFAR- 100 (Krizhevsky) 2209).. as described in Section 5.1 For robustness testing, we additionally use their corrupted counterparts, ImageNet- 1K-C and CIFAR-100-C (Hendryeks & Dietterich 2019a), which introduce common synthetic ruptions (e.g., noise, blur, weather effects) to the original test images.
To test these conditions empirically, we first examine the alignment between the gradient and the pruning perturbation. On CIFAR-100, the inner product between the gradient and the pruning per- turbation, (VLD- (0)) '00, is equal 1.19, while on ImageNet-IIK values are positive, confirming that the perturbation is indeed aligned with the gradient, as required.
0.37. In both cases the
For the Hessian condition, approximate the Hessian by its diagonal due the computational cost of the full matrix. The difference between the two diagonals is strictly positive across both datasets, with average values of 0.45 for CIFAR-100 and 0.17 for ImageNet-1K This suggests that the positive-definttenss assumption holds approximately in practice.
To further support the assumptions of Proposition [ we also evaluate zero-shot pruning on both text and image classification using BERTDevvin et al. (2019) on SST-2Socher et al. (2013) and VIT-B/16 on the previously mentioned datasets (Table 4 this setting, the Taylor strategy shows some benchmarks and OOD splits (ImageNet- 1k), while occasionally yielding mild or even favorable effects others (CIFARI 100). By contrast, AvRCAvG remains consistently stable and overall stronger, providing additional empirical evidence in favor of Proposition]
mixed behavior: it causes clear larger degradation
B.2 STUDYING THE PRUNED HEADS AND UNCERTAINTY QUANTIFICATION
In this section propose study the effect of pruned heads uncertainty quantification. For example, we aim to understand how using Wang et al. (2025) pruning changes and improve the internal representation of OOD. All results are obtained using VIT-B/16 model..
18

---

Published as a conference paper at ICLR 2026
Table 4: Zero-shot Results across Models and Datasets. While TAYLOR pruning shows in-
consistent Sehavior-someting degrading OOD performance and other times mildly improving CIRCAvG remains more stable across settings.
Near-OOD Avg FPR951,
Far-OOD Avg
Method
ID Dataset
Model
Acct
ECEJ
AUROCT
AUPRT
AUROCT
FPR95. AUPR
SINGLE
ImageNet-1k
VFT-B/16
80.67
0.01
77.96
63.08
55.29
90.84
37.42
66.53
TAYLOR
ImageNet-IK Ik
VIT-B/16
66.46
0.01
68.14
77.88
42.87
85.96
48.29
50.02
CIRCAVG
ImageNet- Ik
VIT-B/16
73.61
0.01
73.81
70.51
50.
89.90
41.16
63.88
SINGLE
CIFAR 100
VIT-B/16 92.15
0.007
89.63
37.96
89.71
84,41
40.85
94,51
TAYLOR
CIFAR100
VIT-B/16 86.52
0.01
86.52
46.24
86.35
86.93
41.27
94.66
CIRCAVG
CIFAR100
ViT-B/16 91.15
0.01
87.39
44.16
87.39
86.64
37.78
95.01
SINGLE
SST-2
bert base 92.55
0.05
70.16
70.62
81.93
70.16
70.62
81.93
TAYLOR
SST-2
base 90.6
0.06
58.05
88.19
99.03
73.18
64.15
73.42
CIRCAVG
SST-2
berl base 91.74
0.02
71.95
66.57
99.40
79.07
55.57
78.64
How does pruning affect model's internal representation? To answer this question we use the dataset CIFAR-100 (Krizhevsky 2009) and its OOD benchmark (Yang et al. 2022). To study how attention head pruning impacts internal representations, we extract the CLS token from each head in the last layer. This token is treated as vector capturing the head's 's contribution to the overall representation. For each head, we compute the centroid of ID and OOD vectors. The separation between these centroids is then measured using Euclidean and Mahalanobis distances.
Let the output of the last attention layer's attention be:
Z E RN NxTxHxxd
where is the number of samples, 7 the length the sequence, H is the number of attention heads, and de is the head dimension. Here, Zn,h E RR is the CLS token vector for sample at head h.
For a set of ID samples Xip and OOD samples ROOD. the centroid for head is:
C
ID
1
Σ Zu,hi
Cp
Σ
Zu.h-
KID
KOOD
nEAiD
n
The Euclidean distance between and OOD centroids for head is:
cl
.00D C₂
h
2
Let En be the covariance of the ID representations for head h:
Σ>
Cov(iZm, n E NiD
Then the Mahalanobis distance is:
Mah V (c//
COOD ( 1 - Cn TOD)T.
Finally, we average over all heads:
H
H
d Tucl
Fad
Mah
Mah
H
H
h=1
We report the results for CircOOD defined in D.1.2 and the dense base model in Table 5 which shows how across heads and datasets, CireOOD consistently improves separation, achieving an average Euclidean distance of 2.26 and Mahalanobis distance of 3.32, compared to 1.75 and 3.03 for the dense model.
Additionally, in Figure/ dataset samples to visualize the internal representations of the six most and least important heads-corrrespontee corresponding respectively the last and first heads pruned during CireOOD extraction. The plot clearly shows that the most important heads produce a larger separation between ID and be detrimental.
apply PCA batch of the MNIST (Lecun et al. 2002
OOD data, whereas the least important heads
19

---

Published as a conference paper at ICLR 2026
Table 5: Disaggregated Euclidean and Mahalanobis distances between CLS token centroids of ID and OOD samples across datasets. The green values indicate the improvement (Circuit Dense).
CircuitOOD
Dense Model
Dataset
Euclidean t
Mahalanobis
Euclidean
Mahalanobis
Texture
1.66 (+0.34)
2.59 (+0.23)
1.32
2.36
MNIST
4.92 (+1.23)
7.03 (+0.83)
3.69
6.20
SVHN
2.10 (40.37)
3.21 (+0.03)
1.73
3.18
Places365
1.37 (+0.33)
1.98 (+0.21)
1.04
1.77
CIFAR-10
1.25 (40.29)
1.77 (+0.12)
0,96
1.65
Average
2.26 (+0.51)
3.32 (+0.29)
1.75
3.03
Figure 4: Internal representation of the six most important (left) and least important (right) attention heads for OOD and ID data using PCA. The representation on the left shows a clearer separation between the two distributions, while on the right it illustrates how the distributions sometimes even collapse onto each other.
Table 6: Comparison between circuits ID classification and OOD detection ImageNet- 1k with ViT-B/16. Each circuit shows superior performance on its respective task.
Near-OOD FPR951,
Far-OOD FPR95)
0OD Avg (all OOD)
Method
Heads
Acet
AUROCI
AUPR
AUROC1
AUPR
AUROCT
FPR95),
AUPR
CIRCOOD
8
71.11
74.26
70.36
50.75
89.52
40.11
61.77
81.89
55.24
56.26
CIRCAce
73.41
73.46
44.11
$7.69
53.60
2. Why does Circuit outperform Taylor pruning? Table shows that, in training- free setting, CireAvg consistently outperforms Taylor and in some cases even surpasses the dense model on OOD performance. To illustrate how different the two resulting sub-networks are, we examine the sets of heads they select and find that they overlap by only 25%. One might suspect this low overlap is simply due to Taylor pruning considering importance scores within each layer. To verify that this is not the case, we apply Taylor pruning globally across all layers. The overlap increases only slightly to 29%, and this global view also allows us estimate the difference between the two sub- networks: the Jensen Shannon divergence between the discrete distributions of selected heads is 0.37, highlighting the substantial distinction between them.
3. Does the specific Circuit excel at its task? As described in Appendix D.1.2 different circuits, each optimized for a different criterion. To validate this, we evaluate individually CircOOD and CircAce ImageNet- 1K (Deng al. 2009] and its OOD set (Yang al. (022). The results in Table 6 confirm the expected behavior: CireAce achieves higher classification accuracy (+2.30%), while CircOOD provides stronger OOD detection, yielding improvements of +1.09% in AUROC, 2.45% in FPR95, and +2.66% in AUPR.
extract three
B.3 EFFECT OF THE NUMBER OF HEADS
We conduct an ablation study on the number attention heads removed per block (2, 4, 6, 8) and compare Taylor and CircAvg in both the zero-shot and the fine-tuning setup. We report Top-1 accuracy (ID) and OOD metrics (AUROC, FPR95, AUPR) with ViT-B/16 on ImageNet-IK (Deng
al. 2009
20

---

Published as a conference paper at ICLR 2026
AURDC dalawei)
FPROG
000
0.75
04
0.05
0.4
02
Figure 5: Effect of pruning different number of heads count using Taylor and CircAvg for ViT-B/16 on ImageNet-1K IK (Zero shot)
kp-1 Accenusy (1)
000 AJRDC senooeaa
000 F1R25 suroo colan
000 AUFI, lang scuso daizeds)
0.53
3.00
0.96
3322
0.51
14055 mruned
(1add 1ed
1 Lindd
Figure 6: Effect of pruning different number of heads count using Taylor and CireAvg for ViT-B/16 on ImageNet- 1K (Post training)
Zero-shot (no training). Figure shows that pruning degrades all metries the number of heads pruned increases, with FPR95 rising rapidly. The circuit approach, however, degrades more slowly than Taylor across accuracy, AUROC, and AUPR.
After training. Fine-tuning each pruned model largely recovers performance displayed in Figure 6
Top-1 accuracy stays close to the base dense model (T): Taylor pruning the best at preserving accuracy and CircAvg pruning gives the best OOD performance across all metrics.
B.4 COMPUTATIONAL COST OF HYDRA ENSEMBLES
We now compare the parameter cost of Hydra Ensembles to that of a standard Transformer. For clarity, we focus on a single Transformer block. which is composed of multi-head self-attentin (MHSA) module followed by a Multi-Layer Perceptron (MLP).
Standard Transformer. A Transformer with hidden dimension d and H attention heads splits the hidden dimension evenly across heads:
d d2 = d, H
Each MHSA layer consists of four projection matrices:
Query matrix WQ E Rdxd
Key matrix W K E Rdxd
Value matrix W E IRdxd
Output projection W E Bddd
Thus, the attention block has pstandard 4d parameters.
MHSA
The MLP has two linear layers:
W dd
W2 Rddi
where typically dr 4d. Hence: petandard 2d dn 8ef2
Therefore, full Transformer layer has: pstandard
Player
MHSA
Detandard 12d2
21

---

Published as a conference paper at ICLR 2026
Hydra Ensembles. model m keeps H( S H active heads. For simplicity, assume all models keep the same number of active heads Hgg
We now consider Hydra Ensembles with M 3 models. At each layer L,
Each active head still projects into dy = d/H. Therefore, the dimension the concatenated head representation after pruning is
HI H
= Hyd
d.
Each MHSA layer of Hydra Ensembles consists of four projection matrices:
Query matrix W E Rdd
Key matrix W Rdxxdd;
Value matrix WV E TRd
Output projection W Mddxd
Thus, attention block has PP&M 4dMd,
Multi-Layer Perceptron In Hydra Ensembles, the Multi-Layer Perceptron parameters are shared across models (via averaging and grouping). Hence, the MI.P cost remains:
AaydraEEsennnn MLP
Sdf.
Total per layer. The total parameter cost of Hydra Ensembles is:
He MHSA + PP&M 4M de + 8d..
PP&M
Comparison. Thus, the cost of Hydra Ensembles depends linearly on the pruning ratio MH. If pruning is strong (i.e. H₁ H), the Hydra Ensembles layer can have fewer parameters than standard Transformer layer, even when aggregating three models.
H
In terms of memory cost, have the following bound
He dl + SiP < H
pstandard 12d
P.P. Hydraanneemmen = 4. layer
DeeppEns. 12M layer
Inference Cost Analysis. We compare all methods ViT-B/16 and ImageNet-Ik (Deng al., 2009). keeping on average 8 heads per layer, under both float32 and bfloat16 16 precision. Table reports runtime for the entire test set well per-batch inference time.
bfr 16, we observe that Hydra Ensembles achieves an inference cost that is nearly identical the single model baseline: the ratio Hydra Ensembles/ /Single only 1.07x for both whole-test runtime and per-batch inference. In contrast, Deep Ensembles are almost three times slower than single model under the same setting (= 2.99x).
For f1oat32, Hydra Ensembles incurs higher overhead compared to the single model (= 2.66x), but still remains substantially faster than Deep Ensembles. which require roughly three times the inference cost of single model ( 3.02x). This increase in ratio for float 32 comes from the higher memory bandwidth and arithmetic cost of full-precision operations, which seale less favorably when multiple pruned members executed jointly. In contrast, bfloatle com- putations benefit from specialized GPU hardware (Tensor Cores) combined with FlashAttention significantly accelerates matrix multiplications and reduces memory transfer, making Hydra En- sembles almost as efficient as a single model with no impact on performance.
Overall, these results highlight that approach offers inference efficiency close single model the bf foat16 5 regime, while providing a favorable trade-off between performance and compu- tational cost compared traditional ensembles.
22

---

Published as a conference paper at ICLR 2026
Table 7: Comparison of inference cost between single model, Deep Ensembles, and Hydra Ensembles. We report runtime for full test set and per-batch inference under BF16 and BF32, batch size 4, along with parameter count and multiply-ade operations.
BFP16
FP32
Model Size
Method
Full Test (s)
Per Batch (ms)
Full Test (s)
Per Batch (ms)
Params (M)
Mult-Adds (G)
SINGLE MODEL
23.07
6.15
31.27
8.34
86.57
17.58
TAYLOR
13.18
3,51
28.02
7.47
77.13
15.48
CIRCAVG
13.20
3.52
28.47
7.59
77.13
15.48
DEEP ENSEMBLES
69.06
18.42
94.36
25.16
259.7
52.74
HYDRA ENSEMBLES
24.55
6.55
82.37
21.97
116.31
46.19
Table 8: Effect of keeping different diversity sources.v means the factor is varied across members, means it is kept fixed.
Diversity kept
Classification
Calibration
OOD Average
Variant
Pruning Seed Batch Backprop
ACC1
Brier],
NLL
ECE,
aECE)
AUROCT
FPR95,
AUPR
SINGLE
80.67
0.27
0.71
0.01
0.01
84.40
50.25
60.91
CIRCAVG
80.11
0.30
0.88
0.10
0.10
85.43
50.79
61.60
HYDRA-ER ENS (CIRC)
x
80.80
0.29
0.86
0.12
0.12
86.02
47.94
62.33
HYDRA-ENS (CIRC
80.13
0.30
0.88
0.10
46,93
62.76
HYDRA (CIRC)
50.55
(CIRC)
0.10
50.49
HYDRA-ENS (CIRC)
48.26
62.53
HYDRA-ENS (CIRC)
80.12
0.30
0.88
0.10
85.31
50.67
61.27
B.5 STUDYING THE SOURCES OF DIVERSITY
To understand which factors contribute most significantly to the diversity and robustness of Hydra Ensembles, we systematically ablate different sources of randomness during pruning. In this case, we assume only one circuit to be available (CircArg) and to build Hydra Ensembles we sample 48 heads to prune three times of 100. Table summarizes the impact of varying the pruning seed, batch order, and backpropagation stochasticity on both in-distribution and -istr---isttituttonn per- formance. The results indicate that accuracy improves significantly only when the pruning seed is varied. Introducing pruning-sesn seed diversity yields the most consistent OOD gains: AUROC increases approximately 86 (from 885.3), and FPR95 decreases the high 40s (from ~50.7). Batch-order variability primarily reduces FPR95 (with a modest improvement in AUPR) but produces smaller and less consistent AUROC gains, whereas backpropagation stochasticity alone has limited impact. Combining these factors does not systematically surpass the seed-driven setting. We therefore con- clude that the primary source of beneficial diversity the pruning seed.
In addition to accuracy and OOD robustness, directly measure ensemble diversity using mutual information (MI) and disagreement index (DI) on both ID and OOD data (Table D These met- rics confirm the same conclusion: varying the pruning seed is the only factor that reliably creates non-trivial diversity (ID MI/ID DI zs 0.05/0. 11 and OOD MI/OOD DI 0.12/0.48), reaching lev- els close standard Deep Ensembles. In contrast, changing only the batch order or only the backpropagation seed yields near-zero MI and DI, meaning the resulting members behave almost identically. Even when batch order and backpropagation randomness are combined, diversity re- mains negligible unless the pruning seed is also varied. This strengthens our conclusion that the beneficial diversity Hydra Ensembles is fundamentally driven by the pruning seed, i.c., by the structural differences induced during pruning.
B.6 STUDYING THE COST/BENEFIT OF HAVING MORE MEMBERS
To study how the number of members affects performance, we compare a Single model, a 3-member Deep Ensembles, and Hydra Ensembles with and 5 members (Table I0). With members, Hydra Ensembles keeps inference time close to the Single model (24.55 vs. 23.07 s for the full test) while matching slightly improving OOD performance over Deep Ensembles. Increasing Hydra
23

---

Published as a conference paper at ICLR 2026
Table Diversity ablations for Hydra Ensembles. means the factor is varied across members,
means it is kept fixed.
Sources of diversity batch order
Diversity metrics
Method
pruning seed
backprop.
IDMI
ID
OOD
OOD
Single
CircAvg
Deep ensembles
0.06
0.13
0.23
0.58
Hydra Ensembles (Cire)
0.05
0.11
0.12
0.48
Hydra Ensembles (Cire)
0.00
0.02
0.00
0.13
Hydra Ensembles (Circ)
0.00
0.00
0.00
0.02
Hydra Ensembles (Cire)
0.00
0.02
0.00
0.13
Hydra Ensembles (Circ)
0.04
0.11
0.12
0.48
Hydra Ensembles (Cire)
0.00
0.00
0.00
0.00
Table 10: Ablation the number of members for Hydra Ensembles (float16 inference)
Float 16 Speed
Classifiction Brier
Calibration aECE
OOD Average
Method
whole test (s)
1 batch (ms)
ACC
NLL
ECE
AUROC
FPR95
AUPR
Single
23,07
80.67
0.27
0.71
0.01
0.01
84.40
50.25
60.91
Deep Ensembles (3M)
69.06
18.42
82.19
0.25
0.65
0.01
0.01
85.48
46.93
62.76
Hydra Ensembles (3M)
24.55
6.55
80.88
0.27
0.74
0.01
0.01
86.29
47.62
63.15
Hydra Ensembles (SM)
11.19
81.20
0.27
0.73
0.01
0.01
86.33
47.85
63.20
Ensembles to 5 members further improves accuracy and OOD metrics at the cost of higher, but still substantially lower, latency than 3-member Deep Ensembles.
B.7 MLP FUSION AND DIVERSITY
We expand here why fusing the MLP sub-blocks does not degrade ensemble diversity. In Hydra Ensembles, diversity mainly comes from the attention blocks being different across members: each Hydra Ensembles member is trained separately as a pruned subnetwork with its own of active heads. Because these attention structures are not the same, the members learn different internal features and make different predictions. We fuse only the MLP sub-blocks, and we do after the separate training is finished, so this core source of difference between members stays intact.
This claim supported empirically in Table II The fused and non-fused Hydra Ensembles are essentially indistinguishable on both in-distribution and OOD criteria: they match calibration (ECE/AECE 0.01), attain the same NLL (0.74), and yield nearly identical OOD detection (AU- ROC 86.29 96.26. FPR@95 47.62 47.98, AUPR 63.15 vs. 63.26). If MLP fusion were reducing diversity, we would expect a noticeable degradation in calibration or uncertainty based OOD metrics due to increasingly correlated member outputs. Instead, OOD performance is pre- served (and slightly improved relative to non fused model), indicating that member disagreement and ensemble diversity remains intact despite MLP fusion.
C DETAILS ON BASELINE IMPLEMENTATIONS
In these section we provide all details regarding baselines implementation and data used for each of the experimental sections: supervised image classification, zero-shot image classification and text classification.
C.I SUPERVISED IMAGE CLASSIFICATION
We adopt a two-stage training procedure for Vision Transformer (ViT) following (Dosovitskiy al. 2020). Stage pre-trains a ViT-B/16 on ImageNet-21k (Ridnik et al., (021): Stage 2 fine-tunes on the target dataset (ImageNet-1k (Deng al. 2009) CIFAR-100 (Krizhessyy 2009
24

---

Published as a conference paper at ICLR 2026
Table 11: Effect of MLP fusion on Hydra Ensembles, Fusing MLP sub-blocks preserves both in- distribution performance and OOD detection, indicating no loss of ensemble diversity.
Classification Brier
Calibration
OOD Average
Method
ACC
NLL
ECE
aECE +
AUROC
FPR@95
AUPR
Single
80.67
0.27
0.71
0.01
0.01
84,40
50.25
60.91
Hydra Ensembles (fused)
80.88
0.74
0.01
86.29
47.62
63.15
Hydra Ensembles (non-fused)
81.00
0.27
0.01
86.26
63.26
C.I.1 STAGE 1: PRE-TRAINING ON IMAGENET-211
We train ViT-B/16 from scratch on ImageNet-2Ik (Ridnik et al. 2021) (Winter 2021; 13,153,500
images, 19,167 classes). Each input undergoes the following data augmentation:
Random resized crop to 224 224 with scale sampled uniformly from (0.08, 1.0].
Random horizontal flip with probability 0.5,
Conversion to tensor,
Channel-wise normalization.
We use AdamW (Loshchilov & Hutter 2017) as an optimizer with the following parameters
Mmax 10-3, dropout 0.1, À 0.03, 3 (0.9, 0.999).
The learning rate follows a linear warm-up for the first 10,000 steps, then decays linearly to zero. Pre-training runs for 90 epochs.
C.1.2 STAGE 2: FINE-TUNINGG ON IMAGENET-
We load the pre-trained weights resulting from stage 1, and reinitialize the classifier part N=1000 classes. During the training, we reuse the pre-training data augmentation pipeline. For evaluation, images resized to 256 x 256, center-cropped to 224 x 224, and normalized. The official validation set is partitioned into a small validation subset (1%) and a larger held-out test subset (99%) to monitor convergence.
Fine-tuning uses SGD (momentum 0.9), with no weight decay and no dropout. The best learning rate is selected from {0.003, 0.01, 0.03, 0.063. We apply linear warm-up over 500 steps, followed by cosine decay over 20,000 steps, and stop at validation convergence or when the step budget is reached.
C.1.3 FINE-TUNING ON CIFAR-100
We load the pre-trained weights ImageNet-21k (Ridnik al. 2021 and reinitialize the classifier to ==100 classes. The training split reuses the pre-training data augmentation pipeline and test images resized to 224 x 224 and normalized;
use the official train/val/test splits.
Fine-tuning again uses SGD (momentum 0.9), no weight decay, and no dropout. The best learning rate is chosen from {0.001, 0.003, 0.01, 0.03), with linear warm-up over 500 steps and cosine decay over 10,000 steps, stopping on validation convergence or at the step budget.
C.1.4 PACKED ENSEMBLES, BATCH ENSEMBLE, MIMO AND LORA ENSEMBLE
For both Packed-Ennn Ensembles [aurrett et al. 2023 and MIMO (Havasi al. 2020). we use the same two-stage protocol as above: train each baseline from scratch on ImageNet 21k (Ridnik et al. 2021 then fine-tune on the target dataset (ImageNet- 1k (Russakovsky et al., 2015) or CIFAR-
Krizhevsky 2009)). For MIMO (Havasi et al. 2020), we set the number of estimators to E=3, use p=0.5, and bat repreeat= 4. For packed ensembles, we use E=3 and a=2. For Batch Ensembles Wen et al. 22020) we apply it to the trained checkpoint on ImageNet-21k (Ridnik al.)
2021] due to high cost of this pretraining step, we later fine-tune this model on either ImageNet- Ik (Russakovsky et al., 2015) or CIFAR-100 100 (Krizhevsky) 2009) For Lora Ensemle (Mühlematter ctt al. 2024) (Wang et al. 2023) we attach lora modules to the attention mechanism and fine-tune the model 3 different times with different seeds, r=4 and alpha=8.
25

---

Published as a conference paper at ICLR 2026
C.1.5 IMAGE OOD EVALUATION DETAILS
(OOD) evaluation, we follow the OpenOOD benchmark Yang et splits for both datasets, which define Near-OOD and Far-OOD scenarios. For ImageNet- K. the OOD datasets include SSB (Vaze et al. (021) OpenImage-O (Wang et al. 2022), Ninco (Bitterwolf) et al. 2023). iNaturalist (Huang & Li. 2021 ) and Texture (Kylberg 201 1); for CIFAR-100, they are CIFAR- 10, TinyIman ImageNet (Torralba et al. 20088), Texture, MNIST (Leeun et al. 2002), SVHN (Net-) zer et al. 2 and Places365 (Zhou et al. 2017). We note that we exclude Openlmage-O for ImageNet- IK and TinyImageNet for CIFAR- to ensure an evaluation that is as fair as possible, since their validation sets are used for circuit extraction.
For
p
C.2 SUPERVISED TEXT CLASSIFICATION
(Devlin et al. 2019) gingFace checkpoint on SST-2 (Socher et al. 2013). Since there is no official test set, we use the validation set for testing, while setting aside part of the training set for validation.
We fine-tune peert-base-uncan
classifier initialized from the Hug-
Tokenization We use the bert and padding to max length. A deterministic split is applied: the first 3,000 rows of the GLUE (Wang) et al. 2018) train split serve as validation; the remainder forms the training set. Evaluation is re- ported on the official GLUE validation split (872 labeled examples).
tokenizer with max max_ wax_lengthee 1engthh128.. truncation,
Optimizer and schedule. We use AdamW as optimizer with decoupled weight decay (weight.
0.01) and exclude bias and LayerNorm weights from decay. The learning rate is = 8 x 10-6. The schedule is per step: linear warm-up over 10% of the total training steps followed by linear decay to zero. Gradient clipping is applied with & = 1.0.
Early stopping and checkpoints. Training runs for up to 7 epochs with early stopping on validation accuracy (patience- 2, A = 5 x 10-1). We checkpoint every epoch and select the model with the best validation accuracy; final numbers are reported the held-out test split.
OOD evaluation. We consider two out-of-distribution (OOD) settings: (i) Near-OOD, where the task still sentiment analysis but the data comes from domains other than movie reviews. (ii) Far- 0OD, where the task is different from sentiment analysis, following recent NLP OOD protocols Liu ct al.
2023
Kim et al. (2023).
C.3 ZERO SHOT IMAGES CLASSIFICATION
The complete list of datasets used for evaluation is the following: ImageNet-IK. CIFAR- 100 and CIFAR-10, Food101 [Bossard et al. 2014), SUN397 (Xiao al. 2010). Oxford Pet Parkhi ct al. 2012). DTD (Cimpoi et al. 2014). EuroSAT (Helber et al. 2018 and Caltech101 (Fei-Fei al. 2004). For ImageNet- IK and CIFAR-100 100 we adopt the same evaluation splits in supervised image classification, and the same holds for out-of-distribution benchmarks. For SUN397 we use HuggingFace while
Datasets.
torchvis s sion for the other ones. These datasets cover broad spec- trum, ranging from large-scale benchmarks such as ImageNet- 1K to smaller fine-grained recognition tasks like Oxford Pet and 1011 They also include diverse domains such as textures (DTD) and satellite imagery (EuroSAT), ensuring a comprehensive evaluation across different levels of diffi- culty and granularity, just like standard prior CLIP-based studies (Zhou et al. 2022). Following Radford et al. (2021). we use "A photo of a {label}. as prompt template.
Architecture. For all our experiments we use the backbone CLIP-ViTIE ViT/B-32 model pretrained on LAION2B (Schuhmann et al. 2022). The specific model instance is la i on s34b b79k which made readily available along with the train and validation transform by the OpenCIIP reposi- tory Maarrco et al. 2021).
Baselines Implementation. Taylor pruning (Molchanov et al. 2019) is applied only on the vision encoder removing 24 heads for simplicity. We do so because the scale of the importance scores between the two encoders differs greatly as showed in Figure 7 thus making them not directly comparable. BayesVLM (Baumann et al.) 2024), training- free method that improves uncertainty estimation using a Laplace approximation over 327k samples of LAION-400M and captures uncer- tainties inherent to the model itself. For its evaluation we leverage the public GitHub repository of
26

---

Published as a conference paper at ICLR 2026
Layer
Layer
Layer
Taytor
aayer
Iaper
Layer
Layer
Sto00
01
3
11
11
11
Head Indcx Laver
Hoad Index Layer
1ead Inocx 11
Head Indox
Salor Scone
Index
Index
Figure 7: Taylor scores for text (8x12 heads) and vision (12x12 heads) encoders ordered by layer and head; in all layers, but the first three, the vision encoder has consistently higher scores.
the project where they also make available the hessian estimation. ViLU Lafon et al. (2025) instead adopts very different approach by introducing additional parameters train binary misclassifica- tion classifier to distinguish correct from incorrect predictions. The classifier can be used for OOD) detection by considering OOD samples misclassified ones. leverages frozen and vision fea- tures by creating an embedding composed of reweighted text embeddings for each class, the vision embedding, and the textual embedding of the predicted class. Unlike our method, ViLU requires training on each dataset using a weighted binary cross-entropy loss. Implementation follows the original GitHub repository settings, and we train on each dataset for a number of epochs that range between 100 and 300, which for ImageNet-11 k requires approximately 10 hours 8 A A100 GPUs. For Temperature Scaling (Guo et al., 2017) we use 5000 samples from the training set ImageNet- Ik following the setup used in Bayes LL. Lastly note that don't 't evaluate Bayes VLM and ViLU with Temperature Scaling because they are not compatible with it.
D DETAILS ON METHOD IMPLEMENTATIONS
D. PRUNING TECHNIQUES
D.1.1 TAYLOR PRUNING
Following Molchanov et al. (2019), for each Transformer block I, we score every attention head using first-order Taylor criterion computed small calibration loader. Let D be the embedding dimension, H the number of heads, and d, = D/H. Let (((") w:!) W!!
E DxD be the input- ac ac aw!
projection weights and ! (ll) ( E B Dx their gradients h (1,..., H). define its index R. {()- 1)dn 1,..., h. The final scores per
ac For head aw!!
27

---

Published as a conference paper at ICLR 2026
layer is computed follows:
(1)
Wil G(l) qime
4h
dp
rERA o=l
(1))
W G(ll Kire krre
dn D
rER c=1
(1)
W G vire
0/2 dn D
pine
rERA c=1
(1) 1
(1) 4h
(1) kn 1/
% 3
Given a budget Ti heads to prune per layer, we keep the top Hı heads by 81,6 among the layer heads and structurally rebuild the attention a wet.teeeetttteeeiinneee slicing Wg. Wa. W. (and WO) and their biases. After pruning layer, gradients are recomputed again, ensuring scores reflect the updated network. This guune-recomput step is applied sequen- tially across layers until the pruning budget is met across all layers.
module by
D.1.2 CIRCUIT EXTRACTION AND PRUNING
Inspired by Wang et al. 2025), we prune the attention heads of a transformer without relying on gradients (as in Taylor- based upproaches). Instead, we score each head by its contribution to target task (accuracy or OOD detection), which gives a ranking from most to least useful head. Given a head budget, then structurally prune the lowest-ranked heads, leaving small set that preserves the desired behavior of the model. We consider three behaviors throughout: (i) in-distribution (ID) accuracy, (ii) out-of-distribution (OOD) separability under MSP, and (iii) a balance of both. For each target behavior we define a validation score to optimize:
Sace Acc(f;; DID),
Sood AUROCMse (f: DID, DOOD). Savg = 2 (Sace + Sood),
where ff denotes the model with a proposed head temporarily turned off (i.c., ablated). Here, Sace targets ID accuracy, Sood targets OOD separability under MSP, and Savg trades off both. Depending on the criterion used for circuit extraction, we denote the final model obtained with Sace as CircAcc, with Soud as CircOOD, and with Sang as CircAvg. With S specified, we describe below how heads are selected and removed.
Greedy circuit extraction under a head budget. Given head budget B, extraction proceeds by turning heads off-i.e., ablating iie--indiiddually in a greedy loop and measuring the impact on S:
1. Initialize the candidate set R = h(),h) for all layers and heads}.
2. For each (1, R: temporarily turn that head off (ablate it), evaluate the model to compute S, then restore the head. Record S for that particular (1, h).
3. Identify the single best candidate ( hh) one whose removal achieves the highest S (i.e., currently the least useful head for the chosen objective) and permanently turn that head off (ablate it). Remove (I" /1*) from R.
4. Repeat steps 2-3 until B heads have been removed.
This procedure yields a ranking of attention heads across the whole model by their relevance to chosen objective, this ranking can then be used to create our pruned subnetwork- extracted circuit-optimized for the chosen score S (accuracy, OOD, or their average). Because each step measures the marginal effect of removing exactly one head, the process stable and easy to imple- ment across frameworks. Please refer to for the full algorithm.
Details on head scoring and ablation. For head {O.... .,.,, H1 - 11 we form
At block 1, let the post-layer-nomm input be X, E RR
0 8.W,00.
R(N))
v(th X.W((a
28

---

Published as a conference paper at ICLR 2026
compute attention weights
A softmax
1 Vdn
21 (K( E RTXT
and aggregate values
21mh A (1) v7(1)
E Txdd
All head outputs concatenated and passed through the shared output projection:
2 - ... 2 - E TRTD
MHA((X)) = Z,W? + b,
with w/ DxD Because concatenation assigns each head dedicated channel block, head corresponds contiguous slice of da columns in W/O so to "turn head off" we just need change the output projection WP, by partitioning WP into per-head column blocks Wo RDdx:
(.(N)
wP
[W(.0]]
|
|| W (1
The attention output decomposes as
H111 Zwo = Σ 1=0
( Wo.0)-
Setting the specific block W(( = removes exactly the contribution of particular head h, since the term Z vanishes while all other head contributions are unchanged. Tensor shapes are preserved: Zi remains T x D. the projection is still D -> D, and the residual update Y; = X; + (X))(()) is well-defined. Ablating via the W slice is reversible and numerically stable: it turns heads off one time without modifying the Q/K/V projections or fused attention kernels, which makes it easy to implement across different architectures.
L(Ak)
Ablation operator (non-structural), Let sn h dn and en (h+1) dy be the column range in 0 corresponding to head h.
Temporary ablation (for scoring only):
ABLATETEMP(1,1 h):
A., W99: 8n enl:
w 8h en] + 0; evaluate score S;
WP: sh en Arp.
Permanent ablation (to define the circuit):
ABLATEPERM(1. h): WP 8h : (no restore).
These ablation operators are non-sfructural in the sense that parameter tensors keep their original sizes (no FLOP reduction). This choice makes comparisons across heads and across scores consis- tent and easily reversible.
Given budget of heads prune, remove the first heads given by the circuit algorithm and structurally rebuild the attention as a Mult Hettiieettteen ionP module by keeping only the selected head blocks and slicing the per-head columns/rows of WQ, WKK WV and W (and their biases).
OOD sets and fairness. For circuit extraction on ImageNet- we use OpenImage-O as OOD; for CIFAR-100 100 circuits, we use Tiny-ImageNet; following OOD validation subsets set by OpenOOD. Although these OOD validation subsets splits differ from the ones used during the OpenOOD bench- mark we exclude them from our final comparison tables for more fairesss; they are used only to define the pruning score in the OOD/average circuits. For SST2 dataset we use the AG's News dataset (Zhang al, 2015) OOD validation and we do not include in final results.
29

---

Published as a conference paper at ICLR 2026
Algorithm Greedy circuit extraction under budget B
Require: model ID loader DID: OOD loader DOoD; score type {acc, ood, avg); budget B 1: R 4- [(1, 1)) for all layers/heads}
2: for 1. 4- to B do
3: best 4- 00, (1*, h²) +- None
4: for all (1, hi) R do
5: ABLATETEMP(L,H
6:
(acc, auroc) EVALUATE DID, DOOD)
acc,
8 acc
7:
S auroc,
ood
(acc + auroc),
avg
8:
RESTORE(L, h)
9: S best then best + S; (1", 1()) f h1
10:
ABLATEPERM(I", h"); R c R [[",,))}
D.2
POST-PRUNING FINE-TUNING PROTOCOL.
For each dataset, we fine-tune the pruned transformer using the same recipe, independent of the pruning technique. We freeze most weights, unfreeze targeted subset (attention modules, MLPs, LayerNorms, and the classifier head), and train with SGD using a per-step warm-up followed by cosine decay learning-rate floor. We report validation metrics during training and evaluate on the held-out test split at the end.
IMAGENET-LE 1K: FINE-TUNING AFTER PRUNING
Learning rate. We select 7max via small grid f0.003, 0.01, 0.03, 0.06} (best by validation) and use a per-step schedule: linear warm-up for 5,000 steps followed by cosine decay to a floor fraction fmin 10 5; the total number of steps is 120,000.
Train transforms. RandomResizedCrop to 224x224, RandomHorizooallFFip with probability 0.5, RandAugment (2 ops, magnitude 12) and ImageNet mean/std normalization.
Regularisation. Cross-entropy with label smoothing = 0.1, and dropout p = 0.1 inside each transformer layer.
CIFAR-100 FINE-TUNING AFTER PRUNING
Learning rate. We select /max via small grid (best by validation; e.g. (0.001, 0.003, 0.01, 0.03)) and use per-step schedule: linear warm-up for 500 steps followed by cosine decay to floor fraction fmin 10
10-5: 5; the total number of steps 10,000.
Train transforms. RandomResizedCrop to 224x224, RandomHorizoottlFFin with probability 0.5, and ImageNet mean/std normalization.
Regularisation. Cross-entrops with label smoothing e = 0.1.
Specific details for CIFAR-100. (1) Partial fine-tuning: only the last five Transformer blocks are updated (attention projections, LayerNorms, MLP linear layers); earlier blocks remain frozen. (2) The classifier head re-initialized to zeros before fine-tuning. Following the observation of Raghu et al. 200TT, we find that fine-tuning all layers small datasets such as CIFAR-100 100 tends to degrade performance, as the model quickly overfits and loses useful pretrained representations. To mitigate this issue, we fine-tune only the final classification layer when working with CIFAR-100.
GLUE/SST-2: FINE-TUNING BERT AFTER PRUNING
Learning rate. We set 77 1e-5 and use per-step linear warm-up 10% of the total steps followed by linear decay to zero. Optimization is Adam with decoupled weight decay; gradient clipping & 1.0.
30

---

Published as a conference paper at ICLR 2026
Table 12: Results of Hydra Ensembles on ImageNet-IK and CIFAR-100 across seeds
Classification
Calibration
OOD Average
Dataset
Method
Acc (1)
Brier (U)
NLL (U)
ECE (1)
aECE (1) AUROC (1)
FPR95 (L) AUPR (1)
SINGLE
80.67
0.27
0.71
0.01
0.01
84.40
50.25
60.91
ImageNet-1K
DEEP ENSEMBLES
82,19
0.25
0.65
0.01
0.01
85,48
46,93
62.76
HYDRA ENS (CiRC)
80.88
0.27
0.74
0.01 2000
0.01 =nre
86.31 =0:1
47.62.2001
63. 13:18s
SINGLE
92,15
0.11
0.25
0.01
0.01
85.46
40.27
93.55
CIFAR-100
DEEP ENSEMBLES
93.52
0.09
0.22
0.01
0.01
86.08
38.67
94.26
HYDRA ENS (CIRC)
92.05
:0.44
0.12
0.28 60000
0.07
0:00
0.00
89.72 0.23
35.78
95.25 4.006
Train details. bert-base-uncased tokenizer with max.length=128, truncation, and padding to max length. Deterministic split: the first K examples of GLUF//ST-2 train form validation (K=3000); the official va 1 i dation set is used for test.
Regularisation. Cross-entropy (no label smoothing); default BERT dropout is left unchanged.
D.3 ZERO SHOT IMAGES CLASSIFICATION
For CLIP-based methods, we prune 24 attention heads jointly from both encoders. When pruning both encoders, we rely exclusively on ImageNet-Il Ik (Deng et al., (209) and its associated OOD sets. The datasets are the same as those used in Section 5.1 ensuring a fair comparison.
Train details. When training CLIP, several design choices are possible. Following Zhai et al. 2022), we lock the image encoder during training. This strategy provides three key beneñit: (i) it improves performance compared to finetuning the vision encoder, (ii) it eliminates the need to merge MLPs across circuits, since the pruned vision encoders remain identical and differ only in their attention heads, and (iii) it simplifies training, as the text encoder contains less than half the parameters of the vision encoder. Training is carried out with the OpenCLIP repository (Thaacco et al. 202T). We train for epoch 60M samples from LAION-400M (Schuhmann et al. 2021), ). using learning rate equal 5e 5 selected from a small grid { [1e- - 4. 5e 5, 1e 5 on the train loss. We employ 100 warmup steps, batch size of 1 15500, and 4 gradient accumulation steps (yielding virtual batch size of 96,000). All other settings follow the OpenCIIIP defaults.
Hardware. Training one circuit model requires approximately one hour, for total of three hours to create the Hydra Ensembles, using 16 A100s GPUs.
E COMPLEMENTARY RESULTS
F. SUPERVISED IMAGE CLASSIFICATION
We report additional OOD results for ImageNet-IK: and CIFAR-100 Table 15 and Table 16 More detailed per-dataset results on ImageNet- K are provided in Tables 21 and 22 and per-dataset OOD results for CIFAR-100 appear in Table 23 (Note that CIFAR- 10 is the only near-OOD split for CIFAR-100, so its performance matches the average nea result already shown in Table [6]
Across both ImageNet- IK and CIFAR-100. Hydra Ensembles exhibits stable behavior over eight different seeds, see Table 12 The mean performance varies only minimally across runs, and the associated standard deviations remain small for all metries (accuracy, Brier, NI.L, calibration, and OOD scores). This indicates that Hydra Ensembles deliver statistically consistent results with no sensitivity to initialization or seed choice while maintaining competitive accuracy and strong OOD robustness.
We also evaluate all ImageNet- IK baselines under distribution shift using corrupted ImageNet-C Hendrycks & Dietterich (2019b) inputs five severity levels. Table 13 reports top-1 accuracy and Table reports the corresponding Brier scores (lower is better).Decy Ensembles obtain the best average accuracy across severities (2%), but Hydra Ensembles (Taylor) remain highly compet-
31

---

Published as a conference paper at ICLR 2026
Table 13: Results under distribution shift (ImageNet-C): top-1 accuracy (%). Columns sevl-sev5 correspond to corruption severities 1-5; Avg is the mean over severities.
Method
Acc_sev1
Acc_sev2
Acc_sev3
Acc_sev4
Acc_sev5
Average
Single
70.44
63.31
56.57
45.10
31.97
53.47
Deep Ensembles
73.11
66.59
60.43
49.50
35,97
57.12
Packed Ensembles
68.11
60.18
52.87
41.21
28.81
50.23
MIMO
71.49
64.88
58.78
47.56
34.40
55.42
Batch Ensemble
70.10
62.94
56.21
44.71
31.72
53.13
MC Dropout
70.44
63.64
57.39
46.38
33.22
54.21
LoRA Ensembles
70.27
63.09
56.34
44.88
31.79
53.27
OBA
69.12
62.49
56.40
47.06
35.24
54.06
Taylor
70.63
63.81
57.84
47.84
34.92
55.00
CircAvg
70.00
62.97
56.52
45.81
32.66
53.59
Hydra Ens (Taylor)
71.65
65.10
59.35
49.62
36.79
56.50
Hydra Ens (Cire)
70.81
63.87
57.62
46.96
33.73
54.59
Table 14: Results under distribution shift (ImageNet-C)) Brier score (lower is better). Columns sev1-sev5 correspond to corruption severities 1-5; Avg is the mean over severities.
Method
Brier
Brier sev2
sev3
Brier_sev4
Brier
Average
Single
0.40
0.48
0.56
0.67
0.80
0.58
Deep Ensembles
0.38
0.45
0,52
0.64
0.76
0.55
Packed Ensembles
0.43
0.52
0.60
0.71
0.83
0.61
MIMO
0.39
0.47
0.54
0.65
0.78
0.56
Batch Ensemble
0.41
0.49
0,56
0.68
0.80
0,58
MC Dropout
0.40
0.48
0.55
0.66
0.79
0.57
LoRA Ensembles
0.40
0.49
0.56
0.68
0.80
0.58
OBA
0.43
0.50
0.57
0.67
0.79
0,59
Taylor
0.42
0.49
0.56
0.66
0.79
0.58
CircAvg
0.43
0.50
0.57
0.68
0.80
0.59
Hydra Ens (Taylor)
0.41
0.48
0.55
0.64
0.76
0.56
Hydra Ens (Circ)
0.42
0.50
0.57
0.67
0.79
0.59
itive with an average of 56.5% while clearly improving performance at the strongest shifts: Hydra Ensembles (Taylor) achieves the best accuracy at severity 4 and (49.62% and 36.79% 36.79%), outper- forming Deep Ensembles in this regime. Hydra Ensembles (Taylor) also matches the calibration of Deep Ensembles, with very similar Brier scores and identical performance the highest severities. Compared to alternative efficient ensemble methods (Packed Ensembles, MIMO, Batch Ensemble, MC Dropout, LoRA Ensembles, OBA, Taylor, CireAvg). Hydra Ensembles consistently achieve higher accuracy and equal better Brier scores across severities, confirming that Hydra Ensem- bles 's structured diversity transfers to distribution-shiin robustness on ImageNet-
E.2 SUPERVISED TEXT CLASSIFICATION
Table 17 present more results for OOD performance of Bert (Devlin et al. 2019) on SST2 dataset
Socher et al., 2013). Overall, Hydra Ensembles (Circ) yields the strongest OOD detection: best NoD (AUROC 70.97, lowest FPR95 69.38) and best overall average (AUROC 77.60, lowest FPR95 55.06). CircAvg is the best single-pruned variant (overall AUROC 75.04, FPR95 56.2 56.22), while DeepEns-Downstrenn Downstream attains the top overall AUPR (84.90). See Table [8] for more detailed Far-OOD results.
results Near-OOD performance and tables 20
32

---

Published as a conference paper at ICLR 2026
Table 15: ImageNet-IK (ViT-B/16): OOD detection on OpenOOD near/far splits (MSP).
Near-OOD FPR95L
Far-OOD Avg
Avg
Method
Heads
AUROCT
AUPRT
AUROCT
FPR95]
AUPRT
AUROCT
FPRYSL
AUPR
SINGLE
12
66.53
DEEP ENSEMBLES
12
78.77
61.70
56.15
92.20
32.17
69,38
85.48
46,93
62.76
PACKED ENSEMBLE
12
76.30
65.59
$2.60
90.21
37.70
63.74
83.26
51.65
58.17
MIMO
12
78.06
62.85
55.67
89.21
42.44
62.61
83.63
52.64
59.14
LORA-SINGLE
12
77.92
38.01
65,
84.24
BATCH ENSEMBLE
12
77.93
63.20
55.08
90.54
37.99
65.61
84.24
50.59
60.35
DROPOUT
12
77.13
64.46
53.75
90.27
38.42
64.06
83.7
51,44
OBA
12
76.81
67.43
42.37
57.09
82.61
$4.89
TAYLOR
79.83
65.10
59.22
88.93
43.92
59.71
84.38
$4.51
39.46
CIRCAVG
81.00
62.00
61.33
90.43
38.46
63.49
85.71
50.23
62.41
HYDRA-ENS (TAYLOR)
8x3
80.53
62.31
59.85
90.18
38.70
61.64
85.36
50.5
60:75
HYDRA-ENS (CIRC)
8x3
$1.39
60.10
61.45
35.14
64.85
86.29
47.62
63.15
Table 16: CIFAR-100 100 (ViT-B/16): OOD detection OpenOOD near/far splits (MSP).
Near-OOD (CIFAR-10)
Far-OOD Avg FPR95]
OOD OOD)
Method
Heads
AUROC!
FPR95L
AUPR
AUROC:
AUPR
AUROCI
FPR95]
AUPR
89.63
84.41
94.51
DEEP ENSUMBLES
12
90.88
34.73
91.09
84.88
39.65
95.06
86.08
38.67
94.26
LORA-SINGLE
12
39.18
39
40.24
94.57
LORA- ENS
12
89.47
38.56
89.52
84.85
40.58
94.53
85.77
40.18
93.53
MIMO
12
91.58
34.67
92 24
87.21
37.58
95.87
$8.08
37.00
95.15
BATCH
MC DROPOUT
12
88. 88
41.03
88.97
83.71
42,91
94.16
84.74
42.53
93.12
OBA
88,
40.53
88.72
85,10 10
40.47
94,58
85.85
40.48
93.41
TAYLOR
33.33
91.98
12
95.79
CIRCAYG
10
91.47
34.33
91.91
89,34
38.65
96.04
89.77
37.79
95.21
HYDRA-ENS (TAYLOR)
10x3
91.96
33:11
88: 13
41:19
96.20
39.57
HYDRA-ENS (CIRC)
10x3
91.62
32.92
91.84
88.89
37.32
96.00
89.43
36.44
95.17
Table 17: SST-2 (BERT-base): OOD detection (near/far) with MSP
Near-OOD Avg
Far-OOD Avg
OOD
Avg (all OOD)
Method
AUROC!
[PR95]
AUROC1
[PR95.
AUPR
AUROC1
[PR95]
SINGLE
12
59.45
88.13
99.08
74.45
63.62
75.07
70.16
70.62
81.93
DiEPENS-DOWSSTRER DOWNSTREAM
61.42
99.16
80.16
53.41
79.20
74.81
62.69
84.9
MC DROPOUT
12
62.31
87.21
99.19
76.20
59.42
75.07
72.23
67.36
81.96
LORA SINGLE
LORA-ENS
12
59.83
86.12
99.08
75.23
61.74
75.55
70.83
68.7
82.27
TAYLOR
58.64
90.54
99.07
75.88
59.79
75.32
70.95
68.57
82.11
CIRCAYG
29.76
78.19
56.22
HYDRA-ENS (TAYLOR)
6x.3
60.14
80.35
99.08
76.54
$5.50
75.79
71.85
62.59
82.44
HYDRA-ENS (CIRC)
69.38
80.25
49.33
77.6
55.06
84.16
Table 18: SST-2 (BERT-base): OOD detection on near-OOD splits (Yelp Polarity, Amazon Polarity) with MSP.
Yelp Polarity FPR95
Amazon Polarity
Near-OOD Avg
Method
AUROC1
AUPRT
AUROCT
FPR95
AUPRT
AUROC1
FPR95
AUPRt
SINGLE
86.47
56.47
DEEP ENSEMBLES
63.82
84.17
98.49
59.03
87.61
99.83
61.42
85.89
99.16
LORA-SINGLE
62.61
85.21
98.3
$7.03
87.61
99.81
59.82
86.41
99.08
LORA-ENSEMBLE
84,98
57.07
87.27
99.81
59.83
86.
99.08
MC DROPOUT
65.04
82.12
98.55
59.58
88.30
99.83
62.31
87.21
99.19
TAYLOR CIRCAVG
90.25
56.79
58
99.00 09.07
66.26
70.41
98.50
60.27
80.05
99.83
63.26
75.23
99.16
Hydra Ensembles (Taylor)
79.01
58.18
99.08
Hydra Ensembles (Circ)
73.84
65.60
98.87
68.10
73.17
99.87
70.97
69.38
99.37
33

---

Published as a conference paper at ICLR 2026
Table 19: SST-2 (BERT-base): Far-OOD (20NG, TREC, MNLI) with MSP.
20NG FPR95
Method
AUROCT
AUPRT
AUROCT
FPR95]
AUPRT
AUROCT
FPR95
AUPRI
SINGLE
74.08
59.40
94.61
65.45
60.55
42.68
73.11
77.29
95.98
DEEP ENSEMBLES
80.08
46.56
95.98
78.21
40.37
56.48
77.28
72.02
96.70
LORA -INGLE
58.37
44.40
96.04
LORA-ENSEBBLE
74.81
59.40
94.73
68.04
56.19
44.65
73.59
74.66
96.06
MC DROPOUT
78.68
50.11
95.47
64.57
57.57
41.38
76.14
73.85
96.46
TAYLOR
77.66
49.20
95.22
70:47
52.64
47.09
73.71
68.92
96.
CIRCAVG
77.97
48.51
95.39
74.81
45.18
51.26
78.25
56.65
96.74
Hydra Ensembles (Taylor)
77.06
47.13
95.17
72.89
50.92
50.44
74.43
66.63
96.13
Hydra Ensembles (Circ)
83.69
37.50
96.51
71.99
55.16
49.69
78.07
61.58
96.69
Table 20: SST-2 (BERT-base): Far-OOD (RTE, WMT16) with MSP.
RTE FPR95.
WMT16 FPR95
Method
AUROCT
AUPR
AUROCT
AUPR
SINGLE
84.36
51.26
57.56
75.28
69.61
84.52
DEEP ENSEMBLES
45.18
60.45
78.53
62.96
86.39
BORGLEINGLE
84.35
50.57
58.18
75.22
67.55
84,25
LORA-ENTE A-ENSEMBLE
84.41
50.46
57.96
75.30
68.00
84.38
MC DROPOUT
86.07
44.15
57.87
75.54
71.44
84,20
TAYLOR
82.20
61.01
53.65
75.39
67.20
84,54
CIRCAVO AVG
87.68
40.14
60.80
80.09
52.64
86.78
Hydra Ensembles (Taylor)
82.53
50.46
52.96
75.81
62.39
84.26
Hydra Ensembles (Circ)
88.38
32.91
61.20
79.13
59.52
86.30
E.3 ZERO SHOT IMAGES CLASSIFICATION
In this section we present disaggregated results for both classification and OOD detection relative to Table 3 and the results with pruning applied only the vision encoder in the zero-shot and fine-tuning setup.
E.3.1 EXTENSIVE RESULTS
Classification. Results are reported in Table 24 Hydra Ensembles achieves accuracy comparable to, or better than, both BayesVLM and the single model, with the only exception Food101, where it improves performance by small but significant margin (+0.35%) despite being training- method. The gains are even more pronounced in terms of calibration: Hydra Ensembles reduces ECE by 5.19% relative to the single model, 2.59% relative to Bayes and 0.54% compared to Temperature Scaling. In contrast, ViLU does not consistently improve calibration, and even disregard its calibration its average ECE remains worse than the one achieved by our method (4.17% vs. 3,29%). Hydra Ensembles also improves NLL compared to both BayesVLM and the single model, although falls behind Temperature Scaling by a small margin. Nevertheless, unlike ViLU and BayesVLM, Hydra Ensembles is fully compatible with Temperature Scaling; when combined, the resulting NLL reaches 0.90, effectively closing the gap. Finally, CircuitAvg outperforms Taylor pruning across all metrics in a training-free setting, highlighting its robustness and practical advantage.
ImageNet- Ik (where it achieves the worst ECE across datasets),
OOD Detection. Results are reported in Table 25 for ImageNet- and Table 26 for CIFAR 100. ImageNet- 1k, Hydra Ensembles consistently outperform all baselines on both Near-OOD and Far-OOD detection. The closest competitor is ViLU, but out method improves upon it by notable margins in AUROC (+1.44%), FPR95 (-3.53%), and AUPR (+4.04%).
On CIFAR100 the picture is more mixed. For Near-OOD, Hydra Ensembles improves AUROC (+1.7% and FPR95 (-5. 17%), while ViLU and BayesVLM significantly underperform. On some Far-OOD datasets, such Texture and MNIST, Hydra falls short of the single model, with degra- dation comparable to ViLU and BayesVLM, whereas shows clear gains on SVHN and P1aces365. Overall, Hydra Ensembles achieves the best average performance among the training-free meth-
34

---

Published as a conference paper at ICLR 2026
Table 21: ImageNet-IK (ViT-B/16): OOD detection (Near-OOD splits: SSB-Hard, NINCO) with MSP.
SSB-Hard FPR95
NINCO FPR95
Near-OOD Avg
Method
AUROCT
AUPRT
AUROCT
AUPRT
AUROCT
FPR95,
ALPR1
SINGLE
72.44
74.76
71.76
43.49
51.40
38.82
77.96
55.29
DEEP ENSEMBLES
72.74
74.86
72.02
84. 84.80
48.54
40.28
78.77
61.70
56.15
PACKED ENSEMBLE
70.50
77.98
69.72
82.11
53.21
35.49
76.30
65.59
$2.60
MIMO
OBA
72.74
74.10
71.23
80.88
60.76
16
76.81
67.43
$2.69
LORA SINGLE
72.54
74.66
71.81
83.31
51.57
38.41
77.92
63.11
55.11
LORA-ENSIE EERSEEESEMBBE
72.56
74.60
71.80
83.31
38.37
77.93
63.20
55.08
BATCH ENSEMBLE
72.58
74.66
71.83
83.42
51.30
38.57
78.00
62.98
$5.20
MC DROPOUT
71.29
76.21
70.46
82.98
52.72
37.04
77.13
64.46
53.75
TAYLOR CIRCAVG
74.95
75.58
74.86
84.72
54.63
43.59
79.83
65.10
$9.22
76.05
73,46
76.42
85.95
50.54
46.20
81.00
62.00
61.33
Hydra Ensembles (Taylor)
75.39
74.02
75.22
85.68
50.60
44.49
80.53
62.31
$9.85
Hydra Ensembles (Circ)
76.25
72.94
76.57
86.53
47.26
46.33
81.39
60.10
61.45
Table 22: ImageNet-IK (ViT-B/16): OOD detection (Far-OOD splits: iNaturalist, Texture) with MSP.
INaturalist FPR95
Texture FPR95
Far-ODD Avg
Method
AUROCT
AUPRT
AUROCT
AUPRT
AUROC1
FPR95_
ALPRÉ
SINGLE
24.33
86.58
49.30
90.84
37.42
66.53
DEEP
96.02
19.28
86.18
88.39
45.06
52.58
92.20
32.17
69.38
PACKED ENSEMBLE
94.15
26.45
80.35
86. 86.28
48.96
47.13
90.21
37.70
63.74
MIMO
93.02
31.46
77.27
.1.41
53.42
47.95
89.21
42.44
62.61
OBA
90.51
36.26
68.00
86.34
48.49
46.18
88.42
42.37
$7.09
LORA-SINGLE
94.86
25.40
83.07
86.27
50.62
48.22
90.56
38.01
65.64
DORA-ENSEMBLE
25.52
82.92
86.27
48.31
37.99
65.61
BATCH ENSEMBLE
94.99
25.00
83.55
86.40
50.59
48.90
90.69
37.79
66.22
MC DROPOUT
94.60
25.46
81.76
85, .4
51.38
46.37
90.27
38.42
64.06
TAYLOR CIRCAYG
92.10
33.61
74.58
85.77
54.23
44.84
88.93
43.92
59.71
94.33
26.13
81.04
86.53
50.80
45.94
90.43
38.46
63.49
Hydra Ensembles (Taylor)
93.09
29.80
76.41
87.28
47.60
46.88
90.18
38.70
61. 61.64
Hydra Ensembles (Circ)
94.77
23.38
81.87
87.62
46.90
47.84
91.19
35.14
64.85
ods, offering significant improvements in FPR95 (-3.04%) and only mild reductions in AUROC and AUPR relative the single model.
Finally, although CircuitAvg attains the highest OOD scores, it suffers from poor ID classification, making it inferior overall.
E.3.2 EFFECT OF PRUNING ONLY THE VISION ENCODER AND FINE-TUNING
In this section, we present results from two additional settings.
First, we prune only the vision encoder align with the experiments Section 5.1 considering scenario focused on computational efficiency. Second, we evaluate the effect of fine-tuning in this setup.
For these experiments, circuits are extracted using CIFAR-100 with its corresponding OOD set, to- gether with ImageNet- 1k, due observed performance degradation. Consequently, results from vision-encoder-onle pruning not directly comparable to those obtained when pruning both en- coders, and we leave further investigation for future work.
Given the high cost of training CLIP and the strong performance of Hydra Ensembles when pruning both encoders, we do not explore fine-tuning in that setting. Finally, since pruning both encoders removes different number of parameters than pruning only the vision encoder, also report results with an additional six heads pruned to confirm that the gains are not simply due to a smaller increase in parameters.
35

---

Published as a conference paper at ICLR 2026
Table 23: CIFARIO0 (ViT-B/16); OOD detection (Far-OOD splits: MNIST, SVHN Textures and Places365) with MSP.
Melbed
ATTROCT
AU1PR
AUTROCT
119899
ASTRECC
ALT9R
AUTROCT
ASSROCT
(37001)
AUTPR1
SINGLR
20.09
10.63
92.01
25.32
V0.37
05.19
20.31
Vi.th
$9.28
10.77
05.34
14.-1
-4025
94.21
15.87
50..0
96.16
35.22
$7.47
06:53
18.94
95.06
00:45
15.50
06:83
$4.005
15.65
95-16
PAceoD Theesrn
07: 0565
77.05
02.56
01:13
26.40
07.1
01.50 53
70.00
97.79
50.10
30.35
94.5
35.77
1/114
95 D1
MIMO
24006
03:31
03.05
00.03
21.43
08.24
37.61
95.35
LORA- SINGLE
02.79
72.4
91.42
92.35
30.47
96.95
03.03
20.9
58.55
19.53
37.21
95.43
13.07
45.34
04.57
LORA-EN GORA-ENSONABRE
01.34
24.05
91.25
92.37
98.95
03.45
21.18
93.6
50.56
37.01
95.44
04.25
40.38
06.53
ENSEMBBLE
72
63.00
37.61
TAYLOR CaCCANG
73.07
09.54
96.95
$0.35
41.58
96.14
95.65
18.71
95.1
50.53
38,00
33.12
42.10
95.79
79.81
59.68
95.59
92
34.54
06.5
96.42
17.47
$0.15
42.85
95.80
09.34
39.63
96.66
Plstr Ersenttes (Tagton
71:11
00.73
94.84
92.81
11:10
07:13
97.13
15.07
95.98
91.49
35.12
90.3
85:13
11:19
96.30
Table 24: OpenCLIP ViT-B/32 classification results per dataset.
Metries
Method
Heads
Train IN-IK
C100
C10
Food
SUN
Pot
DTD
EuroSat
Caltech
Avg
SINGLE
12
66.13
75.59
93.68
82.02
68 10
87.27
50.48
40.95
98.62
73.65
BAYESVIM
12
V
65.89
75.61
93.75
82.24
67.70 86.97
50.21
41.78
98.62
73.64
Acc
TAYLOR
10
58.92
37.11
64.49
67.18
63.38
82.83
49.31
25.52
97.97
60.75
CIRCAVG
10
64.29
71.79
91.75
79.63
65.67
86.21
50.90
38.94
98.05
71.91
HLYDRA-ENS (CIRC)
10x3
66.20
74.87
93.46
80.50
68.65
87.84
53.88
41.94
98.62
74.00
SINGLE
12
0.47
0.34
0.10
0.26
0.45
0.19
0.67
0.76
0.02
0.36
TEMP SCALING
12
0.46
0.33
0.10
0.26
0.44
0.19
0.64
0.71
0.35
Brier
BAYESVLM
12
v
0.47
0.34
0.09
0.25
0.45
0.19
0.66
0.73
0.02
0.35
TAYLOR
0.57
0.82
0.47
0.46
0.51
0.25
0.70
0.96
0.53
CIRCAVG
10
0.48
0.38
0.12
0.29
0.47
0.19
0.65
0.75
0.04
0.38
HYDRA- ENS (CIRC)
10x3
0.46
0.35
0.10
0.28
0.44
0.18
0.64
0.71
0.05
0.36
SINGLE
12
1.42
0.65
0.21
0.65
1.14
0.56
1.99
1.90
0.03
0.98
TEMP SCALING
12
1.31
0.85
0.20
0.63
1.07
0.52
1.79
1.66
0.03
0.90
NLL
BAYESVLM
12
v
1.35
0.86
0.20
0.62
1.10
0.54
1.89
1.73
0.03
0.93
TAYLOR
10
1.82
3.38
1.10
1.33
1.36
0.70
2.18
2.85
0.08
1.65
CIRCAVG
10
1.44
1.08
0.27
0.72
1.20
0.41
1.91
1.81
0.09
0.99
HYDRA- ENS (CIRC)
10x3
1.36
0.94
0.22
0.70
1.08
0.42
1.80
1.78
0.10
0.93
SINGLE
12
10.00
7.18
1.65
9.88
1.73
18.21
23.93
1.52
8.68
TEMP. SCALING
12
1.65
0.85
1.04
3.12
1.07
3.92
6.78
15.49
2.38
4.03
BLYESVLM
12
/
5.43
4.05
0.85
5.87
1.79
13.88
18.98
5.95
BCB
VILU
12
V
52.20
3.40
0.50
1.00
6.60
6.60
4.00
10.00
1.20
9.50
TAYLOR
13.58
23.95
9.79
11.03
12.03
3.06
18.39
30.28
1.74
13.76
CIRCAVO
10
4.10
1.53
0.75
2.19
4.56
2.93
7.12
9.53
3.29
4.00
HYDRA- ENS (CIRC)
10x3
1.82
2.34
1.31
1.68
1.89
2.41
6.58
7.73
5.65
3.49
SINGLE
12
10.00
7.10
1.53
3.99
9.87
1.79
18.21
23.93
0.66
8.57
TEMP SCALING
12
1.60
0.76
1.04
3.12
1.19
3.87
6.74
15.25
0.65
3.80
BAYESVLM
12
5.42
3.88
0.78
0.72
5.86
1.01
13.88
18.97
0.65
5.68
aECE
VILU
12
49.40
2.60
0.20
6.50
5.50
3,90
9.30
1.40
8.86
TAYLOR
10
13.58
23.95
9.80
11.03
12.03
2.98
18.40
30.28
1.00
13.67
CIRCAVG
4.13
1.60
0.88
2.17
4.55
2.89
7.40
9.51
4.11
HYDRA-ENS (CIRC)
10x3
1.76
2.44
1.59
1.63
1.60
2.31
6.23
7.74
4.86
3.35
All approaches reduce NLL and ECE compared to the single model (see Table 24). For Hydra Ensembles pruned only the vision encoder, remaining competitive with other basclines requires additional training and temperature scaling. Accuracy, nevertheless, remains slightly higher than BLyesVLM. When pruning both en-
Classification. We report the most important metrics Table 27
36

---

Published as a conference paper at ICLR 2026
Table 25: OpenCLIP ViT-B/32 OOD detection with MSP on IN-IK.
Dataset
Metric
SINGLE BAYESVLM VILU
TAYLOR CIRCAVG HYDRA-ENS (CIRC)
Near-OOD
AUROC f
61.36
61.35
66.90
60.72
66.56
66.12
SSB
FPR95 +
86.14 14
86.60
83.92
86.47
82.69
81.90
AUPR
60.64
60.62
65.69
60.28
66.26
65.57
AUROC t
71.39
72.52
73.67
67.69
75.41
75.68
Ninco
FPR95 +
74.10
72.59
74.90
77.48
69.57
69.18
AUPR
22.94
23.68
25.11
19.87
27.65
28.23
AUROC1 1
66.37
66.93
70.29
64.21
70.99
70.90
Avg
FPR95 1
80.12
79.59
79.41
81.97
76.13
75.54
AUPR
41.79
42.15
45,40
40.07
46.95
46.90
Far-OOD
AUROC 1
76.42
78.98
83.10
70.56
89.14
88.82
iNaturalist
FPR95 |
65.26
63.65
60.99
77.02
45.59
43.66
AUPR
40.58
44.36
54.90
33.38
66.01
64.92
AUROC f
73.88
74.51
77.87
70.80
76.42
76.69
Texture
FPR95 1.
75.35
75.80
66.55
76.68
75.20
77.49
AUPR
26.79
27.69
29.58
21.06
30.66
32.71
AUROCT
75.15
76.75
80.48
70.68
82.78
82.75
Avg
FPR95 1
70.30
69.72
63.77
76.85
60.39
60.57
AUPR
33.69
36.03
42.24
27.22
48.34
48.81
OOD
AUROC1
70.76
71.84
75.38
67.44
76.88
76.82
Avg
FPR95 1
75.20
74.65
71.59
79.41
68.26
68.05
AUPR 37.73
39.08
43.81
33.64
47.64
47.85
coders, removing six extra heads leads to only minor drops-and occasional improvements across metries, resulting in overall comparable performance.
OOD Detection. Results are shown in Table 28 Pruning both encoders in Hydra Ensembles gener- ally yields better OOD metrics on ImageNet- and CIFAR-I00, 100, with the exception of CIFAR-100 Far-OOD datasets. Removing six additional heads leads almost identical performance, except for a drop in Near-OOD AUPR. ImageNet-Ik, fine-tuning the pruned model further improves pruning setting.
AUROC and AUPR over the
37

---

Published as a conference paper at ICLR 2026
Table 26: OpenCLIP VIT-B/32 OOD detection with MSP CIFAR-100.
Dataset
Metric
SINGLE
BAYESVLM VILU
TAYLOR CIRCAVG HYDRA-ENS (CiRC)
Near-OOD
AUROC f
78.06
76.54
79.30
76.17
59.14
78.07
79.24
CIFAR-10 10
FPR95
62.80
64.74 61.00
70.49
92.39
52.77
53.48
AUPR 1
76.40
74.74 77.54
74.66
60.57
72.78
74.92
AUROC f
78.10
76.39
78.80
76.82
59.38
79.61
80.32
TinylmageNet
FPR95
58.38
61,34 57.57
65.37
81,42
58.57
57.37
AUPR 1
67.01
64.23 67.75
65.24
47.18
68.05
68.32
AUROC f
78.08
76.47 79.05
76.50
59.26
78.84
79.78
Avg
FPR95
60.59
63.04 59.28
67.93
86.91
55.67
55,42
AUPR f
71.70
69.48 72.65
69.95
53.87
70.41
71.62
Far-OOD
AUROC f
89.71
74.66
77.19 19
73.67
51.79
81.85
75.84
Texture
FPR95
36.16
65.27
63.60
68.66
86.18
54.11
55.97
AUPR 1
97.98
57.45 61.48
95.49
37.32
65.65
56.68
AUROC
97.03
87.55 91.46
92.23
35.87
85,43
82.80
MNIST
FPR95
14.16
40.53 32.88
25.03
86.68
29.92
37.34
AUPR 1
98.84
97.29 98.36
95.83
82.28
95.51
94.64
AUROCT +
76.18
96.39 97.47
86.55
59.07
96.43
95.63
SVHN
FPR95
64.46
16.37
11.37
53.87
68.88
12.91
15.43
AUPR 1
60.66
98.61 99.01
81.45
74.70
98.45
98.01
AUROC f
73.04
70.23 73.88
62.89
45.28
78.32
76.30
Places365
FPR95
71.77
74.96 70.80
85.88
92.63
67.46
69.90
AUPR 1
89.19
87.50 89.53
83,91
75.24
91.57
90.24
AUROC f
83.99
82,21
78.84
85,51
82.64
Avg
FPR95
46.63
49.28 44.66
58.36
83.59
41.10
44.66
AUPR 1
86.67
85.21 87.09
89.17
67.38
83.28
84,89
OOD
AUROC f
82.01
80.29 83.01
78.05
51.75
83.28
81.68
Avg
FPR95
51.28
53.86 49.53
61.54
84.69
45.95
48.24
AUPR 1
81.67
79.96
82.77
82.76
62.87
82.00
80.46
Table 27: OpenCLIP ViT-B/32 classification results per dataset. Comparison between vision en- coder pruned, in both zero-shot and fine-tuning setting, and both encoders pruned using Hydra Ensembles with circuit extraction.
Metrie Method
Pruning
Train IN-IK
C100
C10
Food
SUN
Pet
EuroSat
Caltech
Avg
HYDRA-ENS
Vision+ Text
66.20
74.87
93.46
80.50
68.65
87.84
$3.88
41,94
98.62
74.00
HYDRA-ENS (-6 HEADS)
Vision+Ten Text
66.11
74.14
03.31
80.28
68.57
87.54
52.34
45.35
98.62
74.03
Acc
HYDRA-ENS
Vision
64.48
75.70
93.53
78.61
67.25
86.73
51.28
40.06
98.62
72.92
HYDRA-ENS
Vision
64.25
75.71
93.25
79.89
66.44
87.14
50.64
45.16
98.46
73.44
HYDRA ENDRA-EENS
Vision+Tes Text
1.36
0.94
0.22
0.70
1.08
0.42
1,80
1.78
0.10
0.93
HYDRA-FAL ENS (-6 HEADS)
Vision+Ten Text
1.37
0.96
0.22
0.72
1.09
0..4
1.84
1.74
0.11
0.94
NLL
HYDRA-ENS
Vision
1.47
0.86
0.20
0.78
1.16
0.60
2.01
1.87
0.07
1.00
HYDRA-ENS
Vision
1.47
0.85
0.21
0.71
1.17
0.59
2.01
1.53
0.05
0.95
HYDRA EYDDR--ENS
Vision+Text
1.82
2.34
1.31
1,68
1.89
2.41
6.58
7.73
5.65
3.49
HYDRA-ENS (-6 HEADS)
Vision+Ten Text
1.83
2.87
1.70
2.79
1.32
2.79
6.80
5.57
5.30
3.44
ECB
HYDRA-ENS
Vision
6.98
3.32
0.70
0.93
6.46
1.62
13.78
20.76
1.37
6.21
HYDRA-ENS ENS
Vision
6.51
2.67
0.95
0.90
6.78
2.07
14.72
11.74
1.56
5.32
38

---

Published as a conference paper at ICLR 2026
Table 28: OpenCLIP ViT-B/32 OOD detection (Near/Far) using MSP across datasets. Comparisons include: pruned vision encoder (zero-shot and fine-tuned), both encoders pruned, and both encoders pruned with fewer heads.
Datuset Method
Pruning
Training
Near-OOD Avg
Far-OOD Avg
OOD Average
AURRCC
AUROC1
FPRUS
AUROCT
FPR95
AUPR
HYDRA-ENS
Vision= Vision+Test Toxt
20.50
75.
46.90
82.75
60.57
48.81
76.82
68.05
47.85
HYDRA-ENs ENs (-6 HEADS)
70.93
75.31
46.86
82.71
60.68
48.52
76.82
67.99
47.69
IN-1K
HYDRA-ENS
Vision
66.51
80.52
41.90
78.43
63.68
37.49
72.51
72.10
39.69
HYDRA
Vision
65.83
$1.10
41.37
78.31
66.59
38.88
72.07
73.84
40.12
HYDRA- ENS HYDRR--ENS HEADS)
Test
79.78
55.42
71.62
82.61
44.66
$1.89
$1.68
48.24
80.16
C100
VisioneTest
78.79
$6.67
70.59
87.56
45.09
84.52
81.30
79.88
HYDRA-ENS
Vision
77.00
63.15
70.91
83.58
48.83
86.92
$1.38
53.00
81.58
HYDRA ENs
Vision
78.15
60.76
71.64
83.67
46.76
36.48
$1.82
51.42
$1.53
39


Reviews:
[{"review_id": "PEY5McA8RK", "reviewer": "Reviewer_e9rh", "summary": "This paper introduces Hydra Ensembles, a novel framework designed to achieve uncertainty-aware efficient ensembling for large-scale transformer models such as ViT, BERT, and CLIP. The key motivation is to retain the uncertainty quantification (UQ) robustness of Deep Ensembles while drastically reducing their computational and memory costs.\n\nHydra Ensembles works by pruning attention heads from a single pre-trained transformer to create diverse submodels, which are then fused into a single network via a Grouped Fully Connected (GFC) fusion of their Multi-Head Attention (MHA) and MLP layers. Unlike conventional ensemble approaches, Hydra Ensembles:\n\t•\tavoids retraining each model from scratch,\n\t•\tallows near-single-model inference cost, and\n\t•\tpreserves ensemble diversity for robust uncertainty estimation.\n\nThe paper’s contributions are threefold:\n\t1.\tTheoretical Analysis: Demonstrates that naïve pruning can degrade model calibration under noisy data conditions, supported by a formal proposition showing loss gap widening in pruned models.\n\t2.\tFramework Design: Introduces structured head-level pruning and GFC-based fusion that maintain model diversity while minimizing computation.\n\t3.\tEmpirical Evaluation: Provides extensive experiments on image classification (ImageNet-1K, CIFAR-100), text classification (SST-2), and zero-shot image classification (OpenCLIP). Results show that Hydra Ensembles achieve comparable or superior uncertainty metrics (AUROC, AUPR, ECE) to Deep Ensembles while being ~3× faster and requiring significantly fewer parameters.\n\nOverall, Hydra Ensembles present a theoretically grounded and practically efficient solution for scalable uncertainty quantification in transformer architectures — bridging the gap between computational efficiency and epistemic robustness.", "questions": "1. On the Theoretical Proposition and Calibration Justification\n\t•\tThe paper presents a theoretical result suggesting that random pruning widens calibration loss gaps.\n\t•\tCould the authors expand or clarify the assumptions underlying this result — e.g., is the bound derived under linearized model assumptions, or does it generalize to nonlinear self-attention layers?\n\t•\tHow does this proposition directly link to empirical calibration metrics (ECE, NLL, AUROC)?\n\t•\tWould it be possible to empirically measure information loss or variance reduction (e.g., via Fisher information or entropy) before and after pruning to validate the theory?\n\n2. On Model Diversity and Ensemble Behavior\n\t•\tHydra’s design implies that pruning attention heads creates diverse submodels whose combination improves uncertainty calibration.\n\t•\tHow do the authors quantify or measure diversity among these pruned heads or submodels?\n\t•\tHave they examined pairwise output correlations or disagreement rates across ensemble members?\n\t•\tWithout such evidence, how can we be confident that Hydra’s calibration improvements arise from true epistemic diversity rather than simple regularization effects?\n\n3. On Efficiency–Uncertainty Trade-offs\n\t•\tThe paper reports efficiency gains (~3× faster inference) while maintaining or improving uncertainty metrics.\n\t•\tCould the authors provide explicit quantitative trade-off curves between pruning ratio, uncertainty calibration, and accuracy?\n\t•\tFor example, what is the marginal drop in accuracy or AUROC per additional pruning step?\n\t•\tThis would help practitioners decide optimal pruning thresholds under different compute constraints.\n\n4. On Generalization Beyond Classification Tasks\n\t•\tHydra Ensembles are evaluated on image and text classification tasks, but transformers are widely used for generation and multimodal learning.\n\t•\tHave the authors explored whether Hydra can be applied to sequence generation tasks (e.g., autoregressive decoding, summarization) or vision–language models like CLIP or BLIP?\n\t•\tIf not yet tested, do the authors foresee architectural or stability challenges (e.g., head dependencies in causal self-attention) that might limit its application?\n\n5. On the Fusion Mechanism (Grouped Fully Connected Layers)\n\t•\tThe Grouped Fully Connected (GFC) fusion layer is a central component, but its mathematical and practical behavior could be clarified.\n\t•\tHow does GFC differ from existing ensemble fusion or parameter-sharing mechanisms (e.g., BatchEnsemble, SplitDense)?\n\t•\tIs there a risk of overfitting or co-adaptation when merging diverse heads via GFC?\n\t•\tHow is group size or fusion granularity chosen, and how sensitive is Hydra’s performance to these hyperparameters?\n\n6. On the Scope of Uncertainty Metrics\n\t•\tThe experiments primarily report ECE and AUROC, which measure calibration and discrimination, respectively.\n\t•\tHave the authors evaluated additional uncertainty metrics such as Brier score, NLL, or predictive entropy?\n\t•\tDifferent metrics capture complementary aspects of uncertainty; including them could strengthen claims about “uncertainty-awareness.”\n\n7. On Robustness and Distribution Shift\n\t•\tHydra’s motivation includes improving reliability and robustness through structured diversity.\n\t•\tHave the authors tested Hydra on corrupted or domain-shift datasets (e.g., CIFAR-C, ImageNet-C, SST-2 with noise)?\n\t•\tIf not, could they provide a theoretical argument or empirical proxy suggesting Hydra’s robustness benefits beyond clean test distributions?\n\n8. On Implementation Complexity and Reproducibility\n\t•\tHydra involves structured pruning, submodel fusion, and fine-tuning stages.\n\t•\tCould the authors comment on the implementation complexity and reproducibility — e.g., how many lines of modification are required for ViT or BERT baselines?\n\t•\tAre pretrained Hydra checkpoints or open-source scripts available (or planned) to facilitate community adoption?\n\n9. On Relation to Other Efficient Ensembles\n\t•\tHydra’s conceptual overlap with BatchEnsemble (Wen et al., 2020) and MIMO (Havasi et al., 2021) is acknowledged but not deeply dissected.\n\t•\tCould the authors articulate the precise difference in diversity mechanism between Hydra and these methods?\n\t•\tSpecifically, how does Hydra’s pruning-induced diversity compare empirically to BatchEnsemble’s multiplicative rank-1 reparameterization or MIMO’s input-sharing scheme?\n\n10. On Interpretability of Pruned Attention Heads\n\t•\tSince Hydra modifies attention structure, there may be implications for interpretability (e.g., loss of certain attention patterns or semantics).\n\t•\tHave the authors analyzed whether the pruned heads correspond to interpretable functions (e.g., positional, syntactic, or semantic attention)?\n\t•\tIf not, do they expect pruning to impact model explainability — and could this trade-off affect trustworthiness in downstream deployment?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality\n\t•\tThe paper presents a novel architectural strategy for ensemble diversity by pruning and recombining transformer attention heads rather than training separate models.\n\t•\tThe Grouped Fully Connected (GFC) fusion mechanism introduces an efficient ensembling pipeline that avoids the typical cost explosion of Deep Ensembles.\n\t•\tAlthough it builds on concepts like BatchEnsemble and pruning-based model compression, Hydra’s hybridization of pruning and ensembling is unique in both motivation and execution.\n\t•\tThe formal proposition linking random pruning to calibration degradation is a valuable theoretical insight, providing justification for structured pruning instead of purely empirical reasoning.\n\nQuality\n\t•\tThe methodology is sound and reproducible, with strong empirical results across both vision (ViT, ImageNet-1K, CIFAR-100) and text (BERT, SST-2) domains.\n\t•\tExtensive ablations (pruning ratios, ensemble sizes, GFC variants) demonstrate that the performance improvements are not cherry-picked but systematic.\n\t•\tThe authors carefully balance theoretical analysis, algorithmic description, and experimental validation — showing maturity in both design and evaluation.\n\t•\tCalibration metrics (ECE, AUROC, AUPR) are properly selected for uncertainty quantification, and their consistent improvement validates the main claim.\n\nClarity\n\t•\tThe paper is well-organized and readable, with clear section flow and minimal redundancy.\n\t•\tFigures such as the Hydra architecture schematic and calibration plots effectively support understanding.\n\t•\tThe theoretical analysis is concise and mathematically consistent, though dense in presentation — yet the accompanying intuition keeps it accessible to non-specialists.\n\t•\tNotation is consistent, and references to related work are appropriate and fair.\n\nSignificance\n\t•\tPractical significance: Hydra Ensembles deliver ensemble-level uncertainty quality at a fraction of the computational cost, which is crucial for large transformer models where full ensembles are infeasible.\n\t•\tResearch significance: The framework provides a blueprint for uncertainty-aware model compression, bridging a long-standing gap between trustworthiness and efficiency.\n\t•\tCommunity impact: The method is relevant to multiple research threads at ICLR — efficient transformers, uncertainty quantification, and scalable AI reliability.\n\t•\tIts applicability to both vision and language domains broadens its potential adoption and demonstrates methodological generality.", "weaknesses": "1. Limited Theoretical Grounding of Calibration Claims\n\t•\tThe proposed theoretical proposition—that random pruning leads to calibration degradation—is intuitively plausible but lacks rigorous derivation or empirical verification linking the theory to measured uncertainty metrics (ECE, NLL, Brier score).\n\t•\tThe argument relies primarily on Fisher Information heuristics rather than a formal probabilistic treatment of epistemic variance or diversity loss.\n\t•\tThis makes the theoretical part informative but incomplete, as it doesn’t generalize to nonlinear pruning effects or the stochastic dynamics of self-attention.\n\nRecommendation:\nThe authors could expand this section by (a) connecting pruning-induced variance loss to epistemic uncertainty via bias–variance decomposition, or (b) empirically validating the proposition using entropy/Fisher metrics before and after pruning.\n\n2. Moderate Conceptual Novelty\n\t•\tHydra Ensembles’ innovation lies mainly in the engineering combination of known techniques: pruning for efficiency (Michel et al., 2019; Voita et al., 2019) and efficient ensemble fusion (Wen et al., 2020; Havasi et al., 2021).\n\t•\tWhile the integration is elegant, it does not introduce a fundamentally new learning principle or uncertainty formulation.\n\t•\tRelated works like BatchEnsemble (Wen et al., ICLR 2020), MIMO (Havasi et al., NeurIPS 2021), and LayerDrop (Fan et al., ICLR 2020) already explore efficiency–diversity trade-offs; the paper could better clarify where Hydra theoretically diverges from these beyond implementation detail.\n\nRecommendation:\nReframe Hydra as a structured synthesis approach rather than a conceptual breakthrough, emphasizing its engineering elegance and scalability benefits.\n\n3. Scope of Experiments\n\t•\tAll evaluations are confined to classification tasks (CIFAR-100, ImageNet-1K, SST-2). No tests on generation or multimodal transformers (e.g., CLIP zero-shot) beyond classification accuracy and calibration.\n\t•\tWithout broader task validation, it’s unclear whether Hydra’s uncertainty improvements generalize to tasks requiring sequence modeling, open-ended text generation, or multi-modal reasoning—key frontiers of transformer research.\n\nRecommendation:\nInclude or discuss pilot results on transformer-based generative tasks (e.g., GPT-style models) or multimodal settings to support claims of generality.\n\n4. Insufficient Discussion of Trade-offs\n\t•\tThe paper highlights efficiency gains (3× faster inference, fewer parameters) but lacks quantitative trade-off analysis between pruning ratio, uncertainty calibration, and ensemble diversity.\n\t•\tThe reader is left without a clear sense of how much diversity is sacrificed at higher pruning rates or how inference cost scales with ensemble size.\n\t•\tAdditionally, calibration–efficiency curves or Pareto plots would strengthen interpretability.\n\nRecommendation:\nProvide explicit trade-off visualizations (e.g., efficiency vs. ECE or AUROC) and discuss how practitioners can tune pruning levels for optimal performance.\n\n5. Missing Analysis of Diversity and Correlation Among Pruned Submodels\n\t•\tSince ensemble robustness depends on model diversity, the paper should quantify inter-head diversity or correlation (e.g., using cosine similarity, pairwise prediction disagreement, or mutual information across pruned models).\n\t•\tWithout such analysis, it’s difficult to confirm whether Hydra truly achieves “diverse ensembling” or merely benefits from redundancy.\n\nRecommendation:\nAdd an empirical diversity analysis to support the central claim of maintaining epistemic diversity post-pruning.\n\n6. Limited Robustness and OOD Testing\n\t•\tAlthough Hydra improves calibration, there are no results under domain shift or corrupted data (e.g., ImageNet-C, CIFAR-C, SST-2 perturbations).\n\t•\tThis omission weakens claims about robustness and “uncertainty-awareness,” since true epistemic reliability is best evaluated under distributional drift.\n\nRecommendation:\nInclude robustness experiments on corrupted or OOD benchmarks to demonstrate Hydra’s behavior under uncertainty-inducing conditions.\n\n7. Presentation and Comparison Clarity\n\t•\tSome mathematical notation is dense and occasionally inconsistent across sections (e.g., subscripts for attention heads and fusion layers).\n\t•\tRelated work could be contextualized more critically — especially contrasting Hydra’s scalability and calibration against post-hoc methods like temperature scaling, EMM, or confidence regularization.\n\nRecommendation:\nAdd a summary table contrasting Hydra with BatchEnsemble, MIMO, and FastGeLU Ensembles, highlighting distinct features, efficiency, and calibration properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uFlmq2tACH", "reviewer": "Reviewer_twYJ", "summary": "The paper proposes Hydra Ensembles: make a few differently pruned versions of the same transformer (different attention heads kept), fine-tune them, and then fuse them into a single model using fused MHA + grouped FC so you can get “ensemble-like” predictions at roughly single-model cost. It targets efficient uncertainty estimation for transformers.", "questions": "1. When you fuse the pruned members into one model, do you still get separate member outputs so we can measure disagreement, or is it combined into one prediction? A small clarification (and maybe a number) would help.\n\n2. If different members keep different attention heads, does the fused layer run the union of those heads (which could increase compute), or do you share some heads to stay close to 1×?\n\n3. You average / group MLP weights across members. Did you try an alternative (e.g. a small per-member adapter) and it didn’t help, or was this mainly for simplicity?\n\n4. How sensitive is Hydra to how aggressively you prune? A short plot or table for one dataset would clarify how robust the method is.\n\n5. For the different Hydra members, how do you ensure that the pruned head sets are actually different (i.e., not largely overlapping)? Do you use different random seeds for the pruning score, or do you enforce low overlap between members?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles efficient uncertainty estimation for large transformers, a problem people actually have. That makes the work naturally interesting.\n\n2. Pruning-as-diversity is a nice angle. Most pruning papers try to preserve one model; here, pruning is used to induce differences between members. That’s a small but genuine conceptual twist.\n\n3. The method is demonstrated on both vision (ViT) and language (BERT-style) models, plus a zero-shot setting, so it doesn’t look tied to one benchmark. That strengthens the claim of generality. \n\n4. The cost story is attractive. Claiming ~1× inference vs ~3× for Deep Ensembles directly supports the motivation; it’s exactly the comparison readers care about", "weaknesses": "1. The whole method relies on “different pruned heads = different members,” but the paper doesn’t show simple diversity metrics (e.g. pairwise disagreement/KL) before and after fusion. For this idea, that’s the key missing evidence.\n\n2. Several main tables give single numbers but no std / CI / ± over seeds, even though the paper is about uncertainty/robustness.\n\n3. On SST2, the ensemble seems cheaper than a real, fully fine-tuned deep ensemble, which makes Hydra look better than it might against a “full” baseline. \n\n4. The attention fusion story is clear; the MLP part is basically “we average/group.” Since members were pruned and fine-tuned separately, that choice could wash out diversity; an ablation or a justification can be beneficial.\n\n5. The small theoretical part about pruning hurting more under noisy/OOD inputs is more motivational than general; assumptions aren’t clearly checked on ViT/BERT. Either support it empirically in the main text or de-emphasize it.\n\n6. Because zero-shot UQ is sensitive to prompts/datasets/temperature, more details should be in the main paper, not just the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A7nmONgQiv", "reviewer": "Reviewer_Ez3D", "summary": "This article deals with uncertainty quantification (UQ) in transformer-based architectures. It observes that UQ is difficult to achieve, with the current state of the art being held by Deep Ensembles (DE), a contribution from 2017. Deep ensemble achieve sometime-excellent results on UQ at the expense of very expensive training and inference, since multiple training and inference with different parameters must be performed every time.\n\nThe article proposes to use pruning methods on the attention heads of transformer architectures only. They justify this approach by suggesting that such pruning is easier to control and that a combined network can then be put together, performing efficient prediction and UQ at once. This proposal is thoroughly justified on both theoretical and practical grounds, achieving close to the state of the art result at virtually no cost on BF16 precision due to the specialised hardware involved. When using FP32 precision, the computational gains are almost non-existent with respect to DE, however.", "questions": "- What would be the cost and gain of using 5 heads in BP16 instead of just 3?\n- Can the source code be made available for review? Many contributors promise to publish code that turn out to be unreadable, uncommented or incomplete.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The article is relatively easy to read, very well justified with complete and detailed theoretical and practical explanations. The source code is promised to be made available upon acceptance.", "weaknesses": "The success of an ensemble heavily relies on the diversity of its component models. If all models make similar mistakes, combining them won't lead to much improvement. Strategies like varying training data, model architectures, or initialization are needed to ensure diversity\n\nAdaptive Attacks: Sophisticated attackers can create adaptive adversarial examples that specifically aim to minimize the uncertainty metrics (like variance or entropy) of the ensemble, attempting to make their attack look like a confident, in-distribution prediction.\n\nMachine-learning models can be fooled by adversarial examples, i.e., carefully-crafted input perturbations that force models to output wrong predictions. While uncertainty quantification has been recently proposed to detect adversarial inputs, under the assumption that such attacks exhibit a higher prediction uncertainty than pristine data, it has been shown that adaptive attacks specifically aimed at reducing also the uncertainty estimate can easily bypass this defense mechanism.\n\nhttps://arxiv.org/abs/2309.10586", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXAKlVhHWA", "reviewer": "Reviewer_J4z9", "summary": "The paper addresses the challenge of uncertainty quantification (UQ) in large-scale transformer models. While Deep Ensembles are known to provide reliable uncertainty estimates, they are computationally expensive due to multiple independently trained models. The authors aim to retain ensemble-level calibration and robustness while significantly reducing computational and memory costs. Hydra Ensembles constructs an efficient transformer ensemble by pruning attention heads in a single pre-trained transformer to generate diverse subnetworks and merging these pruned models into a single architecture using Grouped Fully Connected (GFC) layers, forming a Fused Multi-Head Attention (MHA) and Merged MLP structure.", "questions": "* If each model differs only in the set of surviving heads (line 240), why do you need to average the weights and biases across the M models for the MLP layer (line 247)? Are these not the same?\n* In Hydra Ensembles(circuit), the authors use the Headmap method (Wang et al., 2025) to identify which heads matter most for uncertainty, and remove the rest. What would be the impact of optimizing for a different task? Also, if you're always optimising for which heads matter most for uncertainty and removing the rest, how do you get different results across the M models?\n* Given the strong performance of Taylor and CircuitAverage in the benchmark (on classification and zero-shot tasks), is it worth including them in the benchmark of inference time?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and well motivated. \n* The method can be applied to different MoE architectures.\n* It shows good improvements in OOD performance.", "weaknesses": "* The paper emphasizes ensemble diversity but does not quantify the resulting predictive diversity of the pruned members or the fused model against standard baselines. Appendix B.5 analyzes sources of diversity (e.g., seeds, batch order) but stops short of reporting diversity magnitude after pruning/fusion (e.g., disagreement rate), leaving it unclear whether Hydra is more or less diverse than alternatives.\n* On image classification, in-distribution calibration (Brier, NLL) is roughly on par with a single model, suggesting the method’s gains are concentrated in OOD detection. So the only benefit of the model is for OOD detection, and it does not affect the method's robustness in IND, which is counterintuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QCjRsFIWLP", "reviewer": "Reviewer_UeX6", "summary": "This work introduces a method for uncertainty estimation in transformers.\nThe key idea is producing an ensemble of models via pruning attention\nheads - which leads to reduced computational complexity compared\nto a full ensemble. Evaluation on image and text classification\nbenchmarks indicates close to state-of-the-art performance.", "questions": "- Have you considered re-using existing MoE-s for uncertainty estimation directly?\n- Evaluation is conducted on prediction / OOD tasks. Would the method also work out-of-the-box for generation tasks, MoE-style?\n- I wonder if authors have any intuition on why LoRA ensembles would work worse than fine-tuned pruning?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "*Clarity:*\n- The paper is well-written and is easy to follow.\n\n*Significance / soundness:*\n- This work addresses an important problem of epistemic uncertainty estimation in a widely used transformer architecture.\n- Method itself and technical solutions seems well-motivated and simple to implement, and paper provides sufficient details for reproducing the results.\n- The fact that this method works w/o retraining the base model is a key benefit of the proposed method - re-training large models is intractable.\n- Authors provide multiple variants of the method suitable for different settings (with and w/o uncertainty val set, fine-tuning vs zero-shot).\n\n*Evaluation:*\n- Performance of the method is close to state-of-the-art, both on prediction and OOD tasks, while the cost is significantly lower, especially in the practical bfloat16 setting. This makes the method extremely useful in practice.", "weaknesses": "*Novelty:*\n- MoE models seems to be a very similar approach to what is being proposed in this work. A naive baseline\ncould be re-using an existing MoE model for getting uncertainty estimates?\n\n*Evaluation:*\n- Authors claim that the results do not require additional re-training, but in practice this seems a bit misleading\nbecause for both variants of the model (Taylor and Circ) either needs access to uncertainty validation set or\nactually requires fine-tuning.\n- Paper provides reasonable argument for pruning attention heads instead of MLP, but does not provide a\nquantitative evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)