Pathologies of Predictive Diversity in Deep Ensembles
Authors: Taiga Abe
E. Kelly Buchanan
Geoff Pleiss
John Patrick Cunningham
Venue: Accepted by TMLR ()

Abstract: Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models, e.g. through bagging or boosting. Here we demonstrate that these intuitions do not apply to high-capacity neural network ensembles (deep ensembles), and in fact the opposite is often true. In a large scale study of nearly 600 neural network classification ensembles, we examine a variety of interventions that trade off component model performance for predictive diversity. While such interventions can improve the performance of small neural network ensembles (in line with standard intuitions), they harm the performance of the large neural network ensembles most often used in practice. Surprisingly, we also find that discouraging predictive diversity is often benign in large-network ensembles, fully inverting standard intuitions. Even when diversity-promoting interventions do not sacrifice component model performance (e.g. using heterogeneous architectures and training paradigms), we observe an opportunity cost associated with pursuing increased predictive diversity. Examining over 1000 ensembles, we observe that the performance benefits of diverse architectures/training procedures are easily dwarfed by the benefits of simply using higher-capacity models, despite the fact that such higher capacity models often yield significantly less predictive diversity. Overall, our findings demonstrate that standard intuitions around predictive diversity, originally developed for low-capacity ensembles, do not directly apply to modern high-capacity deep ensembles. This work clarifies fundamental challenges to the goal of improving deep ensembles by making them more diverse, while suggesting an alternative path: simply forming ensembles from ever more powerful (and less diverse) component models.

Published in Transactions on Machine Learning Research (01/2024)
Pathologies of Predictive Diversity in Deep Ensembles
Taiga Abe
lu2507@cokumbia.cco eduu
Center for Theoretioal Neuroscience Department of Neuroscience Cohumbia University
E. Kelly Buchanan
ekb2154 @columbia. edu
Center for Theoretical Neuroscience Department Newroscience Cohumbia University
Geoff Pleiss
geoff pleiss Ostat. ubc.ca
Department of Statistics University of British Columbia Vector Institute
John Cunningham
jpc2181 @columbia edu
Department of Statistics Columbia University
Reviewed OpenReview: https: openreview. net/ forum? id= TQfQUksaCB
Abstract
Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models, e.g. through bagging boosting. Here we demonstrate that these intuitions do not apply to high-capacity neural network ensembles (deep ensembles), and in fact the opposite is often true. In large scale study of nearly 600 neural network classification ensembles, we examine a variety of interventions that trade off component model performance for predictive diversity. While such interventions can improve the performance of small neural network ensembles (in line with standard intuitions), they harm the performance of the large neural network ensembles most often used in practice. Surprisingly, also find that discouraging predictive diversity is often benign in large- network ensembles, fully inverting standard intuitions. Even when diversity-promodl interventions do not sacrifice component model performance (e.g. using heterogeneous architectures and training paradigms), we observe an opportunity cost associated with pursuing increased predictive diversity. Examining over 1000 ensembles, we observe that the performance benefits of diverse architectures/ training procedures are easily dwarfed by the benefits of simply using higher-capacity models, despite the fact that such higher capacity models often yield significantly less predictive diversity. Overall, our findings demonstrate that standard intuitions around predictive diversity, originally developed for low-capacily ensembles, do not directly apply to modern high-capacity deep ensembles. This work clarifies fundamental challenges to the goal of improving deep ensembles by making them more diverse, while suggesting an alternative path: simply forming ensembles from ever more powerful (and less diverse) component models.
architectures/trrainng
1 Introduction
Successful ensembles require component models that make different errors (Dietterich, 2000; Breiman, 2001; Zhou, 2012; Abe et al., 2022a). This intuition be formalized by decomposing ensemble performance into

---

Published in Transactions on Machine Learning Research (01/2024)
two terms: i) the average performance of component models and iii the predictive diversity across models (see e.g. Breiman, 2001; Webb et al., 2020; Gupta et al., 2022, and See. 3). Historically, advances in ensemble methodology have largely focused on the latter term. Consider for example, ensembles of decision trees. Bagged trees (Breiman, 1996) use bootstrapped training data to create different component tree predictions, and random forests (Breiman, 2001) create additional diversity by also subsampling the features used by component models. These training procedures operate at a trado-off; they produce worse component models, but the additional diversity amongst component models yields better aggregabe performance (Breiman, 1996; 2001). Random forests generally outperform bagged trees, which outperform single decision trees.
In contrast, deep ensembles (Lakshminarayanan et al., 2017) do not rely on training procedures that actively encourage differences amongst component models. It is instead common to ensemble neural networks with identical architectures and training procedures, where the only source of diversity is the inherent randomness of parameter initialization and stochastic optimization. At first glance, this homogeneity seems massively ineficiiet- we could improve upon deep ensembles if each component model used diverse architoctures, training procedures, or Mataset//ealuur subsamples? Surprisingly, despite many proposals to make deep ensembles more diverse e.g. by varying network architectures, datasets, hyperparameters, or training objectives (e.g. Wenzel et al., 2020; Wu et al., 2021; Gontijo-Lopes ct al., 2022; Ortega et al., 2022) mechanisms achieve more than a modest gain over the standard deep ensembling benchmark (Ashukha et al., 2020; Abdar et al., 2021; Tran et al., 2022), which remain the gold standard ensembling technique for large scale classification tasks (e.g. Szegedy et al., 2015; Lakshminarayanan et al., 2017; Tran et al., 2022) (see Sec. 2 and Appx. C for a detailed review). More often than not, standard techniques that trade off component model performance for diversity (e.g. bagging) harm ensemble predictions (Tee et al., 2015; Nixon et al., 2020).
This discrepancy between decision tree and deep neural network ensembles should make us reevaluate the notion that encouraging diversity is always a useful technique to improve ensemble performance. On one hand, the relative succoss of standard deep ensembles might indicate that we have not yet found the right mechanisms to encourage further diversity amongst modern neural networks. On the other hand, it may be the case that intuitions about ensembling simply do not carry over from small decision trees to modern neural networks, which often operate in the overparametrized regime with near zero training error. In other words, the trade-off between component model performance and predictive diversity may be fundamentally different for ensembles of decision trees and deep ensembles.
In this work, we provide strong evidence in favor of the latter hypothesis. We find that for modern classificatio deep ensembles, prioritizing predictive diversity over component model performance is counterprouuctive and potentially harmful. We base this claim on two main results. Our first result is concerned with mechanisms that explicitly encourage predictive diversity at the cost of component model performance. We manipulate deep ensemble predictive diversity in a controlled setting via joint-training mechanisms (Mishtal and Arel, 2012; Zhao et al,, 2022; Ortega et al., 2022; Abe et al., 2022a). As with many established ensembling techniques, these mechanisms do not encourage predictive diversity "for free," but rather operate at a trade-off: the increased diversity comes at the cost of individual component model performance. Our results, which consider roughly 600 deep ensembles, demonstrate a dichotomy between established ensemble intuitions and the behavior of modern deep ensembles. In particular, encouraging predictive diversity in ensembles of large neural networks like ResNet 18 (He et al., 2016) consistently leads to worse ensemble performance, as the increase in predictive diversity between ensemble members does not offset the steep cost. incurred component model performance. This behavior contrasts with that of low-capacity ensembles, including ensembles of small neural networks, which improve under diversity encouragement as expected by standard intuitions. Surprisingly, we also find discouraging diversity does not affect high-capacity deep ensembles, and is sometime beneficial. We attribute these counterintuitive phenomena to (1) the often overlooked effect of diversity encouragement on accurate, confident predictions; and (2) recent evidence around the variance-reducing effects of overparameterization (Adlam and Pennington, 2020; d' d'Acooli et al., 2020).
Our second result is concerned with techniques such as using diverse architectures or training procedures- which increase predictive diversity while maintaining component model performance. While not explicitly harmful, we demonstrate that exploiting such "free" sources of predictive diversity involves an opportunity cost. Drawing from a pool of 304 independently trained neural networks, we form over 1000 homogeneous
2

---

Published in Transactions on Machine Learning Research (01/2024)
and heterogencous deep ensembles (combining models with the same vs. different architectures) and measure ensemble performance a function of both predictive diversity and component model performance. In agreement with prior work (e.g, Ashukha et al., 2020; Abdar et al., 2021; Contijo-Lopes et al., 2022), we indeed find that the "free" predictive diversity of a heterogencous ensemble yields slightly better performance than homogeneous ensembles. At the same time, our results demonst that these slight gains pale in comparison to the benefits of simply using more powerful component models, despite more powerful models being less diverse. In other words, the best deep ensembles are formed by trading off predictive diversity in favor of component model performance, inverting the standard intuition. Critically, this opportunity cost and trade-off depend upon the capacity of component models: for example, we find that while the best modern deep ensembles have very low predictive diversity the best decision tree ensembles have the most predictive diversity. We can observe sharp transitions between these two regimes by ensembling neural network models across a wide range of model capacities.
Taken together, our results strongly suggest that ensembles of modern neural networks operate outside the regime of long-held intuitions around the net benefits of increasing predictive diversity. Moreover, they highlight component model capacity as critical and overlooked factor affecting ensemble dynamics. Obtaining mcaningful predictive diversity that bucks the tradc-offs and opportunity costs that we observe will likely require radically new methodology. In the meantime, the best way to improve deep ensembles is simply to use better, but less diverse, individual models.
Summary of contributions. We first establish the relationship between our study and previous work in Sec. 2, and describe our experimental setup in Sec. 3. We go on to demonstrate that the size of component models determines the role of predictive diversity in ensemble performance. We show this phenomenon in two distinct settings, focused on large scale classification:
We show that optimization objectives which encourage or discourage predictive diversity between ensemble members succeed in increasing predictive accuracy on ensembles of small models, but these same objectives fail when applied to ensembles of large component models. We demonstrate this result. on over 600 deep ensembles, across various architectures, datasets, and training objectives, and show that our findings reconcile differences between the traditional ensembling literature and various methods proposed to improve deep ensembles (Sec. 4).
Additionally, we show that our results generalize to many forms of implicit diversity encouragement, i.c. by ensembling across different neural network architectures and hyperparameters. We find that while there may be marginal gains from ensembling more diverse models in such a context, ensemblee can be improved more effectively simply by ensembling more powerful, less diverse component models (Soc. 5).
2 Related work
Traditional ensembles. Historically, ensembling has been applicd to low-capacity base models such decision trees (e.g. Freund, 1995; Breiman, 1996; Dietterich, 1998; Breiman, 2001; Cutler and Zhao, 2001), notably different setting than modern deep networks. Breiman (2001) identifies the performance of base learners and correlation between ensemble members as the key factors that control ensemble performance. More recent works (Gupta et al., 2022; Wood et al., 2023) discuss diversity metrics that can be derived directly from the methods used to evaluate ensemble member predictions.
Deep ensembles. Ensembles of neural networks have long been considered (e.g. Hansen and Salamon, 1990; Perrone and Cooper, 1992), but deep ensembles generally refer to ensembles formed over random initialization of modern deep networks. Deep ensembles are most commonly deployed for classification, at times even reformulating regression as classification (Stewart et al., 2022). The standard approach to deep ensembling is to train multiple copies of the same model architecture (e.g. Lee el al., 2015; Lakshminarayanan et al., 2017; Fort et al., 2019; Buchanan et al., 2022; Tran et al., 2022). If available, ensembling different architectures or training procedures yields modest gains in performance (e.g. Fort et al., 2019; Gontijo-Lopes et al., 2022). However actively encouraging predictive diversity (i.e., diversity in output space) during training (e.g. Mishtal and Arel, 2012; Lee et. al., 2015; Pang et al., 2019; Ross et al., 2020; Thakur al., 2020; Webb et al., 2020;
3

---

Published in Transactions on Machine Learning Research (01/2024)
Wen et al., 2020; Buschjäger et al., 2020; Jain et al., 2020; Gong et al., 2022; Ortega et al., 2022; Pagliardini et al., 2022; Teney et al., 2022; Zhao et al., 2022) has no consistently demonstrated benefit. (see Table 2), except for small networks not often used in practice (Mishtal and Arel, 2012; Opitz et al., 2017; Pang et al., 2019; Webb et al., 2020; Ortega et al., 2022; Pagliardini et al., 2022) or when other training data are available (e.g. Contijo-Lopes et al., 2022). Beyond acknowledging the failure of existing techniques to improve ensemble performance (e.g. Lee et al., 2015; Nixon et al., 2020) the potentially negative consequences of encouraging predictive diversity in modern deep ensembles are often overlooked.
Other studies of predictive diversity. Other works have studied diversity encouragement for logit based averaging of deep ensemble members (Buschjäger et al., 2020; Webb et al., 2020) sctting where the mechanisms proposed in this paper do not necessarily apply. Although recent work suggests that logit averaging is a more theoretically well founded procedure than probability averaging (Gupta et al., 2022; Wood et al., 2023), probability averaging (which the focus of our study) remains the most frequently used form of doep ensembling (Lakshminarayanan et al., 2017). Two of the methods that we employ control predictive diversity in deep ensembles were designed to encourage ensemble diversity, as mentioned above (Mishtal and Arel, 2012; Ortega et al., 2022), while the others were originally used to regularize large-model training (Zhao et al., 2022), or to encourage and discourage predictive diversity in regression and certain large scale models (Abe et al., 2022a; Jeffares et al., 2023), respectively. We go beyond the original uses of these methods by systematically studying how diversity encouragement/ arcourngeeent/discounggeee
differs based on the size of component models across large scale classification tasks. Notably, concurrent work (Jeffares et al., 2023) shows that one of the mechanisms we study here can find degenerate solutions to optimization even in regression tasks or with low-capacity component models. Their conclusions are complementary to our own, and their experiments support the arguments of this work. Finally, do not address methods which encourage ensemble diversity in the weight space, the feature space, or the gradients of ensemble members (e.g. Dabouei et al., 2020; D' o'Angelo and Fortuin, 2021; Sinha et al., 2021), and focus on methods which explicitly target diversity in the prediction space of ensemble members. While studying these other sources of ensemble diversity is an interesting point of future work, our primary goal is to highlight the capacity of component models as critical, understudied feature which determines the impact of encouraging predictive diversity in ensembles.
3 The trade-off between predictive diversity and component model performance
Problem Setting. We primarily consider multiclass classification problems with inputs x RD and targets E [1....,C] which is the canonical task for deep neural networks and thus deep ensembles (Hui and Belkin, 2020; Stewart et al., 2022). A standard deep ensemble consists of M models fi(-)...., (M))).......)),,, where each f. maps input a to the C-dimensional probability simplex. The ensemble prediction f(x) is:
M f(x) A 1 NI S.)0)
(1)
Typically, component models are instances of the same neural network architecture. The component models are usually trained independently on the same dataset with the same loss function ((f((()) v). This procedure is equivalent to minimizing the average empirical single model risk:
Lavg = Ravg
M 1 (p [e (f) v)] = E E (w.w)
M &
A decomposition of the ensemble risk. At test time, use the aggregated ensemble prediction as in Eq. (1). From Eas. (1) and (2) we have the following decomposition of the ensemble risk:
Rens Ep(m.)) [e
(f()),))] = Ravg - Ep(a.w)
1 M EM'
(3)
ens. loss
Jensen gap (pred. diversity)
(where Ravg is the non-empirical average single model risk). We assume that l is a strictly convex loss (c.g. cross entropy, mean squared error, etc.) and thus the last term- the Jensen gap is non-negative. Throughout this paper we will use the Jensen gap as measure of predictive diversity (as in Abe et. al. (2022a)): it will
4

---

Published in Transactions on Machine Learning Research (01/2024)
Decomposing ensemble risk
The diversity/compoonett model performance trade-off. Ensemble performance, as measured as negative log likelihood (NLL), is decomposed into average single model NLL (vertical axis) and Jensen-gap prodictive diversity (horizontal axis)-sce Eq. (3). Diagonal lincs correspond to level sets of ensemblee NLL (lower right is better). The performance of any ensemble can be plotted as point on this graph (x) with a corresponding level set of ensemble performance (thick diagonal line). Along a level set, all ensembles have the same NLL. There are two strategics for improving the performance of any given ensemble: increasing the predictive diversily of the component models (right arrow) or improving the average NLL of the component models (down arrow). If resulting ensembles stay below the thick diagonal, they will improve performance.
Figure/ 1:
0.3
X Baseline ens. performance
(d)
0.2
Increasing pred. diversity
Aug. single m model risk
Improving components
a.
A
W =
Jensen gap (pred. diversity)
be 0 if and only if all component models make the same prediction, and will increase as the fx grow more dissimilar. Thus, Eq. (3) offers a straightforward dccomposition of ensemble performance (Rens) into (i) the average performance of its component models (Ravy) and (ii) the predictive diversity amongst models (Jensen gap), as we will use in our analysis later. Fig. depicts this decomposition: the horizontal and vertical axes correspond to predictive diversity and average component model risk (respectively), while diagonal lines are level sets of ensemble risk. In Appx. D, we verify that the Jensen gap is highly correlated with other metrics of predictive diversity used in previous works.
Trading off diversity and component model performance. Fig. 1 captures many of our intuitions around the trade-offs between diversity and component model performance. For example, ensembles of decision trees demonstrate that one can beneficially trade off component model performance in favor of ensemble diversity. Ensembles that improve in this way can be represented in Fig. in the area right of x, below the thick diagonal. Likewise, if encouraging diversity yields worse ensemble performance, then it must be due to a loss in the performance of the component models (Fig. area right of x, above thick diagonal.) In both cases, we are compromising single model performance to increase ensemble diversity, but there are distinct regimes where this trade-off can help or harm ensemble performance. Note that an analogous statement holds for trade-ofTs that improve the performance of component models well: we might hope to improve component models without losing too much diversity (Fig. 1, area left of x, below thick diagonal) as an alternative method of improving ensemble performance.
4 Manipulating predictive diversity in neural network ensembles
For ensembles of smaller models, methods like bagging and random forests benefit ensemble performance, despite trading off component model performance in favor of predictive diversity. Intuitions from these well-studied paradigms have driven many efforts to achieve this same, favorable trade-off in ensembles of modern deep neural networks. However, in this section we show that this trade-off yields opposite outcomes in modern deep ensembles. In particular, actively encouraging diversity (at the cost of component model performance) is detrimental to deep ensemble performance, while discouraging diversity is benign or beneficial. We find that the degree to which diversity encouragement/ encouragenentt/disouugggg /discouragement is harmful/bentn /bbiign depends on the capacity of component models. If we substantially decrease the size of component models (c.g. using very small works like ResNet 8), diversity encouragement once again becomes beneficial in line with standard intuitions. Nevertheless, practice will be in the regime where diversity encouragement is harmful. We attribute this finding to simple but often overlooked phenomenon: diversity encouragement disproportionately impacts highly confident and accurate predictions, which are a majority of predictions from high-capacity neural networks.
emphasize that the capacity of most modern neural networks used in
Mechanisms to encourage/ /dicouurge predictive diversity. To control predictive diversity in these ensembles, we experiment with four modifications of standard ensemble training that enable us to en- courage/discourage predictive diversity by variable amount. The first three mechanisms consider to
5

---

Published in Transactions on Machine Learning Research (01/2024)
encourage/discourage predictive diversity explicitly regularize standard ensemble training (Eq. 2):
Cens.
Lavs
E Presin (2ey))
(
(4)
diversity regularizer
where is hyperparameter (negative values encourage diversity; positive values discourage diversity) and D is some measure of predictive diversity.
1. Variance regularization (Ortega et al., 2022) sets D to:
D ((1(())......A
2(M (.M-1) max, 11 EE f(())
(5)
which essentially corresponds to the variance over ensemble members' true-class predictions.
2. 1-vs-All Jensen-Shannon divergence-based regularization (Mishtal and Arel, 2012) sets ID to:
TD far( = 1 ME JS (fi(ar) I=
(6)
where JS is the
divergence.
3. Uniform Jensen-Shannon divergence- based regularization (Zhao et al., 2022) sets D to:
DD (f(()))....ff(()))) H (f(z)) H
(7)
where H is Shannon entropy. Note that the term inside the brackets is equivalent to the multi-distribatton Jensen-Shannon divergence with uniform weights.
4. Jensen gap-based regularization (as introduced in Abe et al. (2022a)) corresponds closely to Eq. (3), as we consider the following objective function to srroourag/ddiscourrg diversity:
{ (f(x))))) + (7 +11 1) (1 ( (f(a)))))
Lens
E Ptrain(m..))
(8)
The first term is the ensemble prediction loss (as opposed to the average single model loss), while the second term is the Jensen gap notion of predictive diversity.
For all methods, discourages diversity. Thus, we can frame encouragement/die versions of standard ensemble training (Eq. 2). As with any regularization mechanism, we expect that manipulating predictive diversity also affects the performance of component models, and we will study how trade-offs between these factors impact overall ensemble performance. Sec. 5 studies the effects of diversity sources that do not fall into this paradigm (diversity using different architectures, training paradigms, etc.) In Appx. F.1, we further discuss differences between these individual diversity mechanisms and evaluate the performance of each regularizer independently.
0 corresponds to standard ensemble training, 7 < 0 encourages diversity, and 7 > 0
aecorragmeen/diioomggeeen /discouragement of ensemble diversity as regularized
Datasets and architectures. We train deep ensembles on the CIFAR10, CIFAR100 (Krizhevsky et al., 2009), TinyImageNet (Le and Yang, 2015), and ImageNet (Deng al., 2009) datasets. Ensembles consist of M models of the same neural network architecture for all datasets except ImageNet, where M 3. On CIFAR datasets, study ResNet 18 (He et al., 2016), VGG 11 (Simonyan and Zisserman, 2014), WideResNet 28-10 10 (Zagoruyko and Komodakis, 2016), DenseNet 121 (DenseNet 161 for CIFAR100) (Huang et al., 2017), Res.et 8, and LeNet (LeCun et al., 1998) models. The first four are considered large architectures, while the last two are small architectures. Our exemplar large networks are modern neural networks that make highly accurate but overconfident predictions, while our small networks do not (Guo et al., 2017). Additionally, we consider the ResNet 34 architecture for TinyImageNet and the ResNet 18 architecture for ImageNet. In total, we train 574 ensembles across four datasets and six architectures, using the four mechanisms discussed above to encourage serouurag/dlscoourage /discourage ensemble diversity. (See Appx. B for training and implementation details.)

---

Published in Transactions on Machine Learning Research (01/2024)
RosNet Ensemble (M 4)
Weighted Training: CIFAR1O
Jessen Gap
Variince
ISD 1 vs All
JSD Uniforr
0.50
single madil
(.80
0.70
0.78
cusemble ( 0)
Arerary
x
0.75
6.79
0.70
0.75
0.74
0.65
3
-0.0
-2.
0.8
2.5
5.0
-2
-1.0
0.5
(predictive divenity pendlty)
7 (peedietive diversity pennlty)
(presictive diversity pecally))
7 (poedictive divenda penalty)
ResNet 18 Ensemble (M = 4) Weighned Training CIFARIO
Jecret. Gup
Vatiunee
JSD All
JSD Uriform
0.940
0,910
0,940
30
Aucuney
Accuncy
0.50
0.935
0.985
0.300
0.32
0.925
single muddd
mosmmbl (y 0)
0.31
0,000
0,080
5.0
2.5
0.0
2.5
-1.0
-0.5
0.0
1.0
(predictive divenity penalty)
(peedictive diversity penalty)
7 (predietivee diveritty penalty)
7
[preddctive diveriity penalit)
Figure 2: grroouraing//iscourgggg diversity has different effects on small vs. large neural network ensembles, We train deep ensembles of small (top row) and large (bottom row) neural networks (ResNet 8 and ResNet 18, respectively) with diversity mechanisms on CIFAR10 (one column per diversity mechanism). We compare standard ensemble training (dotted line) to deieursty--ncouurgedd /diiog/u///dcceeeeeeedd training (left vs. right of dotted line). Blue lines/bande are standard ensemble test accuracy; black lines bands are component model test accuracy. Encouraging diversity (7 < 0) improves test accuracy of seall-network ensembles while harming test accuracy of large-network ensembles. Conversely, discouraging diversity (r > 0) actively hurts the performance of small-network ensembles, but appears benign for large-network ensembles. (See Appx. F for CIFARLO0, TinylmageNet, and other architectures).
Regularization strength
y -09
y= - 00.5
Y y= 0
? = 0.1
Test accuracy (%)
70.16
71.10
71.49
71.37
Table Diversity encouragement fails on ImageNet.
Diversity encouragement results with ensembles of ResNet 18 models with M 3, and Jensen Gap regularizer. As with smaller datasets, encouraging predictive diversity has no benefit, while diversity discouragement appears benign.
4.1 Results
Within this section, we show results for one representative small and large architecture (ResNet 8 and ResNet 18), with most results on the CIFARIO dataset. See Appx. F for other architectures and datasets.
Encouraging diversity benefits seall-network ensembles, but harms large-network ensembles. Fig. (top, 7 < 0) rllust the effects of encouraging diversity on the accuracy of CIFAR10 ResNet 8 models. (See Appx. F for the other small architocture-LeNett as well as results on CIFAR100.) Each panel depicts a different diversity encouraging mechanism. The blue line and bands depict standard (unregularized) ensemble performance (mean + two standard error), while the black line and bands depict the error of single models. Relative to the standard ensemble bascline (blue band), cach diversity- performance improvements for a range of < 0 (left of baseline). Across all diversity mechanisms,, datasets, and model architectures, diversity encouragement, improves the performance of 73% of 86 small ensembles (see Appx. for the other small architecture LeNet as well as results on CIFAR100.)
mechanism yields
In stark contrast, Fig. 2 (bottom, 7 < 0) illustrates that diversity encouragement has the opposite effect ensembles of larger ResNet 18 models. For all mechanisms, datasets and model architectures, diversity
7

---

Published in Transactions on Machine Learning Research (01/2024)
CIFAR 10: ResNes8 Tensen Gop Regularierr
CIEAR 10; ReaNes8
CIFAR 10 ResNet18
CIFAR 10: RosNet18
ISD Unilow Regul arizer
Cap Regalaneer
Uniformn Regulorizer
Unmoctifed ensemble
performance d 24
206-11
236-00
0.3
Ang. mmgle trockl
306-81
1123
0220
Liaei
4000
1000
10xt 1.00
Jescen gap pprd. divesiit))
C22 co 3E3 C3B
CE lessca gap [pred divassts)
300
1.44
C3D Ansaca zAp (pood divenily)
CE Jasass pap (poo), disasis)
Y
Figure 3: Trading off predictive diversity and component model performance in diversity regularized ensembles. Each marker represents ResNet 8 (left panels) or ResNet 18 (right panels) ensemble trained with a diversity intervention on CIFAR10 (see Fig. 12 for CIFAR100). Warmer colors correspond to positive 7 values (encouraging diversity), cooler colors correspond to negative 7 values (discouraging diversity). Axes are given by ensemble loss decomposition as in Fig. 1. The level set of standard deep ensemble performance (7 = 0) denoted by and the dotted diagonal line. In all ensembles, encouraging/ discouraging predictive diversity leads to proportionally higher /lower diversity on test data. Small-network ensembles with diversity encouragement (7 < O, blue markers), can achieve higher single model performance, and thus better ensembles (below dotted line). For large-network ensembles however, diversity encouragement comes at high cost to average single model performance, and worse ensembles overall (above dotted line). For small-network ensembles, discouraging predictive diversity (r > 0; red markers) also leads to worse performance of component models: thus corresponding ensemble performance is also worse (above dolted line) than standard ensembling. In contrast, diversity discouraged large-net ensembles can outperform standard training (below the dotted line). Among large-network ensembles, the one with the best test NLL is one where diversity was discouraged. In Fig. 13, we also study the relationship between diversity and average accuracy of these ensembles.
encouragement (7 < 0) yields worse performance than a standard ensemble in 61% of 134 large networks (at times undershooting single networks), and no statistically significant changes to an additional 29%. These results also hold on large datasets: training ensembles of ResNet 18 the full ImageNet dataset (Table 1), find that diversity encouragement follows the same trends architectures VGG, WideResNet, and DenseNet-a as well as results on CIFAR100 and TinyImageNel.)
CIFAR10. (See Appx. F for other large
We next verify that training with diversity encouragement leads to higher diversity on test data. Fig. 3 depicts the performance of ensembles formed from ResNet 8 models (left pancls) and ResNet 18 models (right, panels), using the decomposition shown in Fig. 1. We show results on the CIFAR10 dataset for the Jensen Gap and JSD Uniform regularizer. Relative to standard ensemble training (indicated with x), ensembles trained with more diversity encouragement (y O, blue dots) express proportionally more predictive diversity (further right). Factoring in the average performance of component models provides an alternative quantification of the same trade-off we observe in Fig. 2. While ensembles of small models can benefit from diversity encouragement, (Fig. 3, blue points below the dotted line), for large-network ensembles the effect of encouraging predictive diversity is higher values of the ensemble loss (blue points above dotted line). Tn Appx. F.2 we show additional loss decompositions for ResNet 18 diversity regularized ensembles on CIFAR100.
Discouraging predictive diversity is benign for large-network ensembles, but harmful for small- network ensembles. For the small-network ResNet 8 ensembles (Fig. 2, top, Y 0), we observe that all diversity discouraging mechanisms harm test set error, consistent with our intuitions from traditional ensembles. Across 82 smalll-neworkk experiments (here in Fig. 2 and in Appx. F), diversity discouragement hurt performance 56% of the time, whereas performance remains intact in 40%, and helps for only 4%. In contrast Fig. (bottom, 7 > 0) shows that penalizing diversity does not harm the largo-network ResNet ensembles. Across 149 experiments (depicted in Fig. and Appx. F), 74% of largo-network ensembles obtain similar or better performance with diversity discouragement (again, Appx. F for additional results.)
8

---

Published in Transactions on Machine Learning Research (01/2024)
Once again, verify that discouraging predictive diversity (7 0, red dots), produces ensembles that are less diverse on test data in Fig. 3. In line with the results on predictive accuracy, small model ensembles demonstrate strictly worse ensemble performance when discouraging predictive diversity (red points above the dotted line). For large-network ensembles however, diversity discouragement can sometimes lead to improved ensemble performance (red dots below dotted line). Appx. F.2 shows additional loss decompositions for ResNet 18 diversity regularized ensembles on CIFAR100.
Similar trends hold for probabilistic performance metrics and calibration. In Appx. F.5 we replicate Fig. 2 for the negative log likelihood (NLL), Brier Score (BS), and expected calibration error (ECE) (Nacini et al., 2015) metrics mcasured on the test set. We observe similar trends: encouraging diversity helps/hurts muall/large-networr ensembles, and vice versa for diversity discouragement.
Similar trends hold for out of distribution accuracy, In Appx. D we apply the ensembles from Fig. 2 to CIFAR10.1 (Recht ct al., 2018), an out of distribution (OOD) test datasct for CIFAR10. Our results show that OOD performance follows the same trends as InD data.
Similar trends hold for MLPs tabular datasets. Finally, show that our diversity regularization results generalize to other datasets and network architectures: namely the application of MLPs to tabular datasets (see Appx. B.2.2 for implementation details, and Appx. D.1 for results). Interestingly, in this setting we also discovered models of intermediate capacity that appear to interpolate between the results for the large vs. small model ensembles we consider here.
4.2 Analysis
Having verified that the diversity mechanisms studied above successfully eucourage/disccoraaee test set predictive diversity, we must conclude that small vs. large neural network ensembles trade off component model performance in different ways (helping vs. harming ensemble performance, respectively). At a high level, we claim that encouraging predictive diversity at the cost of component model performance harms lerger deep ensembles because most of their predictions are confident and correct. In particular, encouraging diversity during training is unlikely to impact only erroneous predictions at test time, as diversity encouragement techniques would then constitute oracle error detectors. Unfortunately, increased predictive diversity is strictly harmful for any correct and confident output, as ensemble risk is only minimized if all models produce the same correct/confientt output. This pathology does not affect small-network ensembles, /oanncoouuunnoofideennn unconfident predictions that can be improved through decorrelation. We provide evidence for these clains below.
which tend to produce a greater proportion of erroncous/
Encouraging diversity during training impacts all predictions, even correct ones. Fig. 4 studies if diversity encouragement is more likely to impact prodictions that would otherwise be correct or incorrect. We first measure the diversity encouraged on each datapoint in the test set using ensembles trained with values of > S O. We plot the resulting distribution of diversity values across the CIFARIO test set for ResNet 8 (left) and ResNet 18 (center) ensembles trained with Jensen-gap diversity encouragement (top row), with different lines representing densities for different values of y. Relative to standard (7 0) training, for ensembles trained with diversity encouragement (7 < 0), there is more data in the rightmostt portion of the distribution, demonstrating the impact of diversity encouragement on individual predictions. For each of these predictions, we then consider the predictive accuracy of a standard (7 = 0) ensemble, which we call the counterfactual accuracy (bottom row)- i.., the accuracy of that prediction in the absence of any diversity encouragement. Tdeally, we would want ensemble predictions where diversity encouraged to have low counterfactual accuracy (i.e. we are diversifying predictions which would otherwise result in errors), but this is not what we see in practice. For ResNet 8 (left) and ResNet 18 (center) ensembles trained with Jensen-gap diversity encouragement, we observe that for interventions which incur greater levels of predictive diversity (lighter colored lines), greater portion of diverse predictions (further to the right) are counterfactually correct. In other words, most datapoints where the ensemble expresses increased diversity would alveady have been classiffed correctly in the absence of diversity interventions. If diversity encouragement only decorrelated errors, we would expect the opposite: higher values of counterfactual ensemble accuracy at low values of predictive diversity, and lower values on data that expresses more diversity relative to standard ensembling. It is worth noting that this trend holds for both small- and large-network deep ensembles. Moreover, this
9

---

Published in Transactions on Machine Learning Research (01/2024)
Jensen Gap (CIFAR1O, ResNet 8)
Jensen Gap (CIFAR10, ResNet 18)
Drestty
11F
-0.
1.0
Drastry
-36
-111
33
Standard Enczmbling
11,001
11P
-34
Diversit Encouraged
Diversity Discouraged
18
110
[TOOI
Chuntefictud
Accuanng
Conntefottall
Accuricy
1o.1.00
Base
JEAL11
[0.10]
[BC.1] (8/ [RA/]
0.30
2002
11.05
022
0.4
06
0.5
Per datopoint Diversity (CE Jensen Gop)
Per-dicapoial Liversily (CR Jeneen Gop)
Figure 4: Diversity encouragement hurts confident/accurate classifter ensembles. (Left, center:) we study the per-datapoint impact of diversity encouragement via counterfactual accuracy: the per-datapoint accuracy given by standard (7 = 0) ensemble predictions (see Sec. 4.2 for details). Evaluating an ensemble trained with diversity encouragement (Eq. (8)) produces distribution of ensemble diversity over test set predictions (top row, one line per value). Across this distribution, we then measure the counterfactual accuracy (bottom row), and focus on data in the right tail, which are the most strongly influenced by diversity encouragement techniques. Unfortunately, we see that predictions which are most influenced by diversity encouragement have high counterfactall accuracy: i.e., if we were to test a standard ensemble on these datapoints, it would have been correct anyway. This finding holds for small (ResNet 8, left) and large-network ensembles (ResNet 18, center) (for other diversity-encouuagiin mechanisms, see Appx. 1.) (Right:) Decorrelating correct/cone confident component model predictions yields worse ensemble performance. Correct/confiden component model predictions (x) concentrate in vertex of the probability simplex (top pancl). Due to simplex geometry, encouraging diversity (left pancl) necessarily degrades the the ensemble prediction (o), potentially altering the class prediction. In contrast, diversity discouragement (right panel) need not hurt ensemble predictions.
phenomenon gencralizes beyond the diversity-cnncourgiing mechanisms studied in this section. In Appx. I we demonstrate similar trend in random forests, suggesting that even traditional ensembling techniques like feature-subsetting impact otherwise correct predictions.
Predictive diversity detrimental for otherwise correct/ /confident predictions. The above results has significant implications when ensemble members already make highly accurate and confident. predictions, as large classification neural networks often do (Guo et al., 2017). In this case, component model predictions will all concentrate in the same vertex of the probability simplex (sce Fig. 4, "Standard Ensembling") and it is impossible to increase predictive diversity without harming the ensemble risk in Eq. (3). Increasing diversity would necessarily require at least one component model to become less confident, reducing the confidence of (and potentially changing) the ensemble prediction (see Fig. 4 right for an illustration).
These claims describe a mechanism to relate observed differences in the impact of diversity encouragement directly to component model capacity. Small neural networks, decision trees, and other low-capacity models produce fewer confident/ correct predictions where predictive diversity can only harm component model performance. For the cooonfident/erroneoon predictions that remain, more diversity will not necessarily harm component model performance, leading to more favorable trade-off. As first test of these claims, we would predict that regression ensembles of any size are also immune to this pathology because the output space is unbounded, so underestimated predictions from one component model can cancel out overestimale les from another. In Appx. B.2.3, we indeed confirm that large regression deep ensembles can benefit from diversity encouragement, unlike their classification counterparts. Analogously, models trained with loss that discourages confident predictions such as label smoothing loss (Szegedy et al., 2016) are less susceptible to this pathology, lending additional support for these claims (Appx. D.4).
10

---

Published in Transactions on Machine Learning Research (01/2024)
Why is discouraging predictive diversity beneficial? While the previous claims support our diversity- encouragement results, they do not explain why discouraging diversity in large-network deep ensembles is benign (and occasionally beneficial). Here we offer one possible hypothesis for our observed results. Recent theoretical evidence suggests that increasing the size of overparameterized models amounts to variance reduction (Neal al., 2018; Adlam and Pennington, 2020; d'Ascoli et al., 2020) a result which is supported empirically by Fig. 6 and other work on ensemble distillation (Lan et al., 2018; Zhao et al., 2022). Discouraging diversity through the use of regularizer could thus be an alternative path towards variance reduction that mimics the effects of larger models. Confirming this hypothesis is an open problem we leave for future work.
Relation to existing work. Importantly, as noted in Sec. 2 and Appx. C we find no evidence in the extensive literature on ensemble diversity mechanisms that contradicts our claims. In particular, across many different mechanisms designed to increase predictive diversity, significant gains to ensemble performance are only reported when applied to ensembles small networks, or when using other sourees of training data, and not when training deep ensembles composed of large neural networks used in practice.
5 The best deep ensembles express low predictive diversity
Our results thus far consider methods where there is an explicit cost to increasing ensemble diversity: trading off the performance of component models. There are however sources of ensemble diversity which do not have such a cost, such as ensembling over helerogeneous architectures and training methods. In such cases it is possible to construct diverse ensemble members "for free" by considering component models with different training/arciin architccctuss but similar performance. Even in such cases, we demonstrate that there is opportunity cost associated with selecting ensemble members to increase such "free" diversity. While there is substantial predictive diversity amongst cow-performance architectures (e.g. AlexNet, VGG 11, etc.) perfor-performance architectures (e.g. ResNet 152, DenseNet 161, etc.) tend to produce very similar predictions across datapoints. In other words, ensembling heterogencous architectures offers diminishing diversity as we consider ensembles of larger component neural networks. This finding is confirmed by prior work, which demonstrates that ensembling diverse architectures/ Sttittectuures/rinnnng trrining procedure yields only marginal gains in ensemble accuracy (Gontijo-Lopes et al., 2022). Alternatively, assuming access to heterogencous pool of trained models, could instead construct ensembles that simply use the best component models. Even though the best component models make nearly identical predictions, these ensembles offer far superior performance to those which maximize predictive diversity. These results are a complement to those in Sec. 4: when attempting to form the best ensemble from a pool of candidate models, we find that the best strategy is to trade-off predictive diversity favor of better component. model performance, inverting the standard intuition. (Crucially, we show that this result also distinguishes deep ensembles from traditional small-model ensembles, which are best when the component models are poor but extremely diverse.)
Experimental setup. Drawing from a pool of 31 CIFAR10 and 92 ImageNet architectures, we construct heterogeneous ensembles of M networks in a combinatorial fashion. We randomly select four independently- trained models that were in the same performance quantile. We additionally construct homogeneous ensembles consisting of M = 4 independenttly-rrinet copies of the same ceaiittectuueetrainnii procedure. In total, for we construct 1050/427 heterogeneous and 22/3 homogeneous ensembles. Crucially, the models we consider represents a broad swath of existing architectures, including the standard models in Taori et al. (2020). (see Appx. B for cehhitcctural/trinniii
details). We measure the performance of these ensembles on in-distribution (InD) test sets as well as the CINIC10 (Darlow et al., 2018) and ImageNet-C (Hendrycks and Dietterich, 2019) out-of-distribbution (OOD) datasets.
5.1 Results
Fig. shows ensemble performance (test-set cross entropy) as a function of average component models" performance. By the decomposition in Eq. (3), the difference between these quantitics is the Jenson gap mcasure of predictive diversity. Models further below the identity line are more diverse, while those closer have little predictive diversity. (See Appx. G for more 0OD datasets.)
11

---

Published in Transactions on Machine Learning Research (01/2024)
CIFAR10
CINIC1O
ImageNet
ImngoNet-CC Gaussian 1
Ensemble CB
Ensemble CE
Ensembble CE
Easemble CE
0.30
1.0
5.0
0.15
0.5
2,5
0.00
04 Avg. model CE
0.0
1.6
0.0
0.0
25
5.0
Avg. model CE
Avg. model CE
Avg. model CE
Figure 5: The opportunity cost of predictive diversity. Plots depict the ensemble test-set performance versus the performance of its average component model (as measured by cross entropy). Each marker an CIFARIO/CINIC10 CINIC10 and ImageNet/ SraageNet/ImageNettae /ImageNet-C.. Dotted boundaries indicate inset region, provided for detail. Ensembles are color coded as homogeneous (blue) heterogeneous (orange). Controlling for component model performance (vertical slices), heterogeneous ensembles are more diverse than homogeneous ones (further from the identity linc). However, heterogencous ensembles afford diminishing amounts of predictive diversity (closer to identity line) when ensembling higher performance component models (further left). The best performing ensembles (furthest down) have the best component models (furthest left), and very little predictive diversity.
ensemble of models evaluated on the InD/OOD datasets of CIFAR10/
From this plot we observe that conditioning on average component model performance (graphically: consider any vertical slice of Fig. 5), heterogeneous ensembles generally obtaín better (lower) performance than homo- gencous ensembles. This result suggests that the architectural/trainite ensembles is generally beneficial (as expected, since there is no cost to component model performance), confirming the findings of prior work (Mustafa et al., 2020; Gontijo-Lopes et al., 2022; Agostinelli et al., 2022). At the same time, among all heterogeneous ensembles, the best performing ensembles are not the most diverse (i.c., in the bottom right hand corners of Fig. panels). In contrast, the best ensembles are most often formed from the highest, performance component models (in the bottom left hand corner of Fig. 5), even as they become less diverse (approach identity line). Thus, the diversity gained through heterogencous architectures/trrine powerful component models, which perform beller despite having much lower predictive diversily. Although predictive diversity is not directly harmful to ensemble performance in this case, these results show there is an opportunity cost to forming ensembles to maximize predictive diversity. Counter to classic ensemble intuitions, the best deep ensembles on both InD and OOD datasets in Fig. 5 (the ensembles with the highest average component model performance) are formed by trading off predictive diversity order to select higher performance component models.
diversity afforded by heterogencous
procedures has diminishing returns
consider ensembles of more
5.2 Analysis
Next study if the opportunity cost observe in Fig. 5 is also function of component model capacity. Comparing the behavior of deep ensembles with random forests and bagged random feature models (Fig. 6), we observe how component model performance vs. predictive diversity contribute to ensemble performance as function of model size. For random forests of various depth (Fig. 6 left), we plot the average component model performance (test-set cross entropy) versus the ensemble's predictive diversity (Jensen gap). Larger dots correspond to deeper random forests. As in Fig. 1, ensemble performance given by the diagonal level sets. In direct contrast to deep ensembles, the best random forest (lowest diagonal level sct) is the most diverse (furthest right).
We next study ensembles of random feature classifiers trained on MNIST. In this setting we can map small vs. large size networks explicitly onto a notion of model capacity: to underparametrized (incapable of fitting training data perfectly) or overparameterized regimes (achieving 100% training accuracy), depending on the classifier width (i.e. the number of random features). In Fig. (center left), we again plot component model performance versus diversity, with diagonal level sets depicting ensemble performance. As we incrcase the width of the component classifiers (graphically depicted by dot size) we identify over and underparametrized regimes by looking at only average single model performance (Fig. 6 center left, y-axis) as function of
12

---

Published in Transactions on Machine Learning Research (01/2024)
Avg. single model CE
MNIST
MNIST
CIFARIO Deep Ensembles (Width)
CIFAR1O Deep Ensembles (Depth)
Random Forests
RFF Bagging
0.5
0.39
0.5
0.4
0.38
0.3
0.37
0.3
0.2
0.3
0.2
0.04
0.06
0.08
0.36
0.16
0.18
CE Jensen gap (pred. diversity)
Figure 6: Small vs. large ensembles improve performance through different strategies. Ensembles of decision trees (Ieft), random feature classifiers (center left), and neural networks (center right/right Dotted diagonal line denote the best ensemble cross entropy achieved within a given model class (lower right is better). Dot size corresponds to depth/wiee /width of the component models. (Left): Random forests improve with increasing diversity (obtained by using deeper component trocs). (Center Left): Random feature ensembles monotonically improve with classifier width. When component models are underparameterrized (smaller dots), ensemble improvement corresponds to increased predictive diversity. When component models are overparameterized (larger dots), ensemble improvement corresponds to decreased predictive diversity. (Center Right/Right) Right):: Deep ensembles monotonically improve with width/depth After certain size, this improvement is entirely altributed to belter component models with less predictive diversity.
model size (size of markers). As model size increases, average single model performance demonstrates "double descent" (Belkin et al., 2019) with the peak in performance indicating the interpolation threshold. Strikingly, we see that ensemble diversity also appears to change behavior at the interpolation threshold, increasing as a function of model size in the underparametrized regime, and decreasing the overparametrized regime. While increasing width always improves ensemble performance, this improvement is due to increased diversity in the underparameterized regime, trading off component model performance to achieve this increased diversity, see in ensembles of decision trees (Fig. 6 center left; small dots). Meanwhile, it is due to better component classifiers in the overparameterized regime, which trade off predictive diversity for better component models.
We replicate this finding in ensembles of ResNets by varying component model width (Fig. 6 center right) and depth (Fig. right) as proxies for model capacity (see Appx. B.4 for details). Again, increasing width and depth (graphically represented by dot size) yields better ensembles. Ensemble improvement and diversity are initially correlated when networks models are small, yet they become anticorrelated after the component networks grow sufficiently large. We expect that this transition roughly corresponds to under overparametrized ResNet architectures, but the presence of regularization in the course of training prevents a conclusive identification of this transition point. These results demonstrate that the diminishing returns of encouraging predictive diversity are not specific to modern deep ensembles, but rather a general phenomenon in ensembles large (likely overparametrized) models, which operate in a regime where trading off predictive diversity in favor of component model performance leads better ensembles.
6 Discussion
We demonstrate that encouraging or selecting for predictive diversity when forming deep ensembles (or more generally, ensembles of high-capacity classifiers) is suboptimal and potentially detrimental strategy. This finding holds due to explicit trade-offs or opportunity costs relating predictive diversity to component model performance. We contrast this behavior to low-capacity neural network ensembles (and regression ensembles), where we observe that seeking increased predictive diversity yields a net benefit to ensemble performance, indicating that our findings are thus entirely consistent with the traditional ensemble literature.
At a high level, our findings demonstrate how valuable intuitions from traditional ensembling settings must be qualified by notion of component. model capacity. Other work has identified the capacity of component.
13

---

Published in Transactions on Machine Learning Research (01/2024)
models as important feature of deep ensemble. For example, Kondratyuk et al. (2020) and Lobacheva et al. (2020) study the question of when ensemble is more efficient than a single model of matched parameter count, and Kobayashi et al. (2021) finds that the performance of a deep ensemble degrades to that of a single model as we ensemble models of increasing width. Theisen et al. (2023) provides an alternative characterization of the relationship between predictive diversity and average component. model performance in ensembles. Although they do not study the impact of encouraging/ /sockriinn// predictive diversity in ensemble models, they observe that ensembling becomes less effective when considering models beyond the interpolation threshold, in line with some of our results in Sec. 5. Likewise, theory suggests that many of the benefits of ensembling may simply recapitulate the benefits of increasing width of component models (Adlam and Pennington, 2020; Geiger et al., 2020; Bernstein et al., 2021).
An important goal for further study is to describe the relationship between ensemble member capacity, ensemble member performance, and the phenomena that we observe here in greater detail. Based on our results in Sec. 4, we expect that the difference between regimes where diversity encouragement improves or degrades ensemble performance is due to the proportion and quality (correct vs. incorrect) of predictions that are made with high confidence. While both the performance of individual models and model capacity and has been correlated with high confidence predictions in previous rescarch (c.g. Guo et al., 2017; Nakkiran et al., 2021; Bernstein et al., 2021), it would be useful to know if one (or both) of these factors can predict whether diversity encouragement should help or hurt ensemble performance. We might hope to disentangle these factors by performing diversity encouragement with unregularized models varying capacity (inspired by Fig. 6 for RFF models) to better characterize the transition between helpful and harmful diversity encouragement across different datasets and regularizers. More broadly our results may be related to observations made in Breiman (1996), which already identify the "stability" of learning model's predictions (with regard to training data) as critical feature which determines the success of bagging; unstable predictors can benefit from bagging, but those that generate stable predictions across different training sets will not improve. It would be an interesting point of future work to consider our results within the framework of model stability relative to the various sources of predictive diversity that we consider.
Altogether, our results along with this prior work suggest that as we use larger component models in an ensemble, should naturally expect the diversity within that ensemble decrease. Moreover, results imply that ongoing efforts to introduce additional sources of predictive diversity into deep ensembles (through joint training mechanisms, heterogeneous architectures, and similar) will not change this fact, unless they are able to overcome several fundamental challenges, such as encouraging predictive diversity only upon counteractuull errors, and identifying "free" " sources of diversity which offer a greater advantage than using more powerful, less diverse models. While ensembling deep neural networks is undoubtedly a practically useful technique to improve performance of neural network models current use, we expect that meaningful sources of predictive diversity will become even harder to come by as neural network architectures and training paradigms contimue to improve.
Broader Impact Statement
One conclusion of our work is that creating ensembles from more powerful models is better strategy than attempting to find diverse sets of existing models. Thus, potential negative outcome of our work could be to further encourage the expenditure of computational resources by training new, larger scale component models instead of reusing existing ones.
At the same time, our work finds fow benefits to the use of joint training strategics to create more diverse large capacity deep ensembles, which are compulationally expensive in and of themselves. Our hope would be that this conclusion may help to mitigate some of the current computational costs of creating new ensemble models.
Acknowledgements
We thank John Miller for sharing models trained on CIFAR10, as well as Taori et al. (2020) for making their trained ImageNet models and code open sourced and easy to use.
14

---

Published in Transactions on Machine Learning Research (01/2024)
We would also like to thank Rich Zemel for his thoughtful suggestions, well as Yixin Wang, Yaniv Yacoby, Zelda Mariet, Dustin Tran and Jeremy Bernstein for their insightful comments, and the Zuckerman Mind Brain Behavior Institute and the Center for Theoretical Neuroscience for additional support.
TA is supported by NIH training grant 2T32NS064929-11 EKB supported by NIH 3T32N8064929- 13 and NIH U19NS107613. JPC is supported by the Simons Foundation, McKnight Foundation, and Grossman Center for the Statistics of Mind. TA, EKB, and JPC are supported by NSF 1707398 Gatsby Charitable Trust GAT3708.
References
Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Liu, Mohammad Ghavanzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, Rajendra Acharya, al. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.
Taiga Abe, E. Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. The best deep ensembles sacrifice predictive diversity. In 7 Can't Believe It's Not Belter Workshop: Understanding Deep Learning Through Empirical Falsifieation, 2022a. URL https://openrevieet http://pppeeereeeeeeecccooorraaa
Taiga Abe, E Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? In Aduances in Neural Information Processing Systems, 2022b.
Ben Adlam and Jeffrey Pennington. Understanding double descent requires fino-grained bias-variance decomposition. Advances in neural informalion processing systems, 33:11022 11032,, 2020.
Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability metrics for selecting source model ensembles. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pottern Recognition, pages 7936-7946, 2022.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXi::2002.66/ 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machino-lcarning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849-15854, 2019.
Jeremy Bernstein, Alex Farhang, and Yisong Yue. Kernel interpolation as bayes point machine. arXie, 2021.
Jock Blackard. Covertype. UCI Machine Learning Repository, 1998. DOI: http:////bbo.org/1243//////////
Leo Breiman. Bagging predictors. Machine learning, 24(2):123 140, 1996.
Leo Breiman. Random forests. Machine learning, 45(1):5-32) 2001.
E Kelly Buchanan, Michael W Dusenberry, Jie Ren, Kevin Patrick Murphy, Balaji Takshminarayanan, and Dustin Tran. Reliability benchmarks for image segmentation. In NeurIPS 2022 Workshop Distribution Shifls: Connecting Methods und Applications, 2022.
Sebastian Buschjäger, Lukas Pfahler, and Katharina Morik. Generalized negative correlation learning for deep ensembling. arXio preprint arXim:2011.0000 2020.
Adele Cutler and Guohua Zhao, Pert-perfect random tree ensembles. Computing Science and Stotistics, 33 (4):90-4, 2001.
Ali Dabouei, Sobhan Soleymani, Fariborz Taherkhani, Jeremy Dawson, and Nasser M Nasrabadi. Exploiting joint robustness to adversarial perturbations. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 1122 1131, 2020.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Francesco D' D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Newral Information Processing Systems, 34:3451-3465, 2021.
Luke N Darlow, Elliot J Crowley Antreas Antoniou, and Amos J Storkey, CINIC-10 is not ImageNet CIFAR-10. arXiv preprint arXiv:1810.08 03505, 2018.
Ahmet Demirkaya, Jiasi Chen, and Samet Oymak. Exploring the role of loss functions in multiclass classification. In 2020 54th annual conference on information sciences and systems (ciss), pages 1 5. IEEE, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A largo-scale hierarchical image database. In Computer Vision and Pattern Recognition, pages 248 255, 2009.
Thomas Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting and randomization. Machine leurning, 32:1 22, 1998.
Thomas G Dietterich. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, pages 1-15, 2000.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' A'mour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16458 16468, 2021.
Stéphane d' d'Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, 2020.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. Xiviv preprint 1rxx::1112 02757, 2019.
Yoav Freund. Boosting weak learning algorithm by majority. Information and computation, 121 1995.
Xavier Gastaldi. Shake-shake regularization. orXiv preprint arXiv:1705.07455 2017.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020.
Chengyue Gong, Lemeng Wu, and Qiang Liu. How to fill the optimum set? population gradient descent with harmless diversity. arXiv preprint wxXiv:2202.08977 2022.
Raphael Gontijo-Lopes, Yann Dauphin, and Ekin Dogus Cubuk. No one representation to rule them all: Overlapping features of training methods. In International Conference on Learning Representations, 2022. URL / ntt/
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 95:24991-25004 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
Neha Gupla, Jamie Smith, Ben Adlam, and Zelda E Mariet. Ensembles of classifiers: bias-variance perspective. Transactions on Machêne Learning Research, 2022. URL http:://pppnneeeeeeeeeeftoroot id=1I0QFVncY9.
Fredrik K. Gustafsson, Martin Danelljan, and Thomas B Schon. Evaluating scalable bayesian deep learning methods for robust computer vision. In Computer Vision and Pattern Recognition Workshops, pages 2020.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proceedings of the IEFE conference computer vision and paltern recognition, pages 5927-5935, 2017.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Lars Kai Hansen and Peter Salamon. Neural network ensembles. Transactions pattern analysis and machine intelligence, 12(10):993-100 1001, 1990.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent. subnetworks for robust prediction. Tn International Conference Learning Representations, 2021. URL Sctps://openreviee iepeerreviee.net/fouumiiii DGg9XnKXFAH.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
Dan Hendrycks and Thomas G Dietterich. Benchmnrkingg neural network robustness to common corruptions and surface variations. In International Conference on Learning Representations, 2019.
Jie Hu, Li Shen, and Gang Sun. Squeeve-and-excitatiio networks. In Proceedlings of the IEEE conference on computer vision and pattern recognition, pages
2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700- 4708, 2017.
Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss cross-entropy in classification lasks. arv preprinl 0et:2000607322, 2020.
Siddhartha Jain, Ge Liu, Jonas Mueller, and David Gifford. Maximizing overall diversity for improved uncertainty estimates in deep ensembles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4264-4271, 2020.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint training of deep ensembles fails due learner collusion. arXiv preprint rrXi::230111123, 2023.
Sanjay Kariyappa and Moinuddin K Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint 1rXiv:190..09911, 2019.
Seijin Kobayashi, Johannes von Oswald, and Benjamin F Grewe. On the reversed bias-variance tradeoff in deep ensembles. ICML, 2021.
Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models. arXiv preprint arXiv:2005.00570, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of fcatures from tiny images, 2009.
Ludmila Kuncheva and Christopher J Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine Learning, 51()):181-207, 2003.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
Xu Lan, Xiatian Zhu, and Shaogang Gong, Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems, 31, 2018.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient-based Icarning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Daniel Lee and H Sebastian Scung. Algorithms for non-negative matrix factorization. Advances in neural informalion processing systems, 13, 2000.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXio preprint 1x11:511.0069 2015.
17

---

Published in Transactions on Machine Learning Research (01/2024)
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecifted data. arxiv preprinl arXi::2202 034 18, 2022.
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry P Vetrov. On power laws in deep ensembles. In Aduances in Newral Information Processing Systems, 2020.
John P Miller, Rohan Taori, Aditi Raghumathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, 2021.
Aaron Mishtal and Itamar Arel. Jensen-shannon divergence in ensembles of concurrently-trained neural networks. In 2012 11th International Conference Machine Learning and Applications, volume 2, pages 558-562. IEEE, 2012.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? Advonces in neural information processing systems, 32, 2019.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. arXiv preprint arXiv:2010.06800 2020.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In AAAI Conference on Artificial Intelligence, 2015.
Prectum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021 (12):124003, 2021.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste-Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradcoff in neural networks. arXiv preprint ar Xrv: 1atp:8110000000 810.08591, 2018.
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't Believe It's Not Beller!" NewrIPS 2020 workshop. openreview.nel December 2020.
Michael Opitz, Horst Possegger, and Horst Bischof. Efficient model averaging for deep neural networks. In Computer Vision-ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, pages 205 220. Springer, 2017.
Luis A. Ortega, Rafacl Cabañas, and Andres Mascgosa. Diversity and generalization in ncural network ensembles. In International Conference on Artificial Intelligence and Statistics, pages 11720 11743. PMLR, 2022.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 2019.
Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transferability. Xr preprinl ar #XXiv:222.04414, 2022.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robust via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970-4979. PMLR, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapcau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Michael P Perrone and Leon N Cooper. When networks disagree: Ensemble methods for hybrid neural networks, 1992.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Alexandre Rame and Matthieu Cord. Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation. arXiv preprint arXiv:2101.0620 01.0554,, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize cifar-10? arXiv preprint arXivv1806.00433 00451, 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt,, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning, 2019.
Andrew Ross, Weiwei Pan, Leo Celi, and Finale Doshi-Velez. Ensembles of locally independent prediction models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5527-5536, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint orXit:1409.15550 2014.
Samarth Sinha, Homanga Bharadhwaj, Anirudh Goyal, Hugo Larochelle, Animesh Garg, and Florian Shkurti. Dibs: Diversity inducing information bottleneck in model ensembles, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9666-9674, 2021.
Lawrence Stewart, Francis Bach, Quentin Berthet, and Jcan-Philiipp Vert. Regression classification: Influence of task formulation on neural network features. arXiv preprinl arXie:2211 10n:2211.05611, 2022.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. Tn Computer Vision and Pottern Recognition, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Advances in Neural Information Processing Systems, 2020.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton Van den Hengel. Evading the simplicity bias: Training diverse set of models discovers solutions with superior ood generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16761-16772, 2022.
Sujay Thakur, Cooper Lorsung, Yaniv Yacoby, Finale Doshi- Velez, and Weiwei Pan. Uncertainty-aware (una) bases for bayesian regression using multi-headed auxiliary networks. arXiv preprint arXin:2006.11655, 2020.
Ryan Theisen, Hyunsuk Kim, Yaoqing Yang, Liam Hodgkinson, and Michael W Mahoney. When are ensembles really effective? arXiv preprinl Xrx::22305. 12313, 2023.
Alexandru Tifrea, Eric Stavarache, and Panny Yang. Semi-supervised novelty detection using ensembles with regularized disagreement. In Uncertainty in Artificial Intelligence, pages 1939- 1948, PMLR, 2022.
Dustin Tran, Jeremiah Liu, Du Phan Michael W Dusenberry, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, Neil Band, Tim GJ Rudner, Karan Singhal, Zachary Nado, Joost van Amersfoort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXiv:2207.004111 2022.
Mehmet Ozgur Turkoglu, Alexander Becker, Hüseyin Anil Gündüz, Mina Rezaei, Bernd Bischl, Rodrigo Caye Daudt, Stefano D' Aronco, Jan Wegner, and Konrad Schindler. Film-ensemble: Probabilistic deep learning via feature-wise linear modulation. Advances in neural information processing systems, 35:22229 22224,, 2022.
19

---

Published in Transactions on Machine Learning Research (01/2024)
Andrew Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan Iliescu, Mikel Lujan, and Gavin Brown. To ensemble not ensemble: When does end-to-end training fail? Tn Joint European Conference Machine Learning and Knowledge Discovery in Databases, pages 109 123. Springer, 2020.
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchFnsemble: alternative approach to efficient. ensemble and lifelong learning. In International Conference Learning Representations, 2020.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems, 33:6514-6522 6527, 2020.
Danny Wood, Tingting Mu, Andrew Webb, Henry Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. Xiviv preprinl arXiv:2301 03962, 2023.
Yanzhao Wu, Ling Liu, Zhongwei Xie, Ka-Ho Chow, and Wenqi Wei. Boosting ensemble accuracy by revisiting ensemble diversity metrics. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pages 10469-16477 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 1492 1500, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXie:1605.07143 2016.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong Hc, Jonas Mueller, R Manmatha, et al. Resnest: Split-atlention networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pages 2736-2746, 2022.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez- Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.00 09111,, 2017.
Shuai Zhao, Liguang Zhou, Wenxiao Wang, Deng Cai, Tin Lun Lam, and Yangsheng Xu. Towards better accuracy-offiieney tradc-offs: Divide and co-training. IEEE Transactions Image Processing, 2022.
Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
20

---

Published in Transactions on Machine Learning Research (01/2024)
A Computational resources
The large majority of our experiments were performed on cloud based computational resources (AWS and GCP). Ensembles trained with diversity regularizers in Figs. 2, 3, 6, 12, 14, 15 and 17 to 22 were trained with EC2 instances from the P3 family. We used p3. xlarge instances for model training, unless we experienced issues with GPU memory. In these cases we trained models on p3. .25lage instances. We estimate that on average, large model training lor CIFARIO ensembles required 3 hours of compute time, for CIFAR100 ensembles required 5 hours of compute time, and for TinylmageNet ensembles required hours of compute time. We estimate that small model training required 30 minutes per ensemble for all datasets. Collectively, therefore estimate that these experiments required 1250 compute hours.
Separately, ensembles trained with diversity regularizers in Fig. 8 and Table 1 were trained on 4 x NVIDIA V100 compute instances. We estimate that these experiments required around 700 additional compute hours. Analysis of experimental results and visualization were performed on M1 MacBook Airs.
B Model training
B.1
Code availability.
The code used to train deep ensembles is provided here: attentionn/reeedkkl Additional code for data analysis, as well as training non-deep neural network ensembles located here: https://github. dicated, ensemble training requires a GPU. All model training for ensembles with diversity mecha- nisms managed through a single run.py script, which takes configuration parameters from the file ensemble, attention/eeet wwwoo///oooon/uunnnoooooooooooee default .gg.. Additional parameters, such learning rate sched- haaeeiiggusafuuuttt....m
seglisstig/inter_eeeeeeeee
Unless otherwise in-
ule, number of ensemble members, evaluation out of distribution data can be managed through this configuration file.
Classification ensembles with diversity mechanisms can be trained by running the command:
python ensemble ewwemble_attentttneeee awtpentiii/sccrppptttttmasammaaa py #clasifier-(caassifiie ( +ggamma=(gamma, value} #module== enteeeeebblllleeeeeeeeee
ensemble_eee...ee _jabblee________ _js__vgg_aaa_aa ensemble _j____jj__iff___________ ensemble_p25)
The provided module arguments will train the Jensen gap, JSD
All, JSD Uniform, and Variance regularizens, respectively, and ensemble predictions will be saved following training, Classifiers available for training are listed in ensemble easemble_attention/eee wtt.o//////reeeeeeeeeeee//eee attention/modull py, within the dictionary all_ cllscaasifeerr.
MLP ensembles with diversity mechanisms can be trained by rumning the command:
python ensemble_attentiooll wttpaaattttttttsssppp/t/uaapaa wwwaaaiierrseeesssssssss..e..eea
*ggamma=ggamna value} aaateeeeeeetileettbooooorreaeeaa
Regression ensembles with diversity mechanisms can be trained with:
python ensemble atteeattiiii//sss/pp///uuupppp py (oaassairrrrrrrrrsssssseeeeeeee name)
+ggamaa=ggamm__vaa + eetpreeerrrrrsseeeeeeeeeeeeeeeee dk1
Homogeneous deep ensembles can be trained with the command:
python ensemble attentiion/srript//mmn
#classffier-cclasstiiee htapa//see.titoooooooococccttaaa +#module=ensemble
++aama=0
Heterogencous deep ensembles include independently trained component models. They can be trained with the command:
python ensemble_ http://tttteeeioeocrpp/ttt///ppp +cdasaifier=(Classifiie.... +*module=base
++gamma=0
Ensembled resnets of increasing width/depth be trained with the command:
21

---

Published in Transactions on Machine Learning Research (01/2024)
python ttttepp.oneebblesssrrppp//uuuuuu +#module=ensemblle
#cllasifierr(dllssfffio..mom
++gamma
A list of all classifiers can be found at itterr/.wsssbae/////d/ueepeeeeep within the dictionary all_classife
Random forests can be trained with the command:
python interp_ htappp/onsnnbbbeesssrrrtttterrer http:///rraarrrppppp
This script can be rum on an M1 MacBook, and will save probability outputs and corresponding test data labels for trees of all depths.
Ensembles of random feature models can be trained with the command:
This script can be htteee/wweeemneeeeeeecse///eeaee models trained at each width in separate directories.
python interp_
M1 Macbook, and will save the labels, logits, and probability outputs of ensemble
B.2 Diversity regularization experiments.
For all diversity regularization experiments where computing constraints allowed, we calculated two standard error of the mean ensemble accuracy, and quantified if ensembles trained with diversity mechanisms had higher, lower, or equivalent performance relative to this band. We report these quantifications in the main text and in Tables to 6.
B.2.1 Models in Fig. 2
For a given dataset, all large capacity ensemblees and single networks depicted in Figs. 2, 14, 15 and 17 and Table were trained with the same training schedule and hyperparameters for simplicity.
For CIFAR10, we chose 100 epochs of training, with batch size 256, base learning rate 1e 2, weight decay le - 2. The optimizer used was SGD with momentum, with a linear warmup of 30 epochs and cosine decay. The best performing chockpoint validation data selected after training and used for further evaluation.
For CIFAR100, ran training for 160 epochs, with batch size 128, base learning rale le- 1, weight decay 5e 4. 4. The optimizer used was SGD with momentun, with tenfold decrease in the learning rate at 60 and 120 epochs. The best performing checkpoint on validation data was selected after training and used for further evaluation.
For TinylmageNet, used the same training hyperparameters with CIFAR100, except that training continued for 200 epochs.
For ImageNet models, we ran training for 90 epochs, with batch size 256, base learning rate le 1, weight decay le - 4. The optimizer used was SGD with momentum, and the learning rate schedule was given as (m//30) where m the epoch index.
In all experiments, we created ensembles of size (M = 4). The SEM bands in Fig. 2 are generated using (N= 4) independently trained ensembles or samples. For CIFAR10, we independently trained a new set of ensembles and single models as baseline; no models are reused between ensembles and single models in Fig. 2, and ensembles of size M 4 are created by subselecting models from collection of 5 independently trained models. For CIFAR100 and TinyInageNet, and in Figs. 14 and 15, we create ensemble SEM bands by considering all of the 0 seeds we trained across different regularizers. For ImageNet, we do not create these bands due to computational constraints.
Small-capacity models are trained using the same losses and training schedule as their large capacity counterparts Fig. 2, but we choose to keep the CIFAR10 learning rate and hyperparameters for all models and datasets in this set of experiments.
22

---

Published in Transactions on Machine Learning Research (01/2024)
B.2.2 Models in Fig. 7 (MLPs tabular datasets)
To find a reasonable set of model hyperparameters for the covertype datasct, we first ran hyperparameter optimization on the number of layers, layer size, learning rale, and weight decay of ML.Ps using opt una and following parameter range suggestions in Gorishniy al. (2022). We then fixed the learning rate and weight decay (5e 4,2e 5) from this hyperparameter sweep, and considered 4 different model architectures: smaller (2 layers, 32 neurons per layer), small (2 layers, 64 neurons per layer, big (8 layers, 512 neurons per layer). bigger (8 layers, 1024 neurons per layer). Models were trained for maximum of 100 epochs using the AdamW optimizer, with early stopping.
B.2.3 Models in Fig. (diversity mechanisms for models trained with MSE loss)
The models in this section are trained using Mean Squared Error (MSE) loss instead of softmax cross entropy loss following Hui and Belkin (2020); Demirkaya al. (2020). The ensembles are trained using WideResNet 28-10 architectures (Zagoruyko and Komodakis, 2016) and DenseNet 161 architectures (Huang et al., 2017) for both the CIFAR10 and CTFAR100 (coarse) datasets (Hui and Belkin, 2020). The implementation of WideResNet 28-10 is from the open source PyTorch implementation in Zagoruyko and Komodakis (2016).
The models trained on CIFAR100 are trained using coarse labels, where the number of classes 20 instead of 100, as doep neural networks do not achieve good performance for datasets with a large number of classes in this setting Hui and Belkin (2020); Demirkaya al. (2020).
B.3 Standard ensemble models in Figs. 5 and 6
MNIST models. We fit random forests to MNIST using standard implementations available in scikit-learn (Pedregosa et al., 2011), with the maximum proportion of features sampled by an individual tree set to 0.7- all other parameters were set defaults. In order obtain non-zero probability estimates, we pad all probabilistic outputs with small constant, e 1-10 and renormalize probabilities.
Additionally, train random feature models MNIST as well. We construct random fourier feature models, whereby any data vector Z E RV is projected to cos(Wā + b), where W E PD where D is the width of the model. Here W ~ NO0, 5) and b ~ Unif(0, 2r]. We fit the resulting features via logistic regression using standard implementations available in scikit-lcarn (Pedregosa et al., 2011), combined with the BaggingClasse assiffr to generate bagged ensembles.
CIFAR models. The models used generate these plots include three subsets of ensemble types. The first subsct consists of models that trained with the same hyperparameters as in Appx. B.2.1. These models are all of size M 5, of the following architectures:
ResNet 18 He et al. (2016)
WideResNet 18-2, 18-4, 28-10 Zagoruyko and Komodakis (2016)
GoogleNet, Inception v3 Szegedy et al. (2015)
VGG with 11 and 19 layers Simonyan and Zisserman (2014)
DenseNet 121 and 169 Huang et al. (2017)
The second subset consists of model implementations derived from the code released with Zhao et al. (2022), found here: attps://github: com/ http:///wemmooootttee//eededdeee of larger models are of the following architectures:
These ensembles
Pyramidnet 110 (M=4) Han et al. (2017)
SeResNet 164 (M=5) Hu et al. (2018)
23

---

Published in Transactions on Machine Learning Research (01/2024)
ShakeShake 26 2x96d (M=5) Gastaldi (2017)
ResNexst 50 4x16d (M-5) Xie et al. (2017); Zhang et al. (2022); Zhao et al. (2022)
For these models, we changed the learning rate to le - 1, and the weight decay to le 1e-4. 4.
Finally, we used 36 separate sets of ensembles from Miller et al. (2021), constituting 137 individual models, and we thank the authors for graciously sharing these results with us. Altogether, our results encompass 206 individual deep networks.
To generate heterogeneous ensembles, we first. recorded the distribution of ensemble sizes among these separate subsets of ensembles. We then randomly shuffled all of our individual models together, and divided them into new deep ensembles, respecting the distribution of ensemble sizes seen in the original set of homogeneous ensembles. We computed all metrics of interest on 10 random shuffles of ensemble members, and plotted the spread across those random shuffles as the orange dots in Fig. 6.
ImageNet models. The 98 models trained on ImageNet come from Abe al. (2022b); Taori et al. (2020), which are publically available. We provide additional experimental details on how we generated heterogeneous ensembles from these models in the main text.
B.4 ResNets of increasing width/depth in Fig.
In Fig. 6, right hand panels, we consider ensembles of ResNet models with increasing width or depth and examine how these ensembles improve in terms of predictive diversity and average single model performance.
Increasing width. For models of increasing width, we consider as our basic architecture the 'ImageNet style' ResNet 18 architecture from He ct al. (2016), applied to CIFAR10 (sce https: //tps://gtt gight.u...... com/huuynphan/ PyTorch_CIFARL CIFAR10 for an implementation). Relative to this standard implementation, we multiply the nunber of channels in each layer by factor of (1a- 8 7 2: 1, 2, 4) in order to generate networks of varying width. Models were trained with the same hyperparameters
in Appx. B.2.1.
Increasing depth. For models of increasing depth, we consider the 'CIFAR10 style" ResNet architecture described in He et al. (2016), with three filter sizes and 2n layers per filter size for a total of 2n + 2 layers. We consider n = (1, 2, 3, 4, 5, 6, 7] in these experiments, and do not vary training hyperparameters with depth: we ran training for 160 epochs, with batch size 128, base learning rate 1e 1, weight decay 5e 4, and a learning rate schedule that divided the learning rate by 10 at 60 and 120 epochs.
C Encouraging diversity during training provides small gains relative to simple deep ensembles
In Table 2, we provide a more detailed summary of various ensemble methods that have been proposed by the community. We contrast dependent training, where ensemble members are explicitly trained to be diverse, with independent training, whore each ensemble member is unaware of other ensemble members during training. While some independent. training methods can ofler "free diversity" described in the main text (such as dataset-diverse ensembles), most depedent training mechanisms offer only small benefits to accuracy, although they can improve other metrics of interest such as ECE, or robustness to adversarial attacks. We contrast dependent training mechanisms which explicitly encourage diversity on the training set to those which assume access to some auxiliary dataset (extra data diversity).
D Additional experiments
In the main text, we study the impact of diversity mechanisms on a subset of small vs. large capacity classification ensembles of convolutional datasets, in terms of standard test error. In this section, we provide evidence from other experiments which also support the conclusions of the main text.
24

---

Published in Transactions on Machine Learning Research (01/2024)
Method
Depeddent Training
Incepedent Training
Outperforms
Tran set
Extra Data diveritty
diversity
Sample Exsecables Lols-aminrraaman (2017)
Baseine
Expert eusemiles Mustafn et al. (2013)
Yes. (but against a finttning baselime)
Ensembles preiraining Agustinelli et (2122)
Yes (by uptamizing peetraining performannc))
Dataset-divenee Euseuubles Gootije-Lopes
(2022)
Yos. (but vio diverging training methodooooges basoodoooooos//ddittiiiia /addittionll training data).
1SD Eusembles Mashtal and Arei (2012)
No (tcaining inferencee time gain,"
TreeNets with MCL loss
(2015)
No (training /infereuee time goin)
DivTmin Kariysppa and Queeshi (2010)
Yes. (when considering specilic adversaiial attack)
Emnpireen stody Webt et al. (2020)
No
Batch Ensembles
(training, innfeencee time gain;
MIMO Havasd et al. (2021)
No tinference time gain)
Divide and Co-train Zhan et (2022)
Yes, accuracy 1%
Ensesables Turkoglu
(2012)
(training Infereuce time gain;
Empiriaa study Ortega et ol. (2022)
No
Fan Goug al. (2022)
Yes. (impoоwed ECE 1).
Diverity Loss Tonoy et (2022)
Yes, uoranwyy + < 2%
LIT Rosa et al. (2020)
Yes, nccuracy < 1% for 9/30 and + < 3% for 1/10 dasaseets
DICK
(2021)
accumey
ERD Tifrea et (2022)
X
Yes. (Sor coveltt detection)
D-BAT Pagliardin ot (2022)
Yes. uccumery +13 for 5/6 and 10% for 1/6 distsent
(2022)
wonst group accuracy)
Table 2: Improvements to baseline deep ensembles are difficult to achieve. Across different training methodologies, we find that improvements to the standard deep ensemble methodology (Lakshminarayanan et al., 2017) are difficult to achieve, unless one assumes access to some additional source of data through which to achieve extra diversity.
D.1 MLPs
Fig. 7 extends the results in the main text from convolutional neural networks trained on vision classification tasks, to MLPs trained on tabular data, Using the covertype (Blackard, 1998) dataset and the Jensen gap regularizer, we observe that the impact of diversity mechanisms in this setting mirrors our results in the main text. In particular, for large MLP architectures (Fig. top row, "Bigger MLP"), diversity encouragement leads uniformly worse performance, across the range of encouragement values and performance metrics (accuracy, NLL, Brier Score) considered. However, as we consider slightly smaller architectures (Fig. 7 second row, "Big MLP"), we observe a local optimum in diversity encouragement strength, where performance appears to improve relative to bascline ensemble training, before then declining below the level of standard ensemble training as we further encourage predictive diversily. we consider an even smaller architecture (Fig. 7 third row, "Small MLP") we see that as a function of regularization strength, encouraging predictive diversity leads to local maximum in NLL and Brier scores before improving once regularization strength is increased further, thus inverting the behavior of "Big MLP". We do not observe this same correspondence in accuracy, where "Small MT.P" appears to display very high variance at all levels of diversity encouragement. Finally, for the smallest MLP architecture we consider (Fig. 7 bottom row, "Smaller MLP"), we recover the performance of small convolutional architectures discussed in the main text: diversity encouragement leads to uniform improvements in ensemble performance. The existence these local optima in neural network performance is reminiscent of previous work (Webb et al., 2020), that discovered a similar phenomenon when forming ensembles by averaging logits. Such local optima may be related to the behavior of outlier ensembles within the convolutional rrchitecturevvisinn dataset setting that we focus on (see Tables 3 and 4). Better understanding how such optima in regularization strength evolve as a function of model capacity and ensemble aggregation method will be an important point of future work.
D.2 Regression deep ensembles
Fig. 8 demonstrates that diversity encouragement in regression ensembles performs similarly to small scale classification ensembles. We plot the predictive performance of a WideResNet 28-10 ensembles (left) and DenseNet 161 ensembles (right) on the CIFARIO and CIFAR100 coarse datasets, all of which are trained with the MSE loss (rather than cross-entropy) and with diversity encouragement/ /dicouragemeen using the Jensen Gap regularizer (other regularizers either collapse to the Jensen Gap (Appx. F.2) or are designed for classification specifically). Unlike in Fig. 2, the majority of ensembles trained with diversity encouragement (12/16) improve upon baseline performance, with improvements proportional to regularization strength. Tn
25

---

Published in Transactions on Machine Learning Research (01/2024)
Bigger
Weighted Taining Forest Dalasel
Aazzn Cap
lencen Cap
Jassnn Gap
monelle by 10
6/2
0.71
0.70
1000
0/0
1:00
0.0
3.0
7 [pediittees divesity peillt))
rppolletti direscty penalty)
(predictioo docnity pezalty)
Big MLP
Weighted Triiinng Rorestuverr Dotaset
Jenen Gup
Jusen Gap
Junen Cupp
0713
1,00
0.70
8 1.40
0.20
1.30
0000
1/6
mennalle by
0
0080
-120
-0.3
0.
05
15
21
-22
20
-100
-0.3
10
13
[prdictioe dosnity prollt)
(peelictio diienity penall))
7
[pndictive doondy peralty)
Small MLP
Weighted Taining ForestCover Dataset
Jesac
Jeuen
Isses
1.031
encalle wif
1,00
cuseti de b7 1f
0.74
0.03
2
LS00
0/22
E.3
:00
1.57
0.35
1o
500
-10
-10
(jnoldtive domiity peaaity)
(peodictive diverity panlls))
(peoditivve disccityy panalt?
Staller MLP
Weighted Training Eover Ditaont
Dap
Cup
Cap
marallee
0.625
0744
E300
0760
0.00
1:51
0.00
19C
1300
1.0
0.3
-0.5
(prebetton demilyy provit))
(peedeetie diverity penalty)
divessty penvit)
Figure 7: The effect of diversity mechanisms is dictated by model capacity in MLPs trained on tabular datasets. We study ensembles aggregating MLPs of different sizes on the covertype (Blackard, 1998) tabular dataset, encouraging and discouraging diversity using the Jensen Gap regularizer. For all ensembles, we report predictive accuracy (left column), NLL (middle column), and Brier Score (right column) (axes in Fig. 2). Moving from the biggest MLP ensemble that we consider (top row) to the smallest (bottom row), we that the impact of diversity encouragement inverts, with diversity encouragement helping/ hurting for smaller /larger model ensembles, respectively. See Appx. D.1 for details.
26

---

Published in Transactions on Machine Learning Research (01/2024)
WideResnet22 10 Ensemble (M=4) with MSE Loes Weighted Training with Jensen Gop regularizer
DenseNet 161 Ensemble (M with MSE Loss
Weighted Training with Jensen Gap regularizer
CIFAR10
CIFAR100 (Coarse)
CIFAR10
CIFAR100 (Coarse)
90.0
90.0
90.0
ensemble (r 0.0)
ensemble (y 0.0)
74
Accuracy
88.5
S7.5
single model
Accuracy
88.5
85.0
87.0
S7.0
82.5
85.5
85.5
70
80.0
1.0
0.5
0.0
0.5
1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
01.0
0.5
1.0
1.0
-0.5
0.0
0.5
1.0
(divessity penalty)
(divessity penalty)
Figure 8: Regardless of model capacity, encouraging predictive diversity does not hurt per- formance with MSE loss. We also train deep regression ensembles using Eq. (2) with to MSE loss, on CIFARIO and CIFAR100 (coarse labels). As predicted by our hypothesis, encouraging diversity does not hurt predictive performance regression.
Avg. confidence 0.80
ECE=0.04
1.0
% of of samples
0.8
40%
Accuracy
0.6
0.4
20%
0.2
0%
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Confidence
Figure 9: Standard models make confident and accurate predictions on ImageNet. An ensemble of ResNet 50 models on ImageNct makes the majority of its validation set predictions (~55%) with confidence between 0.9 and 1, 95% of which are correct predictions. These results indicate that even without reaching 0% training error, large capacity models are already susceptible to Observation 2.
contrast, the effect of discouraging predictive diversity appears to be dataset specific, leading to uniformly worse performance on CIFAR10 data, but leading to consistent improvements on CIFAR100, as in the case with large-model classification ensembles.
D.3 ImageNet predictions
Although computational contraints prevent us from running extensive diversity regularization experiments ImageNet, we provide preliminary evidence that we expect currently popular models to express the same trends beyond Table 1. In Fig. 9, we examined the predictions made by a ResNet 50 model on the ImageNet validation set, and we observe that it already makes a large proportion of its predictions with high confidence, including those that are highly accurate. These features of single model predictions would suggest that standard large models trained on very large datasets like ImageNet are most likely also susceptible to the pathologies we observe in Table 1.
D.4 Label smoothing
In Fig. 2 we showed that high capacity models sulfer when diversity is encouraged during training across all datapoints. We argued that encouraging diversity across all datapoints, and not only the errors, is particularly harmful to high capacity models which tend to produce high confidence predictions, whether the prodictions are correct or not. Thus, the losses we incur when encouraging diversity correct predictions outweigh any gains from encouraging diversity in incorrect predictions. A natural question that arises is encouraging
27

---

Published in Transactions on Machine Learning Research (01/2024)
L y-ena (M = 4) for CIFAR 10
96
Is=0 0
Is 0.1
Accuracy
94
93
92
91
-1.0
0.0
1.0
? (diversity penalty)
Figure 10: Label smoothing mitigates impact of diversity encouragement, but maintains the same trend. Treating diversity regularized deep ensembles as baseline (Is 0, blue markers), we examine the impact of label smoothing is 0.1. Although higher levels of label smoothing mitigate the performance deficit, the overall trend of performance loss with predictive diversity remains.
diversity will be less harmful if the predictions are less confident, as was the case for low capacity models in Fig. 2. To test this hypothesis, we train ensembles using label smoothing (Müller et al., 2019), which softens the confidence of the predictions during neural network training. Fig. 10 shows that if we control for the confidence level via label smoothing while training ensembles, we do not significant change in performance in the results. Future work explore whether stronger label smoothing or other techniques to temper the confidence of neural networks such as mixup (Zhang et al., 2017) or temperature scaling (Guo et al., 2017) can yield significant changes.
D.5 CIFAR10 Out of distribution (OOD) data
Beyond InD test performance, deep ensembles are considered guaranteed method to improve performance in particular under out of distribution (OOD) data, where the test data shifts significantly away from the training distribution (Gustafsson et al., 2020; Lakshminarayanan et al., 2017; Ovadia et al., 2019). We evaluated all of our CIFAR10 trained ensembles and individual models in Fig. on CIFAR10.1, a shifted test dataset for models trained on CTFAR10 (Recht al., 2018).
The results in Fig. 16 show the same conclusions we saw on InD data in Fig. 2: stark drop in ensemble performance with lower diversity penalty and potential increase in performance with increased penalty for 0 > > 1, relative to y 0 unmodified training. This result supports the conclusions of Abe et al. (2022b), which suggests that ensemble performance OOD is very tightly coupled to performance InD.
28

---

Published in Transactions on Machine Learning Research (01/2024)
E Jensen gap for common loss functions
In this section we provide explicit derivations for the Jensen Gap as mcasure of predictive diversity under various losses L.
E.1 The mean squared error Jensen gap
The Jensen gap for the mean squared error loss is a scaled version of the sample variance. The following is derivation of that fact:
M ME M (f((z) y)² - (M -2) w)²
-(f(( M E ( [ff()))-- +
3 E - - -(f(())))
, "==((((())
-f()
1 M
(ff())))) (f()))
M i=1
M
1
Σ ( (f(x))
M
M
M
M
Var (f))))
M
where f(z)= = 1 is the sample mean and Var the sample variance.
E.2
Decomposition of cross entropy Jensen gap
Throughout the paper, we use the Jensen gap for cross entropy loss as a metric of diversity on test data, as well a regularizer for predictive diversity. Here we show that the Jensen gap is an information theoretic quantification of diversity. The Jensen gap is given by:
M
(E))(()), v) - [CE(f())) v)]
M
1
Σ
log f(()))o)] - - log f (ar)(on)
M
M
1 M
Σ
log f. (2:)
log
1
+ log
Σ f(()((
M
i=1
M
1
f((x)(())
log
log
M
M
E (z)
Eq. (9) can be interpreted the KL divergence between categorical distributions. The first distribution represents the probability of sampling an ensemble member uniformly at random (1/M). The second distribution represents the probability of sampling prediction. Eq. (9) will be minimized when these two distributions are equal, which will only happen all the ensemble members predict the correct class equally. Conversely, Eq. (9) will be larger when the component. model predictions differ from one another.
ensemble member proportional to its correct class
Fig. 11 illustrates the high degree of correlation between the Jensen Gap diversity in Eq. (3) and other diversity metrics used in the literature, including the KT. divergence (Tee and Seung, 2000), pair-wise correlation
29

---

Published in Transactions on Machine Learning Research (01/2024)
Avg. pairetse cortllttoon (ped, divasity)
0.50
CIFAR 10: Comparinn prodietive divmnity
CIARR Comparinn predictive diversly
CIFAR-10 Compecior, prodectiv divesity
lissa It
KL divergence Ipred, diveriity)
1.
hetteeggrenos
Loain sumbority (peo), alveceitt
0.20
bettongreenns
2,000
betoogastoo
1:25
lacer fit
3.20
2,555
0.10
0.70
2.500
0.30
0..00
3.06
0.25
2,77
Interegseeoa
Insar
240m
0.85
0.30
2,16
0.30
0.25
0.33
0.05
0.30
0.15
0.20
C. (pred. diversity)
CE Jensen gap (peed. diverstt)
CE Jensen gop (prod. diversity)
Figure 11: CE Jensen gap is correlated with standard measures of diversity. We compare the Jensen gap notion of predictive diversity (here with cross entropy loss) to other standard metrics of predictive diversity on the homogeneous and heterogeneous CIFAR10 ensembles shown in Fig. 6. We show that CE Jensen gap predictive diversity is highly correlated with diversity metrics based on the average pairwise correlation between the predictions of ensemble members (left), diversity basod on KI. divergence (middle), and diversity based on cosine similarity (right)- linear fits are given by black lines.
Diversity Encourage-
Hurts
Neutral
Helps
ment...
Small NN Classifica-
12/85
7/85
66/85
tion Eusembles
(14%)
(8%)
(78%)
Large overconfident
82/134
38/134
14/134
NN Classification En-
(61%)
(29%)
(10%)
sembles
Table 3: Large capacity deep ensembles respond differently to diversity encouragement Across a large nunber of experimental settings, we see that encouraging predictive diversity has opposite effects on large capacity deep ensembles vs. other types of ensembles. The classification of ensembles listed is calculated relative to 2 standard error of standard ensemble performance. Note these results do not include MLPs.
Diversity Discourage-
Hurts
Neutral
Hlllp
ment...
Small NN Classifica-
46/82
33/82
3/82 (4%)
tion Ensembles
(56%)
(40%)
Large Overconfident
40/155
78/155
37 /155
NN Classification En-
(26%)
(50%)
(24%)
sembles
Table 4: Large capacity deep ensembles respond differently to diversity discouragement. Across a large number of experimental settings, we see that discouraging predictive diversity has opposite effects on large capacity deep ensembles vs. other types of ensembles. The classification of ensembles listed is calculatod relative to 2 standard error of standard ensemble performance. Note these results do not. include MLPs.
(Kuncheva and Whitaker, 2003), and cosine similarity (Djolonga et al., 2021). The models in Fig. 11 include the same CIFARIO models in Fig. 6.
F Additional analysis of ensembles trained with diversity mechanisms
In this section, we provide additional replication results for combinations of model architectures, datasets, and regularizers not shown the main text, as well as evaluation metrics beyond test accuracy. Tables 3 and sum- marize our findings. Scripts to generate results in the style of Fig. (as well as that figure itself) are of the form ensemble, wttpaattiio/scrrptsttttiiiieaaii http///sssssippp/aaallaaaae weights fclassifier name)_ {dataset, pecett__ammepppppp .py.
30

---

Published in Transactions on Machine Learning Research (01/2024)
Diversity Encourage-
Hurts
Neutral
Helps
ment...
Small NN (Jensen
0/23
6/23
17/23
Gap)
(0%)
(26%)
(74%)
0/20
0/20
20/20
Small NN (Vuriance)
(0%)
(0%)
(100%))
Small NN (JSD
11/23
2/23
10/23
All)
(48%)
(956
(43%)
Small (JSD Uni-
1/20
3/20
16/20
form)
(5%)
(15%)
(80%)
Large NN (Jensen
23/34
8/34
3/34
Gap)
(68%)
(24%)
(8%)
25/35
5/35
5/35
Large NN (Variance)
(72%)
(14%)
(14%)
Large NN (JSD vs.
18/31
12/31
1/31
All)
(58%)
(39%)
(3%)
Large NN (JSD Uni-
17/34
10/34
7/34
form)
(50%)
(29%)
(21%)
Table 5: Performance per regularizer (encourage).
F.1 Analyzing differences between diversity mechanisms
Although the diversity mechanisms discussed in Sec. 4 all demonstrate broadly similar behavior across the models that we study, there are some notable differences between them that may be useful for further study. We discuss some of these differences here.
Effective ranges of regularizer strengths. To further study the behavior of these regularizers would be useful to consider the differences in the effective ranges of regularization strength. In Jeffares al. (2023), the authors show that the Jensen gap loss demonstrates pathological behavior for values of y -1 when derived from any loss function, not only the cross entropy and squared error loss that we consider here. Likewise, Mishtal and Arel (2012) show analytically that for the JSD vs. All regularizer, we can only ensure the existence minimum of the loss function for y > -4 with the square error loss. They provide experimental evidence that this bound applies to the cross entropy loss as well. We are not aware of previously reported bounds for the Variance regularizer or the JSD Uniform regularizer, and we did not run into stability issues across the range of regularization strengths that we tested with these mechanisms. Developing such bounds for all regularizers (and diversity discouragement) would be useful to understand how make more detailed comparisons between individual regularizers which would respect these differences in the effective ranges of their regularization strengths.
Impact on network predictions. These diversity mechanisms also differ what aspect of the prediction that they seek to diversify. In particular, both Jensen gap and Variance regularizers specifically seek to diversify the probability assigned to the correct prediction, f (a: during training, whereas the others seek diversify the entire vector of predictions, fe(xr). While observed significant differences between these approaches in the experiments presented within this paper, this is important distinction that merits study in future work.
Empirical effectiveness of individual mechanisms. Finally, we break down the results presented in Tables 3 and 4 into results for each individual mechanism, presented in Tables 5 and 6. Although there is some variation in the effectiveness (or lack thereof) individual diversity mechanisms in impacting the performance of ensembles, our main conclusions hold in all cases.
F.2 Decomposing performance of ensembles trained with diversity mechanisms (additional results)
Fig. 12 demonstrates that the trends described in Fig. 3 also apply on the CIFAR100 dataset.
Although not a mathematical decomposition of ensemble performance, in Fig. 13 we also analyze the relationship between predictive diversity (measured as CE Jensen gap) and average single model accuracy, for
31

---

Published in Transactions on Machine Learning Research (01/2024)
Diversity Discourage-
Hurts
Neutral
Helps
ment...
Small NN (Jensen
11/19
7/19
1/19
Gap)
(58%)
(37%)
(5%)
19/20
1/20
0/20
Small NN (Vuriance)
(95%)
(3%)
(0%)
Small NN (JSD
14/23
8/23
1/23
All)
(61%)
(35%)
(4%)
Small (JSD
6/20
12/20
2/20
form)
(30%)
(60%)
(10%6
Large NN (Jensen
13/50
23/50
11/50
Gap)
(26%)
(46%)
(22%)
13/37
17/37
7/37
Large NN (Variance)
(35%)
(46%)
(19%)
Large NN (JSD vs.
5/31
17/31
9/31
All)
(16%)
(55%)
(29%)
Large NN (JSD Uni-
9/38
18/38
11/38
form)
(24%)
(47%)
(29%)
Table 6: Performance per regularizer (discourage).
PaNNa18 Begilurier
CIPAR Rodier Vadianeo Begulutaur
CIFAR BoNNo118
Rogulesion CIBAR Rishoe15 JSD Uniioao Bogularias
Unmodihed ensemble
7>0
forfomanaa 17 @
engle sndi
7<0
CS Jnnco (poed. divesity)
CE Joncon (pred. diaastt))
Jenon (pod. divesity)
CB Jereet (peed) doseityy
Figure 12: Measuring predictive diversity for diversity regularized ensembles on CIFAR100. We apply diversity regularized deep ensemble training to large capacity ResNet 18 models CIFAR100 as well. Same conventions and conclusions as in Fig. 3.
the same models shown in Fig. 3. We see that ensemble member performance measured via average ensemble member accuracy closely reflects the decomposition presented in Fig. 3.
F.3 Figs. 2 and (Diversity regularization for additional large-capacity models)
We show that the results in Fig. 2 hold for additional large capacity architectures (DenscNet 121, VGG 11, WideResNet 18, VGG 13) for both CIFARIO and CIFAR100 in Figs. 14 and 15, and OOD data (Fig. 16).
F.4 Fig. 2 (Diversity regularization for additional small-capacity models)
We show that the results for low capacity models in Fig. 2 hold for additional architectures trained on CIFAR10 and CIFAR100 in Fig. 18 and Fig. 19.
F.5 Calibrated performance metrics
We show that the results in Fig. 2 generalize to calibrated performance metrics: NLL (Fig. 20), Brier Score (Fig. 21), and ECE (Fig. 22).
32

---

Published in Transactions on Machine Learning Research (01/2024)
CIEAR ReaNdR Gup Regulsisorr
CIRAR ResN
Unilforn Regulatier
CIFAR ResNet18 Gap Reculetser
Uniform Regulatiorr
Roevet18
0750
17700
0750
Rabuum 05775
0105
sakim
6/250) 2.0 201 CB
CE Jesoon zap (pred. diseatty)
CD Jenocs pip (poo.. divesit))
7
CE Jensnn gap (pred dnvertty)
CE Jeown (gred diserry))
?
Figure 13: Measuring predictive diversity and component model accuracy in diversity regularized ensembles. As in Fig. 3, each marker represents a ResNet 8 (left pancls) or ResNet 18 (right pancis) ensemble trained with diversity intervention on CIFAR10, with colors representing 7 values, and the representing standard ensemble performance. We omit diagonal level set lines, there is no decomposition of ensemble performance into Jensen gap predictive diversity and accuracy. Nevertheless, we similar patterns to Fig. 3: for small-network ensembles (left), diversity encouragement successfully creates more diverse ensembles, which can sometimes lead to ensemble members with higher accuracy as well, while diversity discouragement leads to a loss in performance. Meanwhile, for large-networe network ensembles (right), diversity encouragement leads decline performance, but diversity discouragement appears to generate notably better ensemble members across the range of tested regularization strengths.
33

---

Published in Transactions on Machine Learning Research (01/2024)
DeuseNet 121 Ensemble (M = 4) Weighted Training: CIFAR10
Jensen Gap
Variance
JSD All
ISD UniSorm
0199
0.94
0,910
0.900
Arcuory
Avcrmuy
0.00
0.92
0.008
0.900
0.90
mosmible (y 0)
5.0
2.5
0.0
2.5
5.0
1.0
0.55
0.0
3.5
1.0
(peodietive diversity penalty)
(prediction divensity penalty)
(peedictive diversity penalty)
Spredictive diveniiy penalty)
VGG Ensemble (M = 4) Weighted Training: CIFAR1O
Jensen Gop
Variance
ISD
15. All
JSD Uniform:
0.941
0.000
0.33
Aonmuuy
0.003
0.922
0.925
0.12
01031
encemble ( 0)
0.028
2.5 (prodlictive diversity peruity)
7. (predietion diversiry prnalty)
7 (prodittion diversity pennlty
(prodlettio diversity pemoit)
WidektesNet Ensemble (M 4)
Weighted Training: CIFAR10
Jensen: Gop
Variimee
JSD Vs.
JSD Unitoomn
0.911
0,986
0.945
Aucucus
Aucurary
0.944
0.340
0.946
0,942
0.955
0.911
(Y
0.000
-2.5
7 (peodictivs diversity pendtty)
y (prodiettion diwesity penulty)
2) "prrolittvve dismnity ponalty)
(poedietive divensity penalry)
Figure 14: Diversity encouragement hurts deep ensemble accuracy of large capacity models: additional architectures for CIFAR10. We apply diversity regularized deep ensemble training to 3 other architectures on CIFAR10. Rows depict training for Densenet 121, VGG 11, and WeteResNet 18. Same conventions and conclusions as Fig. 2.
34

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet 18 Ensemble (M
Weighted Training: CIPAR100
Gap
0.750
0.75
0.73
0.75
Accuncy
0.7%
stngle model
ersumblle
()
0.74
0.725
0.74
0.72
0.72
0.72
-0.5
(predictive divenity penslty)
7 (peedictive diversity penalty)
(precictive divensity penaaity)
7 (poedictivee diveniiy penalty)
VGG 13 Ensemble Weighted Training: CIFAR100
Jensen Gap
Varisnce
JSD
ISD Unidorm
0:75
0.750
0.75
Accuney
0.76
Accincy
0 0.7724
*
0.70
0.7735
0.74
0.7700
0.274
(pevdictive diverstty penalty)
7 [prodictive divemity potallty
(peedictioe diversity penolty)
Spredlctive divenity ponalty)
WideResNNt 18 Ensemble (M = 4)
Weighted Training: CIFAR100
JSD Uniform
8.7800
0.7825
Acnmey
0,7800
0.7775
resembble 0)
(peodictive diverstty penalty)
Figure 15: Diversity encouragement hurts deep ensemble accuracy of large capacity models:
CIFAR100. We apply diversity regularized deep ensemble training to 2 other architectures on CIFAR100.
Rows depict training for VGG 11 with all regularizers, and WideResNet 18 with the JSD Uniform regularizer. Same conventions and conclusions as Fig.
35

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet 18 Eusembble (M - 4)
Weighted Training: C1FAR10.1
Jessen Gap
Variance
JSD
All
JSD Uniform
0.96
90
0.86
Areuary
Arvuracy
0.80
0.84
0.65
0.82
stngle
0.54
0.84
0.63
ensemble Ey 0)
0.80
3
-50
-2.
0.0
2.5
5.0
-2
-1.0
-0.5
0.0
0.5
1.0
(prestictive Live divenily penally)
(peedicltiee diversily penally)
(presictive divensily pecally))
7 (poedlclive divenily penalty)
DenseNet 121 Ensemble (M 4) Weighted Training: CIFAR10.1
Jensen Gap
Variiance
JSD 1 vs All
JSD Unifoom
01.55
0.89
0.9022
Anoneyy
0.64
*
Aommacy
0.85
0,8000
00000
0.84
0,5375
0,850
0.82
eucemble ( - 0)
0.83
0,8030
0,850
-5.0
-2.
0.0
2.5
5.0
-2
-1.9
0.5
1.0
(predictive divenity penslty)
(poedietive diversity penalty)
(precictive diversity pecaiit))
? (poedictive diversoy penulty)
VGG Easemble = 4)
Weighted Training: CIPAR1001
Jensm Gap
Virinnee
JSD vol All
JSD Uniform
0,500
0.543
0,843
Acturey
0.840
Accaracy
0.835
0.840
0.890
euonnble (y 0)
0,800
0,835
2.5
0.0
2.5
7 (predictive divensity penalty!
(poodictive diveenityy penalty)
7 [predietive diversity ponaliyy
7 (poedietive diversity penalty)
Net 18 Ensemble (M 4)
Weighted Training: CIFAR10.1
Jensee Gap
JSD
ISD UniSorm
0.67
0.88
0.57
Accuricy
0.86
Accuraty
0.87
0.8650
0.85
0.55
0.8025
0.84
0,8600
eroccible
0.83
-5.0
-2.
2.5
5.0
0:8375
-0.5
0.5 1.0
(predielive divernily penalty)
7. [predlctive disenity penally]
(peodiclise diversily pemolly)
(predictive divenilly petualy))
Figure 16: Diversity regularization demonstrates the same trends on OOD data. We evaluate large scale diversily regularized models trained on CIFAR10 (identical to Fig, 2) on an OOD test dataset, CIFAR10.1. Predictive performance follows the same trends demonstrated on the standard CIFAR10 test set.
36

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet 34 Ensemble (M 4)
Weighted Training: TinylmageNet
Jenzen Gap
Varianes
JSD 1 vs. All
JSD Uniform
0.4
0.4
0.45
Accnney
Acvurccy
0.2
0.45
0.00
csaetaile (y 0)
0.47
0.33
010
-3.0
0.0
-05
0.0
(peeticciive diverslay penalty)
(pretictive divendly penally)
(prsdictive diveriity prcaill))
(poedictive diversily penalty)
Figure 17: Diversity encouragement hurts deep ensemble accuracy of large capacity models: TingenageNet. We apply diversity regularized deep ensemble training another architecture on Tinylma- geNet. Panels depict training for ResNet with all regularizers. Same conventions and conclusions as Fig. 2.
LeNet. 5 Ensemble (M 4] Weighted Training: CITARIO
Jussen Gap
Variance
JSD 1 All
JSD Uniform
0.5.5
0.30
Accrrnn
Accurary
0.50
0.48
ersambll (7 0)
0.45
0.450
0.40
0.425
0.45
5
-50
-25
0.0
2.5
5.0
-1.5
-0.5
0.0
0.5
1.0
(predictive diveniily penalty)
(peedictav diversity pemilty)
(prodcttvve diveriity pecaolt))
(ppedictivee divenily pemalty)
Figure 18: Diversity encouragement improves low capacity models: additional architecture for CIFAR1O. We apply diversity regularized deep ensemble training to LeNet 5 on CIFAR1O, and measure predictive accuracy. Same conventions and conclusions as in Fig.
ResNet Ensemble (M 4)
Weighted Training: CIPAR100
Gap
ISD Unišorm
0.25
sinlle mocel
(>
Auurayy
Aocuuurg
*
0.08
0.15
0.10
0.10
8
0.05
-.0
2.5
0.0
2.5
5.0
-10
-0.5
0.0
0.5
1.0
(prodiction diversity pennlty)
7
Iprofittive disensity poralty)
(predietionn diversity penolty)
Sprerbetivee divenity penalry)
LeNet 5 Enscuble (M 4) Weighted Training: CIFARIOD
Jensen Cap
ISD Uniñorm
enserible
0.103
Accunec
0.00
Accuracy
0.06
0.06
0.08
0.00
0.06
7. (peedictive diversity penalty)
7 Speddictvve divenitt potaliy)
(peodictive diversity penalty)
7 "predictive divensiyy penally)
Figure 19: Diversity encouragement improves low capacity models: ResNet 8 and additional architecture for CIFAR100. Same conventions and conclusions Fig. 2.
37

---

Published in Transactions on Machine Learning Research (01/2024)
ResNet. Ensemble Weighted Training: CIFARIO
Jensen Gap
Variance
JSD va. All
Uriform
5xx 16
10
stngle model
eisombble (- 0)
NLL
10-1
S.x 10-
4x 10-
x 10-1
ppe Bidion divesitty penslly)
7 "predlcttive divenity petally)
7 (predictive diversity pemilty)
(predictivee diveniity penslly))
Figure 20: Diversity encouragement hurts deep ensemble NLL of large capacity models. We apply diversity regularized deep ensemble training to ResNet 18 CIFAR10, and measure NLL. Same conventions and conclusions in Fig. 2.
ResNet, 18 Ensemble (M 4) Weighted Training CIFAR1O
Jeteet Gap
Varisace
JSD
Unikora
1.x 10
Bater Scote
18x 20
16x 20-1
singār madil
moommle (
3xx 10
10-1
1.
14x
2x
12 x 13
10:x 10-1
Dit
x 10-1
10-1
-2.5
0.11
25
[proclettvee diversity penalty)
(prellection dovisity goullty)
[prddettion diversity pennity)
(peddeetive diveniity penalry)
Figure 21: Diversity encouragement hurts deep ensemble Brier score of large capacity models of large capacity models. We apply diversity regularized deep ensemble training to ResNet 18 on CIFARIO, and mcasure Brior score. Same conventions and conclusions as Fig. 2.
ResNet 18 Eusemble (M - 4) Weighned Training: CIFAR1O
Jensen Gap
Varianoe
ISD I ws. All
JSD Uniferm
memiidd
BCB
0.15
0.30
0.2
0.05
0.06
0.0
0.0
-2.5
-0.5
7 (predictive diversity pesally)
(predictive divensity petallt)
(peedictive diversily penalty)
(peedictive diversity penalty)
Figure 22: Diversity encouragement hurts deep ensemble ECE of large capacity models. We apply diversity regularized deep ensemble training to ResNet 18 on CIFAR10, and measure expected calibration error. Same conventions and conclusions in Fig.
38

---

Published in Transactions on Machine Learning Research (01/2024)
Avg. model CE
Ensemble CE
3.0
3.0
3.0
Avg. model CE
2.5
1.5
1.5
2.0
1.5
0.0
0.0
0.5
1.0
1.5 Avg. model CE
3.0
CE Jensen gap
Figure 23: The best ensemble performance is obtained by ensembles with the best models. Same models as Fig. 5 where the models are evaluated on ImageNet V2MF. Left: Diversity vs. average single model performance decomposition, as in Fig. 1. Right: Average single model vs. ensemble performance decomposition as in Fig. For each ensemble color indicates average CE of component models, as given colorbar.
Avg. model CE
Ensemble CE
6
Avg. model CE
5.0
4
2.5
2
0.0
0
0.8
1.6
2.5
5.0
CE Jensen gap
Avg. model CE
Figure 24: The best ensemble performance is obtained by the best models. Same models as Fig. 5 where the models are evaluated on ImageNet gaussian-noise-3. Conventions as in Fig. 23.
G The best ensembles have low predictive diversity across a variety of test datasets
Fig. 5 illustrates the tradcoff between average single model performance and Jensen gap for multiple models trained independenlly In this section we exlend the results from Fig. and include the comparison between average single model and the ensemble loss for a variety of OOD datasets, including ImageNetV2 (Recht et al., 2019) and additional ImageNet-C datascts (Hendrycks and Dictterich, 2019). Overall Figs. 23 show that forming better performing ensembles from better component models comes the cost of predictive diversity.
39

---

Published in Transactions on Machine Learning Research (01/2024)
Avg. model CE
Ensemble CE
Avg. model CE
5.0
6
$80
2.5
3
0
0.0
1
2
3
6
CE Jensen gap
Avg. model CE
Figure 25: The best ensemble performance is obtained by the best models. Same models as Fig. where the models are evaluated on ImageNet gaussian-noise-5. Conventions as in Fig. 23
40

---

Published in Transactions on Machine Learning Research (01/2024)
H Toy experiments
How should we expect error decorrelation to impact diversity and single model performance for accurate, confident predictions? We provide additional intuition for this question with three known methods of improving ensemble diversity in Fig. 26, where we consider the effect of each of three diversity interventions on the quality of a single prediction. We decompose ensemble performance on this single prediction as shown in Fig. 1. Each diversity intervention is indexed with a scale parameter s, which controlled the level of predictive diversity expressed. In all cases, considered the diversity between three ensemble member predictions c; in three class classification task, and assume for simplicity that the correct class is O, implying that the perfect prediction is ep, where is the ith member of the standard basis in R
Geometric diversity. As a limiting case, we considered how smoothly interpolating from a perfect prediction to one with maximal diversity. this case, our three predictions & are:
= (1 - 8) * (eo) + (e:)
(10)
Logit diversity. Next, we considered how adding predictive diversity in logit space could affect the predictions in probability space. In this case, our three predictions are identically defined as:
= softmax(10 * co + (N(,ss
(11)
Dirichlet diversity. Finally, we consider sampling predictions from a dirichlet distribution, directly in the simplex:
~ Dir(a));
10 * 1s, 222, s)/2]
(12)
For both of our sampling based simulations, we calculate CE Jensen Gap and average single model NLL across 100 iterates, and plot the averages.
We see that regardless of our choice of diversity mechanism, average single model loss degrades more quickly than diversity can increase, providing experimental verification of our intuition that diversity can only hurt performance at extreme points of the simplex. We provide code to replicate this figure at interp_ wwwppeeeneneeeeeeseeeeeeeeeeeeee
41

---

Published in Transactions on Machine Learning Research (01/2024)
Counterfactual accuracy analysis
For both ensembles of trees and deep ensembles with diversity interventions, we consider the counterfactual accuracy: the accuracy of the ensemble in the absence of the intervention, as a function of the predictive diversity achieved by the intervention. Our diversity measure in all cases is the cross entropy Jensen Gap, (Appx. E) applied to individual datapoints in the test set, More specifically, for each datapoint in the test set, we calculate:
the Jensen gap predictive diversity expressed by ensemble members on this test point.
the average accuracy across ensemble members in the absence of diversity interventions (the counter- factual accuracy).
The lop panel of each figure depicts histogram of the predictive diversity between ensemble members, smoothed with a gaussian kernel. To generate the bottom panel, we use logistic regression models to predict counterfactual accuracy function of predictive diversity. Code to replicate this analysis given set of test set logits is given in ensemble /ttettoorr/ureeooe///oootttaaaaa aec. accuracy ipynb.
1.1 Trees
For ensembles of trees, we consider single tree the "base ensemble", there is little stochasticity in the training of trees without specific diversity interventions. We consider trees of depth 1 to 12, fit to the MNIST dataset. As component trees become deeper, ensembles express more predictive diversity. To avoid binary predictions (for which our diversity metric is undefined), we pad all predictions with = 1 and renormalize.
1.2 Deep ensembles
For deep ensembles, we consider a standard deep ensemble (formed over initialization) as the base ensemble. As diversity interventions, we consider each of the diversity interventions described in Sec. with values of < 0. Note that 7 corresponds to a standard deep ensemble. As we consider more negative values y ensembles express more predictive diversity.
42

---

Published in Transactions on Machine Learning Research (01/2024)
Encouraging Diversity for a Perfect Prediction
2.00
1.75
Average single model loss (CE
1.50
+
+
1.25
+
+
1.00
0.75
geometric diversity
0.50
logit diversity
dirichlet diversity
0.25
perfect ensemble NLL infeasible
0.00
0.0
1.5 CE Jensen gap (pred. diversity)
0.5
1.0
2.0
Figure 26: Regardless of encouragement strategy, diversity impossible at extreme points. Under different methods of adding diversity to perfect ensemble prediction (geometric scaling, sampling logit space, or sampling from a dirichlet distribution), any of these strategies will degrade average single model loss more rapidly than increases in predictive diversity, leading worse ensemble performance overall.
43

---

Published in Transactions on Machine Learning Research (01/2024)
(MNIST, Random Forest)
10
Depth
Density
100
2
6
10-1
12
1.0
Counterfactuse Accuracy
0.5
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
Per-datapoint Diversity (CE Jensen Gap)
Variance (CIFARIO, ResNet 18)
JSD 1-vs-All (CIFAR10, ResNet 18)
JSD Uniform (CIFARIO, ResNet 18)
Dansity
99.0
-3.2
-1.0
10
-1.5
-1.6
-0.6
0.0
0.0
0.0
Counterlactaal
Accuracy
0.5
0.0
0.2
0.6
0.8
0.0
0.2
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Per-datapoint Diversity (CE, Jensen Gap)
Per-datapoint Divensity (CE Jensen Gap)
Per-datapoint Diversity (CE Jensen Gap)
Figure 27: For all diversity interventions, diversity encouragement decorrelates correct predic- tions. Just with Fig. 4, we find that methods which encourage greater level of predictive diversity decorrelate correct ensemble predictions, where present results here for random forests (top), as well the three diversity regularizers not shown in the main text applied to ResNet 18 (bottom).
44


Reviews:
[{"review_id": "vwdO32a4He", "reviewer": "Reviewer_dz3x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis work addressed some challenging questions that I believe to be of substantial interest to the community. Overall, I am very positive about this work and found it to be a focused but rigorous analysis. \n\n* Given the broad application of ensembling to boost the performance of classifiers in practice I think this paper addresses an important area. I think the conclusions of this work are relatively counter to the general prevailing wisdom on how deep ensembles should be constructed and trained. Therefore, I think this work provides a significant contribution. \n* The main result in my view is that model capacity is a determining factor in the value of diversity. While original research in the statistics literature developed theory around ensembles of _weak_ learners, the results in this work provide strong evidence that these intuitions do not carry over to the high capacity deep ensemble regime. This is a valuable finding.\n* The explanation of the cause of the issue (i.e. highly accurate predictions having diversity enforced causes these predictions to become inaccurate) was intuitive and well supported. I found this explanation convincing.\n* I think that this work should both (a) spark new research (e.g. developing theory around these empirical findings, investigating new approaches beyond the diversity perspective for improving deep ensemble performance) and (b) direct research away from the ineffective approach of optimizing for train set diversity. \n* In general, I thought the experiments were comprehensive and well-designed (although I have some minor suggestions on their presentation in some figures). \n* The paper was generally well-written and clear. \n* Many additional results are provided in the appendix which reinforces the points made in the main text. Code is also provided. \n\n\n**Weaknesses**\n\nI did not find any major weaknesses in this work. As a minor note, I thought that some of the figures could have been more intuitive. At times it took me quite a while to fully understand what was being shown (I provide some specific suggestions in requested changes). I would suggest that it might also be useful to provide a summary of the key takeaways in a list or a table up front. Since there is quite a lot to process in throughout the paper it might be nice to have something to anchor the reader to where they are in the paper. \n\n**Questions**\n\nI have some questions that don't fit as strengths or weaknesses. \n* Are the results provided in Fig 8 on the validation set? I presume so, but I don't think it is explicitly stated. \n* Notation: Is there a reason $\\lambda$ is used in Table 1 for regularization strength rather than $\\gamma$?\n* One thing that is a little unclear is if it is model capacity or individual model performance that dictates whether we are in the _weak learner_ regime or the _strong learner_ regime. I suspect it to be the latter but, for a fixed dataset, increasing capacity results in improved model performance. I suspect that this might be a useful point to be clear on for future work that might analyze this change of regime from a model complexity perspective. Do the authors agree with me on this? If so, it might be worth mentioning this point somewhere in the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides an empirical analysis of the role of predictive diversity in deep ensembles. Therefore, the primary contributions lie in the results of the investigation of the research questions posed. The first of these is that, when optimizing a deep ensemble, encouraging diversity is beneficial for small networks but not for larger networks and, relatedly, discouraging diversity has minimal effect on large networks but a harmful effect on smaller networks. This appears to be related to the fact that larger models tend to obtain very high accuracy and, as a result, diversity can only come at the expense of predictive performance on individually strong models. Next, the authors investigate if increasing diversity _implicitly_ (e.g. via diverse architectures) can still benefit from diversity. Even in this case, it appears that the marginal gains from diversity are still less performant than simply ensembling with individually strong but low diversity ensembles. In addition to these results, some minor contributions are provided including an analysis of the ensemble objective and the choices of diversity measure as well as a helpful overview of the existing work on optimizing for diversity including Table 2 which highlights that previous works have generally failed to obtain performance gains by optimizing for diversity on the training set.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I request that the figures mentioned be tweaked to improve their clarity. I have provided some suggestions on what might help, but I leave it to the authors to implement whatever changes they find to work best (i.e. I am not insisting on these specific suggestions). \n\n* Fig 1 - Even this relatively simple opening figure is a little tricky to \"get\".  Why is the blue/red line two colors, could it not just be blue _or_ red? One option might be an additional arrow showing the directions of loss decreasing (i.e. perpendicular to the level set line). Or some shading maybe?\n* Fig 3 - it would be nice to see the actual max and min values of $\\gamma$ used in addition. Also, the text is too small. It also seems that the plots could be zoomed in to make it easier to read the points in the plot. \n* Fig 4 (left and center left) - I found this to be the most challenging plot to understand of all (although the result it conveys is important once understood). Some factors that contributed to my confusion were: (a) counterfactual accuracy is only defined in the main text, not in the caption or the figure itself. I had no intuition for what this meant; (b) there seems to be more lines/colors in the plot than in the legend; (c) lines for density plots are a little non-standard; (d) it took me a while to notice what was the difference between the left and center-left plots (i.e. ResNet-8 vs ResNet-18).\n* Fig 5 - Again, seems to be overly zoomed out. xlims and ylims could be reduced.\n\nFinally, I have some additional suggestions. \n* Although it is not the main point of the paper, I think it would be useful for future research to include some discussion comparing the four diversity encouraging/discouraging mechanisms from page 5 (even in the appendix). Given the authors have run considerable experiments with these terms, some high-level perspective on their relative strengths/weaknesses might be useful for future research that intends to optimize for diversity (where it is suitable). \n* As mentioned in my questions, I think a summary of the paper's contributions in the introduction would help anchor the reader in the text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FAGs4ScM3k", "reviewer": "Reviewer_5PH2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is decently well written and relatively easy to read\n* The main claim seems novel (I may be wrong as I am not an expert on ensembles and deep learning).\n* After reading the paper, I would actually say that the main claim is intuitive and would represent a commonsense perspective. So, it all makes sense to me. A bit surprised that no one did this before when there is such a large body of work on deep ensembles. Can the authors perform a thorough literature survey on this in order to make sure that right credits are given?\n* The design of the empirical validation is fine, but not optimal.\n* The developed code is provided for easy reproducibility.\n\nWeaknesses\n* The paper presents facts, draws some informative conclusions, but it doesn’t explain ‘Why?’. Anyway, answering ‘why’ can be considered (as also the authors mentioned), the focus of follow up works.\n*  Several statements in the abstract and the main paper leads the reader to the misleading idea that the paper studied all deep learning architectures which is not true. A common mistake in the literature, but the reality is that Deep Learning is not equal with Convolutional Neural Networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the impact of predictive diversity on the performance of deep neural networks ensembles. It mainly focuses on convolutional neural networks and multiclass classification, while studying a bit also regression (in an appendix). If my understanding is correct, the main finding (claim) of the paper is that in the case of classification a higher predictive diversity helps deep neural network ensembles performance when each ensemble member is a very light model with a small capacity; while when each ensemble model is a very large model with high capacity the predictive diversity hurts the ensemble performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Important aspects:\n* 1) Please either change some paper statements to identify well the paper boundaries (e.g., focus and claims empirically demonstrated just on convolutional neural networks and multiclass classification for computer vision), either expand the experimental design to cover at least two extra different Deep Learning architectures and data types (e.g., MLP on tabular data, and whatever else the authors prefer).\n* 2) On page 9, the statement about regression “As a first test of these claims, we would predict that regression…” may not be fully correct (I am not sure) and can be an outcome of a flawed experimental design. Can you try the following systematic experiment: \n(i) pick a few tabular datasets for regression;\n(ii) train simple MLPs with few hidden layers on them. Increase gradually the number of hidden neurons from the hidden layers (e.g., start with a very small number like 5 neurons/layer and go up to ridiculously large numbers for those datasets) and on all these cases study the relation between predictive diversity and (ensemble) models’ performance;\n(iii) What is the relation now between predictive diversity and model capacity? Is it still not confirming your findings from the classification experiments?\nAlternatively, please consider to remove all statements about regression from the main paper and their corresponding experiments  from the appendix  (anyway, they don’t bring much value in their actual form to the main paper claims) and to make it clear from the begin that the paper focus just on classification. \n\nMinor aspects:\n* The last phrase of the abstract “These results call into question the benefit of efforts to create more diverse deep ensembles, especially in the face of an easier alternative: simply forming ensembles from ever more powerful (and less diverse) component models.” is too strong and prunes some valid research directions. Try to ponder down and rephrase a bit this phrase and similar phrases from the main paper to be more in line with your findings and to encourage constructive fundamental research. For instance, just to understand how it can negatively affect the reader and I will give you my case example. After reading the abstract, I had the feeling of a paper which just promotes extremely large networks (and huge amount of compute powers) and somehow I perceived it on the reject side. After reading the paper, I changed my mind, although I still believe that the paper needs a major revision (see my other comments).", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MCvh1xJNKk", "reviewer": "Reviewer_2UB5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors have written a nice and easy to follow paper with numerours experiments provided in the appendix and in the main text\n- The number of experiments is high, and they approach the problem from different angles, leading to the same conclusion\n\nWeaknesses:\n- I would agree with the main conclusion of the paper and with the statement that it is supported with experiments, however, we see, that usage of the variance as the diversity measure leads to better models (Figures 13-14). It means, that even larger models aren't large enough to completely ignore the diversity issues.\n- Reporting only CE loss can limit the explanatory power of the experiments, as one-hot true label is an incorrect categorical distribution for an example. In my opinion, reporting of Accuracy lead to deeper understanding and lead to better insights in what is going on (e.g. provided in Figures 12-20 vs Figures 21-23)\n- The training procedure can harm the conclusions: simultaneous training of models in an ensemble often leads to the performance degradation on average. The models should be trained indendently with little intersection over losses for them (e.g. boosting-style training). \n- Figure 17 y-axis label is wrong", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper examines the phenomenon of trading off between diversity and accuracy for ensembles of deep learning models. Experiments come to a conclusion, that encouraging diversity for strong models lead to the degradation of accuracy in ensembles. The conclusion is supported by numerous experiments including experiments for TinyImageNet and ImageNet", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Make an ablation study on the training procedure, look at the relevant procedure for training\n- Provide accuracy values as well as CE for later experiments", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are supportive of this paper, and it clearly meets the bar for acceptance.

"Minor revision" selected because the authors still need to integrate the figures provided during the rebuttal into the main pdf, and make the small changes promised in the responses. Other than those items, no changes needed.
Decision: Accept (poster)