Joint Training of Deep Ensembles Fails Due to Learner Collusion
Authors: Alan Jeffares
Tennison Liu
Jonathan Crabbé
Mihaela van der Schaar
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Ensembles
Deep Learning
TL;DR: Training deep ensembles via a shared (joint) objective results in degenerate behavior.

Abstract: Ensembles of machine learning models have been well established as a powerful method of improving performance over a single model. Traditionally, ensembling algorithms train their base learners independently or sequentially with the goal of optimizing their joint performance. In the case of deep ensembles of neural networks, we are provided with the opportunity to directly optimize the true objective: the joint performance of the ensemble as a whole. Surprisingly, however, directly minimizing the loss of the ensemble appears to rarely be applied in practice. Instead, most previous research trains individual models independently with ensembling performed _post hoc_. In this work, we show that this is for good reason - _joint optimization of ensemble loss results in degenerate behavior_. We approach this problem by decomposing the ensemble objective into the strength of the base learners and the diversity between them. We discover that joint optimization results in a phenomenon in which base learners collude to artificially inflate their apparent diversity. This pseudo-diversity fails to generalize beyond the training data, causing a larger generalization gap. We proceed to comprehensively demonstrate the practical implications of this effect on a range of standard machine learning tasks and architectures by smoothly interpolating between independent training and joint optimization.

Joint Training of Deep Ensembles Fails Due to Learner Collusion
Alan Jeffares University of Cambridge j66590cam.. ac. uk
Tennison Liu University of Cambridge t1522@camm. ac
Jonathan Crabbé University of Cambridge j jc21338cam. ac uk
Mihaela van der Schaar University of Cambridge
mv4720cam.o ac.
Abstract
Ensembles of machine learning models have been well established as a powerful method of improving performance over single model. Traditionally, ensembling algorithms train their base learners independently or sequentially with the goal of optimizing their joint performance. In the of deep ensembles of neural networks, we are provided with the opportunity to directly optimize the true ob- whole. Surprisingly, however,
jective: the joint performance of the ensemble directly minimizing the loss of the ensemble appears to rarely be applied in prac- tice. Instead, most previous research trains individual models independently with ensembling performed post hoc. In this work, we show that this is for good reason joint optimization of ensemble loss results in degenerate behavior. We approach this problem by decomposing the ensemble objective into the strength of the base learners and the diversity between them. We discover that joint optimization re- sults in phenomenon in which base learners collude to artificially inflate their ap- parent diversity. This pseudo-diversity fails to generalize beyond the training data, causing larger generalization gap. We proceed to comprehensively demonstrate the practical implications of this effect on a range of standard machine learning tasks and architectures by smoothly interpolating between independent training and joint optimization
Introduction
Deep ensembles [1] have proven to be remarkably effective method for improving performance over a single deep learning model. This success has been attributed to deep ensembles exploring multiple modes in the loss landscape [2]. in contrast to variational Bayesian methods, which gener- ally attempt to better explore single mode [e.g. 3] Recent work has proposed an alternative view suggesting that deep ensembles' success be explained being effective method of scaling smaller models to match the performance mechanism, deep ensembles' empirical success has been the heart of numerous state-of-the-an deep learning solutions in recent years [e.g. 7].
single larger model 44. Regardless of the specific
Historically, non-neural network ensembles such as bagging predictor 8 or random forest 99 were generally trained independently and aggregated at test time. Sequential training methods such as boosting (10) were also developed to produce more collaborative ensembles. Training individu- ally and evaluating jointly was sensible for these methods, in which the base learners were typically
Code is provided at http::///ghhhhc.cco//aaaa/t//eer
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

decision trees that require individual training. Formally, for a single example x, a loss function L and denoting the prediction of learner j on x as f, E R with target y E R., we define the joins objective as C M Σj= Y) and the independent objective as W ΣE= y) Interestingly, modern deep ensembles are also generally trained independently and evaluated jointly. This is less whole (i.e. the joint objective) is the true objective
intuitive as the performance of the ensemble of interest and jointly training a deep ensemble is quite natural for the backpropagation algorithm. Indeed, one might expect there be no need for the proxy objective of independent training in this case, as the joint objective should find a solution in which individual learners collaborate to min- imize the ensemble's 's loss. In practice, however, independent training generalizes better than joint training in this context. A phenomenon that we demonstrate in Section 3 and investigate throughout this work.
In Section we demonstrate that ensemble diversity is the foundation upon which understanding the limitations of joint training sits. We show that any twice differentiable joint loss function of an ensemble can be decomposed into the individual losses and a second term, which can be interpreted as ensemble diversity. This diversity term encourages some level of ambiguity among the base learners' predictions, a generalization of well-established work in the regression setting [II. Math- ematically, this term is exactly the difference between independent and joint training and, therefore, must encapsulate the disparity in performance. In Section we introduce a scalar weighting on the diversity term allowing us to smoothly interpolate between the extremes of fully independent and fully joint training.
1/2
Then, in Section 6 we diagnose and empirically verify precise cause of the sub-optimal performance of jointly trained ensembles. We show that the additional require- ment for diversity in the training objective can be trivially achieved by an adverse effect we term learner collusion. This consists of a set of base learners artificially biasing their outputs in order to trivially achieve diversity while ensuring that these biases cancel in aggregate as illustrated in Figure []] We perform extensive experiments verifying this hypothesis. We then show that although training loss can be driven down by learner collusion, this does not gen- eralize beyond the training set, resulting in larger gener- alization gap than independent training. Finally, we exam- inc the practical implications of learner collusion stan- dard machine learning benchmarks.
fs
S2
S3 Learner collusion
f1
Learner collusion. An illus- trative example in which a subset of base learners (f1 & f2) collude in output space by shifting their predictions in opposing di- rections such that the ensemble prediction (f) is unchanged, but diversity is inflated.
Figure 1:
This work addresses the pervasive yet relatively understudied phenomenon that directly optimizing a deep ensemble fails. (1) We unify various strands of the ensemble literature to provide a deep, practical categorization of ensemble training and diversification. (2) We present and verify a com- prehensive explanation for this counter-intuitiv limitation of deep ensembles with both theoretical and empirical contributions. (3) We investigate the catastrophic consequences of this issue on practi- cal machine learning tasks. As this investigation takes a somewhat non-standard format, we provide an abridged summary of this work and its technical contributions in Appendix
2 Background
Independent training. This refers to ensembles in which each base learner is optimized indepen- dently, with no influence from the other ensemble members either explicit or implicit. This includes many non-neural ensemble techniques in which diversity between the base learners results in ensem- ble performance better than any individual. This diversity is often achieved by sampling the input space [e.g. bagging in g and the feature space [e.g. random forest in [9] In the case of deep en- sembles of neural networks, independent training is also generally applied, with diversity achieved using random weight initialization and stochastic batching 2 Interestingly, bootstrap sampling of the inputs is ineffective in the case of deep ensembles [12] Several notable state-of-the-arr results
In Section 4 we further generalize this definition account for (a) non-equal weighting between the predic- tions of ensemble members and (b) aggregating ensemble outputs before normalization (i.e. score averaging). 'In this work we consider ensembles in the least prescriptive sense (i.e. any algorithm that aggregates multiple target predictions).

---

have been achieved by ensembling several independently trained but individually performant neural networks. For example, the groundbreaking results in the ImageNet challenge were often achieved by independently trained ensembles such as VGG [13] GoogLeNet [5), and AlexNet [14]. Seq2Seq ensembles of long short-term memory networks [15] achieved remarkable performance for neural translation. More recently in the tabular domain, ensembles of transformers have outperformed
state-of-the-ar non-neural methods 77).
Gnsemble-aware individual training. This refers to the case where each base learner is still being optimized as an individual, but with some additional technique being applied to coordinate the en- semble members. A prominent approach is sequentially trained, boosting ensembles in which each new member attempts to improve upon the errors of the existing members (e.g. AdaBoost [TO] & gradient boosting [16]). In the case of deep ensembles, an alternative approach consists of training each base learner with a standard individual loss with the addition of an ensemble-level regulariza- tion term designed to manage diversity. This often consists of an f-divergence hetween the learners' predictive distributions. Examples include the pairwise X² <-viiergeecce [77, K LT-divergence, but between the learner and the ensemble (18), and computationally efficient, kernelized approxima- tion of Renyi-entropy [9]. Another prominent regularization approach is based on the seminal idea of negative correlation learning (NCL) which penalizes individual learners through correlation penalty on error distributions [20 21.. This penalty weakens the relationship with other learners and controls the trade-off between bias, variance, and covariance in ensemble learning [22] Recent works have adapted NCL for improved computational efficiency with large base learners [23] and attempt to generalize NCL beyond the regression setting [24] 25 encourages ensemble diversity in weight and functional space using a "repulsive" term while other works show that a single learner can be viewed as ensemble with diversity encouraged implicitly (e.g. 26 27).. The efficient BatchEnsemble [28) can also be characterized as an ensemble-aware individual training approach.
Joint training. This refers to the case where the ensemble's 's predictions are optimized directly as if it were a single model. Intuitively, one might naturally expect joint training to outperform the previously discussed methods as it directly optimizes (at training time) the test metric of all three approaches ensemble loss. Empirically, however, this is not the case as joint training achieves inferior performance in general. [29] reported poor results when joint training averages the base learners' output scores but excellent performance when they are averaged at the log- pog-pobbabylity or loss levels. However, later work showed that the two alleged success cases were, in fact, implement- ing independent training rather than joint training unbeknownst to the authors [30]. This later work also reported poor generalization for score-averaged ensembles. We provide detailed clarifica- tion and gradient analysis in Appendix E Elsewhere, (31] experienced poor performance from joint training for both score and probability averaging. Concurrently to this work, [32 33 also found that joint training consistently resulted in worse empirical performance. To put the matter to rest, we demonstrate these issues with joint training in the next section.
3 Empirically Evaluating Joint Training
In this section, we briefly demonstrate the effect that we investigate throughout this work. That is, when evaluating the ensemble loss at test time, optimizing for that same ensemble loss objective at training time (i.e. joint training) is less effective than optimizing each of the base learners indepen- dently (i.e. independent training). We do this by performing a straightforward comparison of the two methods. We compare the test set performance of independent and joint training on ImageNet [34]. We consider several architectures as base learners of an ensemble of various sizes determined by computational limitations. We report test set top-1 and top-5 accuracy for both independently and jointly trained ensembles. These results are included in Table Complete experimental details for all experiments in this work are provided in Appendix B
Here, observe the counter-intuitive phenomenon that independently trained ensembles consis- tently outperform jointly trained ensembles, despite all models being evaluated using the joint ob- jective. This is consistent with previous works that have attempted apply joint training (30] 31. In the following sections, we will proceed to investigate and explain this effect through the lens of ensemble diversity.
Summary: Joint training performs considerably worse than independent training in practice.
3

---

Table 1: Joint training on ImageNet. We compare independent joint training across a range of architectures trained from scratch on the ImageNet dataset. Top 1 & 5 test accuracy is reported for both methods, where independent training consistently performs better.
Number of learners
Independent
Joint
Base learner
Top-1
Top-5
Top-1
Top-5
ResNer- 18
35
5
68.10%
88. 88.43%
62.74%
84.03%
ResNet-34
35
3
73.11%
91.25%
68.91%
88.01%
DenseNet [36]
3
70.62%
89.51%
60.81%
81.35%
ShuffleNet V2 (0.5x)
15
10
61.00%
81.94%
53.41%
74.88%
ShuffleNet V2 (1x)
37
5
68.56%
87.64%
61.96%
81.56%
SqueezeNet V1.1
38
5
56.25%
78.65%
42.69%
65.24%
MobileNet V3-s
339
5
67.43%
86.82%
57.34%
78.30%
Vib
40
61.29%
82.84%
54.69%
76.79%
4 Diversity in Ensemble Learning
To understand the limitations of joint training, we will need to examine the joint objective through the lens of diversity. In this section, argue for unified definition of diversity which generalizes much of the existing literature on the subject.
4.1 Preliminaries
We begin with some notation. We consider ensembles made up of M individual learners. The j j'th individual learner maps input x E R to d-dimensional output space such that fy RR - Bd and attempts to learn some regression target(s) y E Rd or, in the case of classification, labels which lie probability simplex A Without loss of generality,
refer to y V. We primarily consider combining individual learner predictions into a single ensemble prediction as a weighted average _== (x){((x) f(x) where Wj E R denotes the scalar weights. Although not strictly required for much of this work, we will generally assume that these weights satisfy the properties
EE=1 wy 1 and w 0 V j. ensuring that each individual learner can be interpreted as predicting the target. Scalars are denoted in plain font and vectors in bold. We generally drop the dependence on in notation such that individual and ensemble predictions are denoted f, and f respectively. Finally, we formalize the respective losses of the ensemble and the base leamers in Definition 4.1
Definition 4.1 (Ensemble and Individual Loss). For a given loss function L )22 + R+ we define:
(a) The overall ensemble loss ERR == y(f.y).
(b) The weighted individual loss of the base learners ERR = EM W yf)
Intuitively, ERR measures the predictive performance of the ensemble, while ERR measures the aggregate predictive performance of the individual ensemble members.
4.2 A Generalized Diversity Formulation
Diversity in regression ensembles. The prevailing view diversity regression ensembles began in [1I]. which showed that the mean squared error of regression ensemble can be decomposed into the mean squared error the individual learners and an additional ambiguity term which measures the variance of the individual learners around the ensemble prediction showing
En5(t f)² =}== Σ
(fs - -²
(1)
DIV ERR ERR.
This decomposition provides an intuitive definition of diversity whilst also showing that a jointly trained ensemble, that is one that simply optimizes ERR, encourages diversity by default due to it being baked into the objective. Later work from the evolutionary computation literature proposed an apparently alternative measure of diversity called Negative Correlation Learning, which argues

---

that lower correlation in individual errors is equivalent to higher diversity [21] The apparent dis- agreement between these two seemingly valid views on diversity was resolved when it was shown that they are in fact equivalent up to a scaling factor of the diversity term [22]
Generalized diversity. Motivated by the regression case, we will argue that the difference between the individual and ensemble errors has a natural interpretation as the diversity among the predictions fact, this gen-
of the ensemble members for any twice (or more) differentiable loss function. eralizes several existing notions of diversity proposed throughout the literature for specific cases. Specifically, we define diversity as in Definition 4.2 and will proceed to justify this definition in what follows.
Definition 4.2 (Diversity). The diversity of an ensemble of learners (DIV) is the difference between the weighted average loss of individual learners and the ensemble loss:
DIV == ERR ERR
Diversity in classification ensembles. It is natural to next verify that this definition is appropriate in the classification setting. We proceed by concretely deriving the diversity term in this setting. There two standard methods for aggregating base learners' predictions into a single ensemble predic- tion in classification. Averaging at the score (preactivation) level and averaging at the probability level. We address each of these in turn.
! Score averaging. We consider classification task with target distribution y E e E A with d classes, and consider ensembling by averaging the base learners' scores, before applying the nor- malizing map & (typically a softmax). We define Pj : X' - A Φ ff as the normalized pre- dictive distribution for learner j and q á f the predictive distribution for the ensemble, where f - W as before. this setting, can be shown [4] that
M
M
DIV = Σ wy wyDKk(yylp,) DKz(yyl)) Σ wy Dxz
where DKL is the K Liiveergeece Intuitively, this diversity term measures the weighted diver- gence between each ensemble member's predictive distribution and the full ensemble's ' predictive distribution, a sensible expression for diversity. It should be noted that this decomposition holds for normalizing functions beyond the softmax [30]. Specifically, this exact expression holds for all exponential family normalization functions with score averaging. This allows the diversity decom- position described in this work to apply to disparate settings as demonstrated by its recent application in modeling diversity among the Poisson spike train outputs of spiking neural networks 42
② Probability averaging. This setting diverges from score averaging by setting the ensem- ble predictive distribution p to be the average of the base learners' probabilitiess such that p Σラ= Wy (ő fj). This is more natural strategy, averaging the scale-invariant probabilities while also providing each base learner with unique gradients during backpropagation (see detailed gra- The resulting expression for diversity is included in Theorem 4.3
M
dient analysis in Appendix
Theorem 4.3. For a probability averaged ensemble classifier, trained using cross-entrop loss Lee with 1-t E K. denoting the ground truth class, diversity is given by
Σ w y(k)) log p(k-)) j=1
M
DIV Σ
w.Locly. P,) Lce(y, P)
pj (k+)
Proof. Please refer to Appendix C.3
Remark 4.4. It is not immediately obvious that this expression has natural interpretation di- versity. However, in Appendix C.3 we show it to be well aligned with the notion of diversity used throughout this work by analysis of the distribution of p, (k*) around p(k*).
Beyond the KL-divergence family, alternative loss functions also result in sensible diversity expres- sions. Brier score, expressed as (p- y) (p - y), can be shown to obtain DIV = E, "y p) P,), the variance of the base learner probabilities around the ensemble probabilities (4]
5

---

For 0-1 loss and the case of majority voting, diversity is also measured by the disagreement between the (categorical) predictions of the base learners and the ensemble prediction 4
Diversity in a general setting. The previous paragraphs verified the diversity definition introduced in Definition 4.2 to be sensible interpretation for several common loss functions. A more general question can be asked - Is DIV reasonable for any choice of loss fimctions? The answer is yes, and this turns out be no coincidence. To demonstrate why, we analyze the properties of the diversity expression for any twice differentiable loss function and discover a general form that permits a clear interpretation as diversity. To do so, we present Theorem 4.5) multidimensional generalization of one proposed in [44] which, in turn, generalizes our notion of diversity and analysis of joint training to practically any loss function.
Theorem 4.5. [Diversity for General Loss] For any loss function L 322 - Rt with dimn)") E N that is least twice differentiable, the loss of the ensemble can be decomposed into
DIV Σ wy (f) FTT- (T7(7),) -
1
(t) (f)-f) f)
j=1
with F f cf,(( - f),
where Hfy the Hessian of & with respect to its first argument and some cj E [0, for [ol [M].
Proof. Please refer to Appendix C.2
Here, the measure of diversity is obtained by performing a Taylor expansion on the loss of the output of f, near the ensemble output f and using Lagrange's remainder theorem to obtain an exacr equality. Additionally, f is an uncertain number, which approaches a constant in the limit. This diversity admits the same interpretation as the variance of learners around the ensemble, but is now weighted by term dependent on the second derivative of the loss function.
One can easily verify that this generalized definition encompasses the uni-dimensional MSE ambi- guity decomposition in Equation (I). where ("f",, y) = 2. Similarly, the multidimensional MSE y(f.y) = |ff 1if-ylli y1 leads to the Hessian 7 2I, where is the identity matrix on V. From this, we get the multi-dimennionall generalization DIV = Σ, wyllf, f]]2 2 0. Importantly, diversity has the same non-negative property (DIV 2 0) for any loss function with semi- mositiositiv definite Hes- sian Kf ž= O. such that ensemble performance is always better than individual learner performance. Another typical example is the classification loss C(f, y) = -y log f. Here, is trivial to show that H (f, diag(y 2 f2), where @ denotes the component-wise division. Again, this Hessian matrix positive semi-definite for y. E A so that DIV 2 Revisiting our previous question, we verify that, for least twice- diicediiffrentiable loss functions (e.g. logistic loss: L = log(1 + exp vf) & exponential loss: L = exp "/), DIV admits a generalized interpretation as the variance of indi- vidual predictors around ensemble prediction. This makes robust case for Definition 4.2 which provides a principled estimate of diversity (as opposed to heuristic measures) and can be easily obtained by calculating the difference between ensemble loss and weighted average learner loss.
Summary: The difference between independent loss and joint loss has a natural interpretation as diversity, providing a robust generalization of much of the existing literature.
5 An Augmented Objective
Given this generalized definition of diversity, one might naturally consider placing a scalar weighting on its contribution, resulting in an augmented training objective. In fact, such an approach has already been applied for special cases of diversity including regression [22), score averaging (30),
and, concurrently to this work, in [3 32). Throughout our experiments, we will consider this
augmented objective Equation (2). allowing us to analyze the effects of smoothly transitioning between independent and joint training.
L (f. = ERR B DIV
(2)
= 3) HRR + 3 ERR.
When 3 = 0. this amounts to training each learner independently (i.e. independent training) and 3=1 1 corresponds to optimizing the loss of the ensemble as a whole (i.e. joint training). Clearly, intermediate values of 3 can be interpreted as interpolating between the two.

---

When examining Equation 2 one might be tempted to encourage more diversity by setting 3 >> 1. Previous work by Brown al. 4 showed that, in the case of regression, optimization can become degenerate for larger values of B. proving an upper bound of [ < M-I- More recently, (30 showed that in the case of score averaged ensembles using their modular loss, the ensemble loss ERR was unbounded from below for 8 > 1 1-i.e. i.e. DIV can trivially be increased faster than its corresponding increase in ERR, thus (superficially) minimizing the loss. In Theorem 5.1 we generalize this same bound on 3 to all loss functions and further include the case of probability-average ensembles.
M
Theorem 5.1. [Pathological Behavior for 3 > 1/ We consider a target set y = R' or y C Rd being compact and convex subset of R.. Let L ) + Rt be continuous and finite function on the interior in²) We assume that the loss is not bounded from above on y in such a way that there exists y int(Y) and sequence (Ye)teN y/ such that L(y,,y) +00 for - co. If 3 1, then the augmented loss L defined in Equation (2) is not bounded from below on ya.
Proof. Please refer to Appendix C.4
Remark 5.2. Note that we consider the case where y is a compact and convex subset of Rd to include classifiers, for which the target set a probability simplex Ad-1.
As a consequence of the above theorem, one needs impose the tighter bound 3 < 1 in order to have a well-defined optimization objective. Furthermore, intermediate values where 3 E (0, 1) may have an additional interpretation beyond interpolating between independent and joint training. In Appendix De we show that, in the regression setting, L is exactly equivalent to applying independent training while adjusting the targets to account for the errors of the ensemble as measured by its gradient. We derive exact equivalence between B in L and the magnitude of the adjustment of the target this gradient-adjusted target setting.
Summary: An interpolation between independent training and joint training is well-motivated and useful objective but, avoid pathological behavior, one should not extrapolate to higher diversity.
6 Why Does Joint Training Fail?
Given the universal interpretation of diversity developed in the previous sections paired with its ex- clusive impact on the joint training objective, we now turn our attention to answering the question why does joint training of deep ensembles perform worse than independent training? Some pre- vious works have attempted to partially address this question. [29] suggested that score averaging fails as the softmax function "distorts" the averages. However, little evidence exists to support this claim. When faced with the same issue, [3]] suggested the issue could be due to score averaged ensemble members receiving identical gradient values. We address this point in a gradient analysis in Appendix [. where we (a) point out that the optimal solution to joint objective would still achieve equal or lower loss for score averaging, and (b) show that probability averaged ensemble members receive variable gradient expressions and still suffer from poor performance. Finally, (30) noticed vastly different test set performances among the base learners which they referred to as model dom- inance. Indeed, this is symptom of the underlying issue with joint training which we describe and verify throughout the rest of this section. The explanation we propose for the poor empirical results achieved by joint training consists of two claims:
Claim 1: the additional diversity term in the joint learning objective (Equation (2) results in a dual objective. The DIV term can always be artificially inflated if multiple learners collude to bias their predictive distributions in opposing directions which cancel out in aggregate and, therefore, is easily maximized across the training data. We refer to this phenomenon learner collusion.
Claim 2: while learner collusion can still minimize the loss on the training data, this solution may contain little genuine diversity resulting in a solution that achieves worse generalization than inde- pendent training. In Appendix [F] we provide a detailed illustrative example of learner collusion for the regression case from an optimization perspective.
In Section 6.1 we empirically demonstrate this learner collusion effect emergent. Then, in Sec- tion 6.2 we show that the low training loss achieved by learner collusion generalizes poorly onto unseen data. Finally, in Section 6.3 we establish the practical implications of learner collusion on standard tasks.
7

---

6.1 Diagnosing Learner Collusion
In this section, we focus on claim and empirically investigate the extent to which learner collusion does occur. Recall that the effect being investigated that subset non-diverse base learners can trivially adjust their similar, or even identical, predictions to appear diverse (i.e. Figure D If learner collusion does indeed take place we suggest that the following three effects should be observable empirically: (1) the relative magnitude of diversity should be excessive relative to the ensemble loss, (2) if we remove constant output biases from the individual learner, the ensemble diversity should decrease significantly, and (3) learners should become highly dependent on each other such that removing a small subset of base learners is detrimental to the ensemble performance. We investigate each of these effects in turn.
We perform these experiments on four tabular datasets of the style upon which ensemble methods typically excel [46]. Here we focus on regression as it offers the most controlled setting to isolate the learner collusion phenomenon (see Appendix [T) but in later experiments we demonstrate its effects in the classification setting too. We smoothly interpolate between independent and joint training using the augmented objective controlled by | from Section [ We report mean + one standard deviation evaluated on a test set throughout. An extended description of the experimental protocol for these experiments is provided in Appendix
Diversity explosion. We begin with the most simple expected effect of learner collusion, that the relative magnitude of diversity is excessive with respect to the ensemble loss. The results are in- cluded in Figure 2 where we note a sharp increase in diversity for the joint objective. Specifically, note that mean test set diversity remains relatively low for most values of 3 but quickly explodes as B + 1 to levels that are often significantly greater than the mean squared error of the ensemble predictions. This indicates that the base learners' variation around their own mean much greater than the ensemble's 's variation around the label.
Bioconsentration
Weather
Facebook
Abalone
0.278
10111
2.623
0.431
MSB
0.281
0.11
2.332
0.174
0.264
00 108
2.04
0.316
Diveralty
1.28
456
2.066
0.755
0.638
1.067
0.379
0.019
0.068
0.6
0.0
0.8
9
/
Figure 2: Diversity explosion. Test set diversity explodes relative MSE across four datasets. We note two further empirical phenomena: (1) the non-linear relationship between 3 and diversity which discuss further in Appendix and (2) the minimal effect diversity has on the test MSE on the Facebook dataset which is discussed in Appendix
Debiased diversity. We now proceed to the second postulated effect of learner collusion. Although learners could perform learner collusion on per-example basis, the most straightforward form of this behavior would be learners that bias their predictions without any dependence on the example at hand. To investigate the extent to which this trivial form of collusion does occur we apply post hoc debiasing to each of the individual learners in the ensemble which affects the trade-off between ERR and DIV without affecting ERR. Specifically, we adjust learner j's predictions such that
pdcbiased (xi) = fz(x;) - bj
N
1 1 N F(x)
fx,(x,
(x,)
N
In Appendix C.I we prove that the debiasing term by is the optimal diversity minimizer that keeps the ensemble predictions unchanged. In Figure 3 we show that applying this technique significantly reduces the ensemble's 's apparent diversity to far more reasonable values. In general, we find that this typically removes around half of the ensemble's diversity at the joint training region (3 = 1). While example-dependent collusion could also emerge, these results already indicate that a significant proportion of the apparent diversity in jointly trained ensembles due to an artificial biasing of outputs.
8

---

Weather
Skilkraft
Facebook
Abslone
61
44
67
% &.veniity
Denesp
0-
0.0
1.0 0.0
1.0
0.0
1.0
0.0
1.0
Figure 3: Bias removal. The percentage of diversity explained by the most simple form of learner collusion across each dataset. the region where learner collusion appears to occur (3 + 1). large proportion of the diversity can be explained by simple additive bias in the predictions.
Learner codependence: The final effect we predicted emerge due to learner collusion is an exces- sive codependence among the base learners. If individual learners are engaging in learner collusion, the removal of subset of these learners test time should be catastrophic the performance of the ensemble. This is because a learner producing excessive errors needs to be counterbalanced by other leamners performing similar errors in the opposite direction, otherwise the ensemble predictions will be influenced by the poorly performing base learners. To test this we consider dropping fraction of the individual learners at test time and evaluate the resulting ensemble performance. This similar to experiments performed in [30]. although from a very different perspective. In Figure T we present the results of randomly applying various levels of test time base learner dropout 100 times for each model and report the average resulting increase in test set error. These results clearly show that such a procedure is detrimental to ensemble performance at precisely the levels of which we have hypothesized learner collusion to occur, while having only minimal impact elsewhere.
Weather
Skilleract
Facebonx
Abelone
locresse in tos esror
0.17
With 0.2 drepont
0.29
0.17
0.16
dropout With UE dropout
0.21
0.14
0.21
0.08
0.2
0.1
0.0
0.8
1.0
0.2
0.1
11.0
0.8
1.0
0.2
0.1
0.8
1.0
0.2
0.1
0.0
0.8
1.0
Figure 4: Learner codependence. Applying various levels of base learner dropout test time and observing the relative increase in MSE. As + 1, dropping a subset of learners causes greater increase in test error suggesting the occurrence learner collusion.
Summary: Extensive empirical evidence confirms that training with the joint objective results in learner collusion.
6.2 Learner Collusion Generalizes Poorly
Given that we have shown that learner collusion emerges from joint training, now proceed to examine the second claim. That is, although learner collusion does not prevent the training loss from being minimized, the solution it obtains will generalize more poorly than independently trained ensemble with non-trivial diversity. This claim is more straightforward to investigate and only requires comparison of the difference between the training and testing loss curves over the course of training. The difference hetween the two, which we refer to as the generalization gap, captures the drop in performance when model transitions from training to testing data.
For this experiment, we return to the more stan- dard machine learning benchmark of classification on the CIFAR tasks as introduced in Section 3 On each dataset train joint model (ERR) and an independent model (ERR) and report the generaliza- tion gap of ensemble loss for both methods over the
CIFAR-10
CIFAB-100
Geoeralantion
course of training. We repeat this six times and in-
loiit
Indipunkat
clude shaded region representing +2 standard er- rors. The results are included in Figure 5 where we consistently find th joint training displays a signifi- cantly larger generalization gap as training proceeds, This strongly suggestive of a much greater ten- dency towards overfitting in the jointly trained model while the independently trained model's generaliza- tion gap plateaus at a much lower level. We provide further analysis of the distinct training and testing curves in Appendix
Epoch
Figure 5: Generalization gaps. Joint training re- sults in a larger generalization gap (C (f. Chrin (f,y)) than independent training, despite both methods reporting test performance using the joint objective.
Summary: Learner collusion caused by joint training (ERR) results in greater overfitting exhibited by larger generalization gap.

---

6.3 Practical Implications
We have now described the phenomenon of learner collusion caused by the joint training of deep ensembles, empirically verified its existence, and demonstrated how it results in poorer generaliza- tion. We complete this analysis with an empirical investigation of how learner collusion manifests in the ensemble performance standard machine learning tasks. We use the augmented objec- tive, _ as described in Section 5 to examine the effects smoothly increasing the level of joint training in the objective validation performance of the ensemble and base learners throughout training on ImageNet. Results provided in Figure [ We also provide extensive additional results in Appendix G confirming the generality of this degeneracy. There we consider: larger ensembles, score-averaging, alternative models, additional datasets, and ERR as metric. Complete experimental details provided in
100 with ResNet architectures. We also examine the
Appendix B
CIEAR-10
CIFAR-100
TrangoNet
TmagyNNo
0.90
0.715
0.6
0.6
Aocurcy
0.700
Accurazy
0.94
0.4
07225
Ensemblle B =D
0.50
ReaNed-34
ResNet-18
0.200
Bus lummrr
Bovo leanness 8
Fnaembie
0.5
0.5
100
Level of diversity
Training iteration
Figure 6: Practical implications of learner collusion. Left:: Test accuracy with increasing diversity in the objective CIFAR-10/100 with ResNets. Right: Validation accuracy throughout training ResNet- 18 base learners on ImageNet. Jointly trained ensembles (8 1) perform significantly worse than independently trained ensembles (3 = 0).
Consistently across all experiments, we find that joint training (8 1) results in the worst test set performance. We also note the poor performance among base learners in this regime, highlighting that inflating diversity comes at the cost of degenerate individual performance. Interestingly, across all of our experiments learner collusion typically emerges as B > 1 implying that partial joint training may be feasible direction for future work. However, in general, independent training emerges as a primary choice of training strategy for deep ensembles.
7 Conclusion
In this work, we have provided detailed account explaining the unexpected observation that jointly trained deep ensembles generally outperformed by their independently trained counterparts de- spite joint training being the objective of interest at test time. We presented a comprehensive view of diversity in machine learning ensembles through which analyzed the joint training objective. From this perspective, we uncovered a learner collusion effect in which non-diverse base learners collude to artificially inflate their apparent diversity. Empirically, both isolated this phenomenon and demonstrated its catastrophic effect on model generalization. Finally, the practical consequences of this limitation were exposed for standard machine learning settings. Note that also provide a visual summary of this work and its technical contributions in Appendix A We hope that future work will investigate how to optimize deep ensembles to operate collaboratively whilst avoiding the degeneracies of learner collusion. In Appendix G.7 we include a negalive resull of our initial attempts in this direction. However, several promising avenues remain open (c.g. jointly optimizing worst-case bounds instead [7 48).
10

---

Acknowledgments
We would like to thank Alicia Curth, Fergus Imrie, and the anonymous reviewers for very insightful comments and discussions on earlier drafts of this paper. AJ gratefully acknowledges funding from the Cystic Fibrosis Trust. 'T. funded by AstraZeneca. JC funded by Aviva. This work was supported by Azure sponsorship credits granted by Microsoft's Al for Good Research Lab.
References
[1] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. Advances neural information process- ing systems, 30, 2017.
[2] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: loss landscape perspective. arXiv preprint vrxiv:99/.0020000 2019.
[3] Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
[4] Taiga Abe, E Kelly Buchanan, Gcoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? arXiv preprint arXiv:2202.0660 2022.
[5] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[6] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965- 23998. PMLR, 2022.
[7] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 18932-18943, 2021.
[8] Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
[9] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
[10] Yoav Freund, Robert E Schapire, et al. Experiments with a new boosting algorithm. In icml, volume 96, pages 148-156. Citeseer, 1996.
[11] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 1994.
[12] Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep en- sembles not better? In "I Can't Believe It's Not NNurrlP 2020 workshop, 2020.
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90 2017.
[14] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 10t.:::1409.5555000 2014.
[15] Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[16] Jerome Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.

---

[17] Hendrik Alexander Mehrtens, Camila González, and Anirban Mukhopadhyay. Improv- ing robustness and calibration in ensembles with diversity regularization. arXiv preprint arXiv:2201. 10908, 2022.
[18) Xiaohui Zhang, Daniel Povey, and Sanjeev Khudanpur. A ing method for deep learning. In Sixteenth Annual Conference of the International Speech Communication Association, 2015.
ensemble train-
[19] Hong-Jie Xing and Xi-Zhao Wang. Selective ensemble of svdds with renyi entropy based diversity measure. Pattern Recognition, 61:185-196, 2017.
[20] Bruce E Rosen. Ensemble learning using decorrelated neural networks. Connection science, 8 (3-4):373-384, 1996.
[21] Yong Liu and Xin Yao. Ensemble learning via negative correlation. Neural networks, 12(10): 1399-1404, 1999.
[22] Gavin Brown, Jeremy I. Wyatt, Peter Tino, and Yoshua Bengio. Managing diversity in regres- sion ensembles. Journal of machine learning research, 6(9), 2005.
[23] Zenglin Shi, Le Zhang, Yun Liu, Xiaofeng Cao, Yangdong Ye, Ming-Ming Cheng, and Guoyan Zheng. Crowd counting with deep negative correlation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5382-5390, 2018
[24] Sebastian Buschjäger, Lukas Pfahler, and Katharina Morik. Generalized negative correlation learning for deep ensembling. arXiv preprint arXiv:2011.02952 2020.
[25] Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34:3451-3465, 2021.
[26] Pierre Baldi and Peter J Sadowski. Understanding dropout. Advances in neural information processing systems, 26, 2013.
[27] Alan Ieffares, Tennison Liu, Jonathan Crabbé, Fergus Imrie, and Mihaela van der Schaar. Tangos:
Regularizing tabular neural networks through gradient orthogonalization and special- ization. arXiv preprint arXiv:2303.05550 2023.
[28] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning, arXiv preprint arXiv:2002.066715 2020.
[29] Anuvabh Dutt, Denis Pellerin, and Georges Quénot. Coupled ensembles of neural networks. Neurocomputing, 996:346-357 2020.
[30] Andrew Webb, Charles Reynolds, Wenlin Chen, Henry Reeve, Dan Ilieseu, Mikel Lujan, and Gavin Brown. To ensemble not ensemble: When does end-to-end training fail? In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 109-123. Springer, 2020.
[31] Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why heads better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.0331 2015.
[32] Taiga Abe, E Kelly Buchanan, Geoff Pleiss, and John P Cunningham. Pathologies of predictive diversity in deep ensembles. arXiv preprint 20XX::23322000044 2023.
[33] Taiga Abe, E Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. The best deep ensembles sacrifice predictive diversity. In I Can't 't Believe It's Not Better Workshop: Under- standing Deep Learning Through Empirical Falsification, 2022.
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11::2112252, 2015.
12

---

[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- lion, pages 770-778, 2016.
[36] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[37] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guide- lines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116 116-131, 2018.
[38] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint 16Xiv:6602.07366 2016.
[39] Andrew Howard, Mark Sandler, Grace Chu, Liang- Chieh Chen, Bo Chen, Mingxing Tan, Wei- jun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEF/CVF international conference computer vision, pages 1314-1324, 2019.
[40] Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.021720 2021.
[41] Tom Heskes. Selecting weighting factors in logarithmic opinion pools. Advances in neural information processing systems, 10, 1997.
[42] Georgiana Neculae, Oliver Rhodes, and Gavin Brown. Ensembles of spiking neural networks. arXiv preprint M.Ki::2010.161111 2020.
[43] Gavin Brown and Ludmila Kuncheva. "good" and "bad" diversity in majority vote ensembles. In International workshop on multiple classifier systems, pages 124-133. Springer, 2010.
[44] Zhengshen Jiang, Hongzhi Liu, Bin Fu, and Zhonghai Wu. Generalized ambiguity decomposi- tions for classification with applications in active learning and unsupervised ensemble pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[45] Gavin Brown, Jeremy Wyatt, Rachel Harris, and Xin Yao. Diversity creation methods: survey and categorisation. Information fusion, 6(1):5-20, 2005.
[46] Léo Grinsztajn, Edouard Oyallon, and Gaäl Varoquaux. Why do tree-based models still out- perform deep learning on tabular data? arXiv preprint arXiv:2207.08811, 2022.
[47] Andrés Masegosa, Stephan Lorenzen, Christian Igel, and Yevgeny Seldin. Second order pac- bayesian bounds for the weighted majority vote. Advances in Neural Information Processing Systems, 33:5263-52777 5273, 2020.
[48] Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural network ensembles. In International Conference on Artificial Intelligence and Statistics, pages 11720-11743, PMLR, 2022.
[49] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[50] Arthur Asuncion and David Newman. Uci machine learning repository, 2007.
[51] Francesca Grisoni, Viviana Consonni, Sara Villa, Marco Vighi, and Roberto Todeschini. Qsar models for bioconcentration: is the increase the complexity justified by more accurate pre- dictions? Chemosphere, 127:171-1799 2015.
13

---

[52] Dongjin Cho, Cheolhee Yoo, Jungho Im, and Dong- Hyun Cha. Comparative assessment of various machine ceaning-based bias correction methods for numerical weather prediction model forecasts of extreme air temperatures in urban areas. Earth and Space Science, 7(4): 22019EA000740, 2020.
[53] Sérgio Moro, Paulo Rita, and Bernardo Vala. Predicting social media performance metrics and evaluation of the impact brand building: A data mining approach. Journal of Business Research, 69(9)3341-33511 2016.
[54] Samuel George Waugh. Extending and benchmarking Cascade-Core Correlation: extensions to the Muscade-Correlatd architecture and benchmarking of feed-forward supervised artificial neural networks. PhD thesis, University of Tasmania, 1995.
[55] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6982. 2014.
[56] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[57] Milton Abramowitz, Irene A Stegun, and Robert H Romer. Handbook of mathematical func- tions with formulas, graphs, and mathematical tables, 1988.
[58] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
14

---

A Paper Summary
Background Section
Categorizes the three forms ensemble training: independent truining. ensemble-awure individual training and joint training (the focus of this work) and relates these concepts existing literature.
Section contrihutions
Technical contributions
Empirically Evaluating Joint Training- Section 3
Empirically demonstrates the surprising effect in- vestigated throughout this work- the joint training of deep ensembles fails in practice.
Results of empirical comparison between the two objectives in Table
An expression for diversity in the case of probability averaging in Theorem [.
Diversity in Ensemble Learning Section[4] [
A general expression for diversity for any twice differentiable loss function in
Introduces and justifies unified definition en- semble diversity which understanding the limitations of joint training due to its role in the joint objective.
foundation for
Thenrem 1.
A tight upper bound on diversity in the
learning objective in Theorem [5.]]
An Augmented Objective Section 5
equivalence between the augmented objective L and gradient adjusted
Describes an objective that acts as an interpolation between independent training and joint training and shows this to be well-motivated and a useful tool later analysis.
independent training in Appendix [
A gradient analysis of ensemble aggregation methods in Appendix B
Why Does Joint Training Fail?- Section 6
Empirical discovery of learner collusion in
Hypothesizes explanation for the failure of joint training learner collusion. Comprehensively demonstrates the existence, generalization conse- quences and practical implications of this effect.
Figures
& 4
Demonstration of generalization
consequences in Figure [
Investigation of collusion implication on practical benchmark tasks in Figure 6
Figure 7: A condensed description of this work and its main contributions.
B Experimental Details
This section compares training using cross- entropy loss as the objective applied to each learner's predictions independently (independent train- ing) and to the ensemble's aggregated predictions (joint training). As described in the main text, we compare the test set performance of these two training methods using various popular archi- tectures on ImageNet [34] evaluating the joint objective on the validation set. Specifically, we use the architectures cited in Table I with ensemble size determined by computational limitations. We use implementations from 49 except for Mobile where the implementation is from 44)]. Base learners' are aggregated using their probability outputs (i.e. probability averaging Section 4.2) We use the standard ImageNet training- vaaidnngvlliittionn splits. For all models, we adjusted the following hyperparameters to optimize performance: learning rate, learning rate scheduler (and corresponding decay factor, gamma and frequency), momentum, and weight decay. Batch size was typically set to 128 but was divided by a multiple of 2 for larger base learners. We train models with early stopping and patience of 15 test set evaluations which occur after every 2000 batches. We apply stochastic
Empirically Evaluating Joint Training (Section 3).
15

---

gradient descent as our optimizer. We apply standard preprocessing and augmentations consisting of input standardization, random crops, and random horizontal flips.
Diagnosing Learner Collusion (Section 6.1 These experiments investigate the performance of ensembles of multi-layer perceptrons on four regression datasets from the UCI machine learning repository [50]. These are Bioconcentration [51]. Weather [52], Facebook [53), and Abalone [54]. Standard preprocessing is applied each dataset including standardization of the features and tar- gets, one hot encoding of categorical features, and log transformation of skewed features. Due to the smaller size of these tabular datasets, these experiments are less computationally expensive and, therefore, we perform hyperparameter optimization using Bayesian optimization for each run. For each run, we apply 20 iterations of Bayesian optimization searching over the following parameters. Learning rate E {0.01, 0.005, 0.001 J. batch size E {16, 32, 64). weight decay E {0, 0.01, 0.001), number of base learners E (5, 10, 15, 20). hidden layer neurons E {10, 20, 30), and number of hid- den layers E (1, 3). Additionally, the ReLU activation function is used with mean squared error loss. At each run, 20% of the data randomly used for testing with the remaining data being further split into training and validation with an 80/20 ratio. Each ensemble is trained with early stopping with patience of epochs. The berf-perf hyperparameters are selected based on validation set performance and then retrained on the train/validatis data for all values of B E {0, 00.......... 0.9, 1) and evaluated on the test set. This is repeated for 10 runs. The mean = a standard deviation is provided in our results.
In Figure we report test set ERR and DIV as MSE and Diversity respectively. In Figure we we report the percentage decrease in test set DIV for each value of B after the debiasing process described in the main text. In Figure for each model being evaluated on the test set we randomly drop a percentage of the base learners p (0.2, 0.4, 0.6) and report the mean increase in test set ERR (ensemble mean squared error).
Learner Collusion Generalizes Poorly (Section 6.2). This section uses a similar experimental setting Section 6.3 on the CIFAR datasets, as described in detail in the next paragraph. The key difference is using Adam optimizer [55] without learning rate decay to evaluate performance smoothly over the course of training. We train for a fixed period of 100 training epochs. We note that this number of training epochs is only for our analysis as the best validation performance generally occurs significantly earlier. We also only use the standard training/testing split as third split not required. In Figure 5 we report the difference in ERR measured on the training set vs the test set over the course of training and refer to this metric as the generalization gap. This can be expressed as C y) _ (f, y) where the loss function is cross-entropy loss. Further results from this experiment are provided in Appendix G
Practical Implications (Section 6.3) In Figure 6 (right) we use the same experimental setup as
in Section In the two left-side plots we train and evaluate on CIFAR-10 and CIFAR-100 (56). We train ensembles of either ResNet-18 18 or ResNet-34 [35] as base learners with each ensemble consisting of five learners. Base learners are again aggregated using their probability outputs (i.e. probability averaging in Section 4.2) We use the standard training and testing splits with an ad- ditional 80/20 split on the training data to provide a validation set in order to ensure our test set performance is independent of implicit bias from the training process. We use a similar learning schedule as proposed in (35). Specifically, we train for 80 epochs using stochastic gradient descent with momentum of 0.9. The learning rate is initialized at 0.1 and exponentially decays at rate of 0.2 at epochs [30, 50, 70). We apply weight decay of 5e - 4. Validation performance is evaluated at each epoch with the best-performing model across all 80 epochs evaluated on the test set. A batch size of 128 was used. We repeat each experiment for a total of six runs and report the mean and two standard errors in our results. We apply standard preprocessing and augmentations consisting of input standardization, random crops, random horizontal flips, and random rotations. The key differ- ence in the left figure is that instead of only training deep ensembles using independent training and joint training, we now also consider a hybrid objective using the augmented objective for Section [5] We train this objective for 8 E (0, 0.2, 0.4, 0.6, 0.8, 1} and report accuracy evaluated the test set in the figure. In the right figure, we provide the validation set accuracy of both ImageNet models throughout training. We provide both the performance of the individual learners and the aggregate ensemble performance.
Additional experiments from this section are provided in Appendix G Unless otherwise stated, results reported in the appendix are aggregated over three runs.
16

---

Compute. All experiments are run on Azure NCv3-series VMs powered by NVIDIA Tesla V100 GPUs with 16GB of GPU VRAM.
C
Proofs
C.1
Optimal Debiasing
We hypothesize that the predictions each learner in the ensemble associated with bias term by which cancel in aggregate but significantly affect our measure of diversity. This is the most simple conceivable form of learner collusion and in this section we prove that the debiasing term used in Section C is optimal for this purpose.
affect the aggregated ensemble prediction require that Ej 1 bÿ) Σ5 fj and therefore we require that Σ, by = O. Then we wish to find the set of such bias terms that minimize the standard diversity, DIV, of the ensemble over the data. arrive at the optimization objective in Equation for
We begin by noting that for these biasing terms
Combining these two requirements ensemble of size M with N examples.
M N
min ha
Σ
M (x))),) by f(x)))) subject to
Σ
by
(3)
By the method of Lagrange multipliers we obtain M + 1 equations for the M values by and the Lagrange multiplier! X
N M
M
V
AA
b =0 0 A
J.
(4)
N
Σ bj =
(5)
Solving equation 4 for an arbitrary by, we obtain
2
x
-b
N.
M
T=1
N
A M
by
((x.))
2
1=1
And subbing into equation 5
A M
1
Σ (J)(x)) -(xx)) [(x3))= =0
Σ
2
N
X M
M
-f(x)) = (f(x:)
2
N
==0.
Finally, we can substitute À 0 back into equation 6 to obtain the solution
1 Σ (x,(x)) - f(x:)) j.
by = N
17

---

C.2 General Diversity Expression for Multidimensional Outputs
Theorem 4.5. [Diversity for General Loss] For any loss function L y2 + R$ with dim()) N* that is at least twice differentiable, the loss of the ensemble can be decomposed into
M DIV
2 w (f) ff) (F (). (fz f) i=1
with fj =f c -f),
where Hg the Hessian of with respect to its first argument and some E [0, for all j [M].
Proof. We follow the same approach taken in [44] but extend their proof to account for mul- tidimensional output space. For each j E [M]. we define the curve 7/ [0, 1] + y by (1) - Lff + (f, y) for all [0, 1]. note that y( = (1). Since the loss L is at least twice differentiable, we can perform first-order Taylor expansion around / = 0 with Lagrange remainder:
1 (1)) + (f) - f)+ 2(1 - F)) N1,(H y) (f, - f).
with F3 defined in the theorem statement. We note that this expression for (1( is exact for any twice differentiable loss thanks to the Lagrange's s remainder theorem (see e.g. p.878 of [57]. We
deduce the following formula for the average loss over the learners:
ERR
wy Cff((f,
M
(1)
Ø=1
0
C(F,
y]
Σ w. |=!
+
1
(f) f)T HG y) (f--
2 j=1
where the second term vanishes due to the identity Σ/e w f. Since DIV this proves the theorem.
ERR ERR,
C.3 Probability Averaged Diversity Term for Classification
Theorem 4.3. For a probability averaged ensemble classifier, trained using cross-entropy loss LeE with ks E K. denoting the ground truth class, diversity is given by
DIV Σ wy Lo.ly,y - Lcr(y, Σ wy v(k*) log p(k*)
P3 (k*)
18

---

Proof.
M
DIV
=
wy wyCoely.P)) - Lce(y,p)
M
wy (KL((y)))); + H(y) DKE(yllp) Hyy)
(where is entropy)
==1
ykk)
(k) log y(kk)
Σ
w
(k) log
Σ
P2(k)
p(k)
j=1
kEK
p(k)
Σ
W
(kk)
log
(k(k)
REK
M Σ ==1
p(k*)
Wj v(kk)
)
log
kkt
Interpretation. This expression has a sensible interpretation as the diversity of a probability- averaged ensemble. Examining this expression of diversity:
M Σ wy y(k*) log p(k") j=1
DIV
Pj (1**)*
where k* is the true class, and we note that this expression's only non-zero contribution comes from the predictions of the ground -rrth class. We now view Py(k* as coming from a distribution of probabilities and assume that p)(k") is distributed around p(k*) following Beta distribution with
its mean at p(k*), i.e. Py (k*) Beta(a, B), where [k])(k*)] p(k"). Consequently, we can examine the expected value of the term with respect to ensemble learners as follows:
Ep, (k*) log p(k²)
Ep, [log [log Ep (k= (lop
(k+)
Vlpz(k*)]
Ep (logg) log (k+) pPs (k*)] 2E, (K-) (p)(k))
=logp(k*) -og (ok*(())) + (k*)
2p(k*)²
2p(k*)2
a
(a +B)(((a + 3 + 1)2p(k*)²
111 - 14) (1+ k)2kk)((k*
((k))( - p(k*)) (1 o)²p(k*))
where the approximation is from taking second order Taylor expansion around Pjo (k*) (p (ape>>(p,((e (k*)] ]] to approximate Epy(t>>]logg (k*)]. The fourth line is due to reparameteriza- parameters with = 01 - n)). Using this reparameterization, we arrive at an expression that emits an intuitive interpretation that for fixed p. the smaller the value of Φ (conversely, the larger the variance), the larger the expected value of diversity.
tion of the Beta distribution terms of mean / and precision
Φ y and
19

---

Furthermore, we might ask if this expression of diversity in Theorem is connected to an inffomatiionthhoreel quantity as in the score averaging case While this expression does not take the form of any divergence term that the authors of this work are aware of, the following link can be made.
p(k)
Σ
W
ykk)
log
(2(k)
KEK
M
vkk) k(k)
log (kk)
Σ KEK
j
k(k)
Py (2(k)
1=1
M
((k))
pkk)
Wy (P))E
log
((k)
p2(k)
1=1
M
7(k)) log (kk)
where y(k) v(k)
Σ (py
p(k)
1=1
M
Σ wy (KLkp))))) as 1.
Therefore, as the ensemble predictive distribution approaches the label distribution, the diversity term approaches the weighted average of the KL-divergence between the ensemble predictive distri- bution and each of the base learner predictive distributions, analogous to the equivalent expression of the score averaged ensemble.
C.4 Pathological loss for 3 > 1
Theorem 5.1. [Pathological Behavior for 3 > 1/ We consider target set y Rdd or y C id being a compact and convex subset Rd. Let L : y22 R be a continuous and finite function on the interior int) We assume that the loss is not bounded from above on y² in such way that there exists y E int(Y) and a sequence (yt)seN Cy such that L(y,,y) + +00 for + sc. If 1, then the augmented loss L defined in Equation 2 is not bounded from below on y2.
We first consider the case where y = TRd. Without loss of generality, we consider an ensemble such that 11, W2 O. We fix f, = y for all < j S M. We then consider the following sequence of learners: (f1,))eN = (yt)IEN and (f2,)LeN
Proof. Case 1.
w W2 y))SE,. We note that for all E N, the ensemble prediction equals zero fç = (why: - W2202 - +0) = 0. The ensemble loss is therefore constant and finite ERR, + L(0, < xo as t - 0a since (0, int ()/2). On the other hand, we have that the average loss diverges ERR, 2 wi L(y,, y) + + as + co. We deduce that the augmented loss is not bounded from below when FRR. - -C for - Dc.
as ERR, + (1 (1-8).
Case 2. We now consider the case where y C IRd is a compact and convex subset of Rdd Without loss of generality, we consider an ensemble such that 0 < Wi 1. Since y is compact, it is closed and admits a topological boundary that we denote oy. Since the loss is finite on the interior in²2) our assumption implies that the loss admits a singular point (y with you ay. We fix f2 = y for all 1 < S M. For the first learner, we consider a sequence (f1,))eNt with fi =t - (1- Hence, once again, we have that the average loss diverges as ERR, 2 wi C(f, + too as t + co. remains show that the ensemble loss ERR is finite. We note that the ensemble
We trivially have that f,, + Yoo for + co, so that ((f,,r,)) + co.
prediction also defines a sequence (fr)eNt such that for all L E N
f, w1 [t- y + (1-t-) yool +y Σ
Wj
j=2
= [1 + wi (t- 1)] y |w (1 - + yo..
20

---

We deduce that fr E D as it is convex combination of and Yx.. Furthermore, we trivially have that f. + fxx = (1 w1 y + wi Yco as t co. We shall now demonstrate that f.x ()) and conclude. We note that, since E i3)) there exists an open ball B(y,r] of radius e R centered at y such that B[y,r] _ y.y Let us now consider the open ball B fo (1 - -w1) rr. We note that any y' E B(f (1 w1) r] is of the form
y = foo + (1 w1) vv
= (1-wi) w1) (y + v) w1 Yoo:
where v E B[0, r]. Clearly, (y +v) E Bly, r] S y. hence y' E E as it is a convex combination of two elements of Y: We deduce that B[f (1 -ww wi) y., which implies that fox int(()). We conclude that ERR, - [ff y) by continuity of the loss on in)) We also know that L(f y) finite as (fx.; y) E int. We deduce that the augmented loss is not bounded from below when 1 as ERR, + (1 B) ERR, + for t +
D Gradient Adjusted Targets
Throughout this work we have discussed the two extremes of training neural network ensembles. In- dependent training, where each ensemble member is trained independently and joint training where the predictions of the ensemble as a whole is optimized. We have also shown that interpolating be- tween these two extremes via the parameter can result an effective hybridization. In this section, we provide an alternative interpretation of these intermediate values of [3 as an optimization objec- tive. In particular, we show that the hybrid objective can in fact be interpreted as an independently trained ensemble in which the targets y are adjusted by the gradient of the ensemble.
Specifically, in the of mean squared error can adjust the target y by step in the opposite direction to the gradient. Then each individual learner S. will have its target adjusted to compensate for the errors of the full ensemble. The size of this step is determined by the parameter a. This results in the gradient adjusted target (GAT) objective in Equation (7). Note that throughout this section we consider scalar regression but, of course, these results would hold for multiple outputs by applying the same reasoning to each output.
1 M
GAT (f:.
M Σ
(f+ 0)3
j=l
where
1 a g
and
g = of
8 1
1
( - y)²
Conveniently, it can be shown that applying individual training with this GAT objective is equivalent to using the hybrid objective described in equation 2 where the step size a has a similar effect to the interpolation parameter B.
Theorem D.1 (Residual adjusted mean squared error). The ensemble gradient adjusted mean squared error objective
1 M (Ji
M a
Σ
(v
Σ U1
=)))
(8)
M 1=1
M
j=1
is proportional to ERR (1 (a+1)) DIV.
21

---

Proof.
(f: (v
[Jz y)))
M
M
i=1
M
(fi - + a:
M ==1
Σ ((f f) +
M
Σ [Ui - f)2 + (1 (1+a) y)² + 2. (1 + a) - DJ - v)
[ DIV +1++a) ERR +
0
]
EERR
(1-- (1-1++a)
x ERR
DIV
(1+0)
When we take no gradient step at a O. this is equivalent to 3 = or independent training. As we take larger gradient step approach joint training noting that - oo find 3 + 1. Should we choose to take a step of exactly the gradient itself (i.e. a 1), it is equivalent to hybrid training with 3 = 0.75.
Alternatively, we might wish to only consider the M other members of the ensemble in our gradient step g. this case we might consider f M-1 Ejtt M-1 fy rather than f. Again, we find that this can be expressed in terms of ERR and DIV. We begin with prerequisite in Lemma D.2 and then proceed Theorem [33 to show that there also an exact equivalence in the M case augmented objective L described in Section 5
Lemma D.2.
M E - 1i)(w - 11) DIV.
(9)
Proof.
M
E - -f- fD)
M
= - -E 13)
-- 1. p+ Es
- f + Σ (f, M
M
M.DIV.
22

---

Theorem D.3 (M - residual adjusted mean squared error), The M ensemble gradient adjusted mean squared evvor objective
M
M M
1 M
(f.
(U
a
(f) E
B))),
(10)
is equivalent ERR 7 DIV up to a scalar constant where y (a+1)²
M((a+2) M-2aa))))))+
(M-1))
Proof.
M
1
(fs
(y
a
-
M
M
A#i
1 M
Σ
(Js
1
M Σ
$q 0x 01)²
j#i
1 M
a
M
[+ M - J3 - (1 a) -1)
M
M
M
1 M
M
M
M
(f ft) (1+ (1+a) .) (using: 1+a
M
M
M
M
1 M
a M
(f fi) - a1+ (y- f())
M -
1=1
a.-M
DIV + (1+a)* ERR 2.
(1 +
(f - folyy fi)
M 1-1
M
1=1
a a.MM M
DIV + (1 + a)² ERR 2. M
(1 + a.) M DIV
(using Lemma D.2
M -1
ERR aM((a + 2)M 2(a + 1)) DIV. (M 1
(1 + a))²
While the scalar y might initially appear unwieldy, at closer inspection it performs a similar function to the full ensemble gradient equivalent in Proposition [.I The key difference being that this term also accounts for the ensemble size M and ensures that smaller ensembles have a higher weighting on diversity. This relationship is visually depicted in Figure g where again observe sensible connection to hybrid training at various values of B.
10
0.8
0.6
04
02
0.5
0.0
20
80
100
Ensemble size M
Figure 8: M gradient adjusted targets. 7 (equivalent to 3 in the augmented objective ( as a function of ensemble size M and gradient step size a. Again we observe sensible relationship between a and
23

---

Investigating the relationship of 3 and diversity via GAT. In the regression experiments in Fig- ure [ one might intuitively have expected diversity to increase linearly with 9. Instead, we observe sharp increase in diversity as 8 1. The GAT perspective described in this section may provide some insight into why this relationship takes this form. Recall that the / term from the augmented objective in Equation (2 is related to the term the GAT objective in Theorem D.1 according to
= (1 (I+a)=). ). We include a visualization this relationship in Figure This illustrates that the gradient step a grows exponentially as 3 - resulting in each individual learner having their tar- gets biased excessively (given that the targets are standardized in these experiments) account for the ensemble errors. This exponential increase in the bias in the targets would appear to be related
the simultaneous increase in diversity at the same point.
30
25
0 15
10
5-
0-
1.0
Figure 9: Understanding the non-lincar increase in diversity with 3. Increasing 3 in our aug- mented objective is exactly equivalent individual training with the targets adjusted to account for the errors of the ensemble by step size of a. The relationship between these two terms is given by
(1+a)=) which we visualize this plot. Thus the gradient step a grows exponentially as B + 1 resulting in each individual learner having excessive biases dominate their targets which coincides with exponential increase in diversity.
E Gradient Analysis of Classification Ensemble Averaging Methods
On the claims of 311. We begin by taking closer look at the claims of 31 where, using their notation, they presented the following equation for an equally weighted ensemble using either output averaging or probability averaging.
Oe
de 0p aE
0x;
0p Ox;
0u N
where [((x,,"' .NN) DE x; represents the averaging of a set of N model outputs and some loss function.
The authors claimed that, because this expression does not depend propagated into all ensemble members are identical". On this basis, they argued that this "lack of gradient diversity" combined with numerical stability issues explains the poor performance of joint training. We wish to make two points on this claim. (1) Contrary to what is claimed, this constant gradient effect is only true in the case of score averaging and is not true for probability averaging which also performs poorly under joint training. (2) Even in the case of score averag- ing, although all ensemble members receive the same gradient signal, that signal is a function of the ensemble performance under joint training and the individual performance under independent training. Therefore the joint training optimal solution would still maximize ensemble performance if we could obtain the optimal solution.
the "gradients back-
We begin with the first point (1), If each x represents the scores of each ensemble member and we presume that the softmax activation Ø is baked into f, then this is the score-averaged classification setting, and the author's expression holds. However, if we wish consider the probability averaged setting, then the averaging function should in fact be p(x1,".. .NN) >. d(x;) and the derivative takes the following form.
24

---

0e
de
0p
@σ(x)) d0(x;)
0x1 du (x(()) 0xi
dy N
0x1
is clear that the final term in this expression is indeed a function of x; and therefore each learner can receive unique gradients in the case of probability averaging.
With respect the second point (2), we can simply note that the loss term e is function of just x; in independent training and all X1, --- XN
XN in joint training. Concretely, we note that the gradient with
respect learner i should be expressed as
@((x)) for independent training and as Def((x xx)) for
8x;
8x;
joint training. This should clarify that although the learners could potentially have constant gradients in the case of joint training, they are still optimizing the correct objective and, therefore, the constant nature of these gradients would not in itself explain why the joint training objective results in poorer ensemble performance. We argue that a better explanation lies in the observation that the additional diversity term in the joint objective surprisingly results in poorer solutions is the observation that these multivariate entangled objectives result in an optimization problem that is significantly more challenging solve in practice. In other words, individual training performs better it results in better solution to a slightly sub-optimal objective.
A gradient analysis of cross-entropy loss. Here extend the previous gradient analysis by com- paring the exact gradients obtained for independent training, probability-averann acore-averaged joint training in the case of cross-entropy loss. We use the following notation. fg. the pre-activation output, or score, for learner j of class k. ø is the softmax function with o(f,) referencing the the &' ''th element of the softmax function applied to the vector fy. L is the loss function.
joint training and
We summarize the objectives for the three cases in Table The columns correspond to the training method (independent vs joint) paired with the output averaging method (probability vs score). The first row corresponds to a general loss function, while the others are the specific cases of cross- entropy loss. The third row simply re-expressing the second row by considering the true class k* such that yı-
Table Ensemble loss functions. A summary of primary loss functions used throughout this work. The first row corresponds to the expressions for general loss function, while the following rows contain equivalent expressions for cross-entropy loss. Note that the final row is just expressing the previous with the correct class k* replacing the sum.
Independent training
Joint training
Output aggregation
Both
Probability averaging
Score averaging
General loss
Cird
M1 Eg ((ol))))))
C = C E, ((f)).))
L- C(d E, f)). E UN log (0( Eg Fyle)
Cross-entropy loss
17 Σ. Σ3 31k log (D(())))
Ex Un log E, ((S)))
Cruss-entropy loss (l: k*))
# E log (o(())e))
log
+ E, ((jik-))
log (o E, Bile.))
Now we evaluate the gradient of each training method and output averaging combination with re- spect to arbitrary learner for arbitrary class k. We begin with the case of independent training.
A o(f)).
dLind
0 M 0f.kk
Elog log (d(f))x-) 1
FI.
0f.k
M
Φ(f))e-
1 1=+(f())))
if K*
o(f))
if of*
M
Next we examine the equivalent gradients of the joint training objective with probability averaged predictions.
25

---

9 Th. (ffi)
OC
a
log
Σ o(fs)a
0fin
Ofin
M
Miee (1 (1-0(Sibe))
if k k*
2, olfsie
ofdee(ffee 2, "l,s)
if + k*
Finally, we calculate the same gradients for the case of score averaging.
① 0l M Fi)
2h
0( 1 M
log
0fi,
0fi,
Ø M E, fz)a.
1 M (1 E, fa)a
if k-,
M 0l t E; Si)s
if +
From these gradients, we draw the following conclusions for given ensemble member fi-
Independent training- Each member receives a variable gradient that is a function of their own predictions but not the predictions of the rest of the ensemble. The gradient for learner f, takes the form g(ft).
Joint training with probability averaging Each member receives a variable gradient that is function of their own predictions and the predictions of the other ensemble members. The gradient for learner fü takes the form g(i, f1, .M))
Joint training with score averaging Each member receives constant gradient that is function of their own predictions and the predictions of the other ensemble members. The gradient for learner fi takes the form g(f).
F Learner Collusion in Regression
In this section, we provide an illustrative example of the learner collusion effect in the case of regres- sion with mean squared error. We begin by recalling the relevant decomposition from Equation ! for an equally weighted ensemble.
ERR ERR DIV,
1 2) M E v)2
1 M
EV,
(U;-Jy
(11)
Without loss of generality, consider each base learner predict some ensemble prediction and learner-dependent sealar adjustment E R such that f; f cj. We also write M Σ, 9 = straightforward to check that Equation [1] can then be expressed as
f R
ERR = Σ y
1
c2)
-cg-af
M
M
3
(f - y)²
M Σ
2 3
Eg
4)
M E(e
(c -22
M
ERR
DIV
Setting aside f. we focus on the optimization of the cj's. We note that in the setting of independent training where we only minimize ERR, the unique minimum is achieved when C; 0 A j. However, when we add the additional DIV term, this changes. When optimizing the joint objective, ERR, any solution satisfying y- will minimize this loss. This is exactly learner collusion. We
26

---

note that as we optimize on training data, loss typically approaches zero, and ē - O. Furthermore, if we were to require that the two terms ERR and DIV should be relatively, approximately equal, we note that this would only occur as the ej terms become large. Specifically, HRR ~ DIV as |cyl + co. While this analysis illustrates how artificial diversity may emerge in the regression setting, we note that this is still not guaranteed to result in poor test performance in all cases. One such example is the Facebook dataset in Figure 2 where the test mean squared error is largely unaffected by this excessive diversity. In this case, it appears that almost the entire predictive is contained within single feature resulting in no benefit from true diversity and, therefore, no substantial impact due to learner collusion
Finally, we highlight that during independent training, the learners are not aware of each other's predictions and, therefore, the premise that there would be single shared ensemble prediction can not hold. this case, all diversity will be due to external factors such as random initialization and improved performance over single model can be explained by standard ensemble theory [e.g. 2 In the case of joint training, the premise that the ensemble produces a single prediction disguised as multiple predictions (i.e. fj- = f + Cj where गे Σn Cj 0) is plausible and even likely. However, if this is the case, then clearly there is no genuine diversity among the learners implying that overfitting and poor generalization may become more likely.
G Additional Results
G.1 Generalization Gap Loss Plots
This section provides an extended analysis of the experiments investigating the generalization con- sequences of learner collusion discussed in Section 6.2] Figure I0 provides the same generalization gap plots from Figure but decomposed into the training and testing loss. For analysis purposes, we trained these models for 100 epochs. but this plot highlights that convergence occurs much carlier. These results reveal that the generalization gap is caused by both a slower training set convergence in addition to greater test set loss. This slower convergence may in part be due to the two terms of the joint objective (3- = 1) competing against each other in the direction of their gradient updates. More concerning is the larger test set loss. This indicates that the solution achieved training loss even- tually becomes low, generalizes much more poorly than the solution achieved by the independent training objective (3 0).
CIFAR-10 (8 =0)
CIFAR-1O (8 =
CIFAR-100 (8 0)
CIFAR-100 (0
2.00
TTrin losas
1.00-
'Test losss
0.70
6:300
0.50
0.25
0.00
0.000
02
Epoch
Figure 10: Train & test loss. The jointly trained model (8 1) converges more slowly on the training set and reaches a solution that generalizes more poorly than the independent objective (B = 0).
We also provide decomposition of the loss into individual error (ERR) and diversity (DIV) on both the training set (Figure [T] and the test set (Figure 12). From the training plots, appears that the joint objective (33 1) maximizes diversity early in training before beginning to minimize individual error later. This aligns with the learner collusion effect we describe in Section 6.. as is trivial to inflate diversity while minimizing the individual error requires solving the true objective of the task. The test set decomposition in Figure 12 reinforces that the solutions obtained with joint training generalize poorly beyond the training data.
27

---

CIFAR-10
CIFAR-10 (0
CIPAR-100 :0
CIFAR-100
DIV
ERR
Value
1.0
1.0
65
0.0
0.0
Epoch
Figure 11: Train loss decomposition. individual error (ERR) highlights that diversity is trivially inflated early in the training process when joint training is applied 1).
The training loss decomposed into diversity (DIV) and
CIFAR-10 (8= =0
CIRAR-10 (8 E
CIFAR-100 (8
CIFAR-100 (3
DIV
Valuo
Epoch
Figure 12: Test loss decomposition. The test loss decomposed into diversity (DIV) and individual error (ERR) reinforces that optimizing for learner collusion does not generalize effectively onto a test set.
G.2 Larger Ensembles
In Figure 13 we reproduce the experiments with ResNet-18 base learners from Figure 6 (left) but with a larger ensemble such that M 10. We observe that, while there small improvement in test accuracy, the shape of the curves with increasing B are consistent with those of the smaller ensemble.
0.75
0.79
Ascurary
031
0.7
RosNet:13 18
0.70
Level diversity 9
Figure 13: Larger ensembles. Identical experiments with larger ensembles (M -10) result similar trend in the accuracy curves.
G.3 Score Averaging
Figure 14 includes the results of repeating the experiments in the top row of Figure using averaging rather than probability averaging as described in Section Similarly to previous works, find that joint training performs poorly in this setting too. We also note that test set performance is similar to that probability averaging on both datasets.
using score-
G.4 CNN Ensembles
We repeat the experiments described in Section 6.3 with ensembles consisting of base learners with a vanilla convolutional neural network architecture. The architecture of each of these base learners provided in Table Dropout with probability 0.1 applied throughout the network during
training. Furthermore, apply Adam optimizer [55 with learning rate 0.001 and early stopping
28

---

CIFAT-
CIFAR-100
0.70
051
0.74
0333
0333
0.72
Bo
032
01.
Level diversity
Figure 14: Score averaging. This objective also performs significantly worse when applying joint training (3 1).
with patience of epochs. Otherwise, all experimental parameters are consistent with those previous experiments. We apply this ensemble to CIFAR-10 10 with ensembles of size M = 5 and M = 10 with results provided in Figure [5] The results of this experiment are consistent with those of previous architectures and we again note the poor test set performance of the jointly trained model (B = 1). Interestingly, we find in both cases that the lowest test loss is achieved at interpolating values of as achieved by the augmented objective L as described in Section 5
Table 3: Vanilla Convolutional Neural Network Architecture.
Layer Type
Hyperparameters
Activation Function
Conv2d
Input Chansels:3 Output Channels:10: Size:5 Strid:2 Padding:1
ReL.U
Conv2d
Input Chanaels: 10: Outpuit Chanacis:20, Kernel Size:5 Stidd:22 Padding:1
ReL.U
Flatten
Start Dimension:1
Linear
Input Dimension: Ourput Dimension:
ReLU
Linear
Input Dimension: 50 Output Dimension: 10
18
0.2
0.20
CXN
0.68
Negatree log Uidhved
1.00
CKN
1.00
0.53
0.30
0.50
disnally
Figure 15: CNN ensembles. Standard architectures achieve similarly poor results for joint training (3 1) but obtain their lowest test loss at intermediate values (3 E (0, 1)) using L
G.5 SVHN with VGG architecture
In this experiment we repeat the experiments in Section 6.3 on the SVHN dataset [58] using a VGG style architecture [14). We report the exact architecture Table 4 Again, ensembles of size We use the same optimization protocol as applied to the ResNet-based ensembles as reported in Appendix B Consistent with all previous experiments observe that joint training again results in degenerate performance this setting.
G.6 ERR as metric
For completeness we report the ERR test set performance corresponding to the experiments in Fig- ure [6 (left). We note that this is exactly the ensemble loss which optimized at / This is useful to report it is the exact quantity being optimized during training. Consistent with all pre- vious results, performance is worst as - (i.e. as we approach optimizing for the metric being reported).
29

---

Table 4: VGG Architecture
Layer Type
Hyperprameters
Activation Funcrion
Dropout rare
Conv2d
Inpul Channels: ; Outpui Channels:64 Kernel Size:3; Strid: ;; Padding:
ReL.U
Con-2d
Input Channels664 64, Output Chamnels:64 Kernel Size:3; Stride:1
Padding:
BN & ReLU
MaxPool2d
Kernel Size:2: Padding:
Conv2d
Inpat Channels.64 64; Output Channels:128; Kernel Size:
Stride: Padding:
BN ReLU
Conv2d
Input Chamels:122 Output Chanadls:128 Kernel Size:3 Stride:
Padding:1
BN & Rel.1l
MaxPool2d
Kernel Size:2; Padding:
Conv2d
Input Channels:122 Output Chamadss:266 Kernel Size:3
Itride:
Padding::
BN & Rel.Uf
0.4
Cou-2d
Input Channels:266 ; Output Chamnels:266 ; Kernel Size:3 Stride: Padding:
BN & ReLU
0.4
Conv2d
Input Chamels:2566 ; Output Chumnels:256; Kermel Size:3 Stride:
Padding:1
ReLU
MaxPool2d
Kernel Size:2; Pudding:
Conv2d
Input Channels:256. ; Output Channels:512 12; Kernel Size:3
S
Padding:
BN & Rel.ll
0.4
Conv2d
Input Channels:512 ; Output Channels:552 12; Kernel Size:3 Stride: Padling::
BN ReLU
Cony2d
Input Chamels:512 Output Channdls:512 Kernel Size:3 Suide:
Padding::
BN & Rel.1l
MaxPool2d
Kernel Size:2: Pudding:
Conv2d
Input 111 ; Output Channolo:512 12; Kernel Size:3 Itride: Padding::
BN & ReLU
0.4
Conv2d
Input Channels:112 Outpot Channelk:512; Kernel Size:3 Stride: Padding::
BN & Rel.Ul
0.4
Conv2d
Input Channel:512 Output Channels:5 12; Kernel Size:3 Stride: Padding:1
ReLU
MaxPoolZd
Kernel Size:2: Pudding:
NA
Flatten
Start Dimension:
0.5
Lineur
Input Dimension: 512 Output Dimension: 512
BN & ReLU
0.5
Lincar
Input Dimension: 512 Oulput Dimension:
SVHS (anouay)
SVHN (Boe)
Negathen lop Hivillloo
0.9590
Aretinay
089025
0.0
022
11.4
0.0
000
Level diversity
Figure 16: VGG ensembles on SVHN data. The same phenomenon emerges when we consider an alternative VGG style architecture on the SVHN dataset.
G.7 Learner dropout as a resolution
In this section we provide a negative result in which we attempt to resolve learner collusion by randomly removing subset of base learners from the ensemble at each batch during training. The hypothesis that if we drop a sufficiently large portion of base learners but still perform joint training on the remainder, this should remove the ability to deterministically collude and, therefore, cause inflated diversity be actively harmful even on the training data. One might hope that this would cause the ensemble to avoid this degeneracy. In what follows we provide the details of this experiment.
We repeat the setup on CIFAR-10 with ResNet- 18 architecture as described earlier in the paper. We train each model using the joint objective. but at each batch we drop proportion p [0, 0.2, 0.4, 0.6] of randomly selected learners from the ensemble. We then investigate if this (a) reduces collusion and (b) improves ensemble performance. The results of this experiment are included in Figure [88 We find that dropping learners does indeed significantly reduce the diversity score indicating a re- duction in learner collusion (although a reasonably large proportion of learners required to be dropped). Unfortunately, the improvement in performance by reducing collusion is negated by a de- crease individual performance due to the base learners becoming weaker on average. The reason why is apparent once we notice that by dropping base learner from some proportion of batches, this is exactly equivalent to bootstrapping. While bootstrapping has been effective for ensembles of weak learners (c.g. random forest) it has already been shown to be harmful for deep ensem-
30

---

CIPAR-10
Negatove lop lidhood
11:50
Ros 18
RisPet-S1
01300)
0.17
Level diverstty 3
Figure 17: Loss plots. Corresponding ensemble loss (ERR) for the results presented in Figure (left).
Unfortunately, this resolution appears to only reduce the effect of learner collusion by simultaneously harming the performance of the ensemble due to bootstrapping.
bles 12..
Proposing methods overcome learner collusion work that we and, we hope, others in the research community will pursue. We believe that the comprehensive diagnosis of issue we have presented this work will provide a foundation for future methodological progress.
valuable but non-trivial direction for future
0.93
Accuracy
0.4
ERR
Accuracy
DIV
0.92
Loss
0.2
0.91
0.2
0.6
0.4
Learner dropout rate
Learner dropout rate
Figure 18: Investigating learner dropout. We evaluate randomly dropping subset of learners dur- ing training to remedy the learner collusion effect. We repeat the setup on CIFAR-10 with ResNet- 18 architecture from Figure 6 with 3 1 but each training batch, we drop proportion of learners in [0.0, 0.2, 0.4, 0.6). We find that, while this seemingly does reduce learner collusion, the benefit is negated by a decrease in individual performance due base learners becoming weaker average. Results are reported over 5 runs.
31


Reviews:
[{"review_id": "lq9boxFbYD", "reviewer": "Reviewer_XMTw", "summary": "This paper mainly studies the reason behind the failure of jointly training deep ensembles. It discovers that joint optimization results in a phenomenon in which base learners collude to artificially inflate their apparent diversity. Both theoretical and empirical evidence are provided further to verify the hypothesis.", "questions": "Please see my above weaknesses.", "limitations": "N/a", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "(1) This paper attempts to study a seemingly under-explored question, that is, why joint training of ensembles fails to generalize better than individual training and ensemble. This research topic is very important and helps us to understand the foundations of the deep ensemble.  \n\n(2) Both theoretical proof and empirical proof are provided.", "weaknesses": "(1) One big issue of this paper is that all references do not contain titles, which to be honest, I have never seen in top-tier conferences like NeurIPS. \n\n(2) I do not understand why joint training is easier to train but results in poor generalization. What does \"easier to train\" refer to? \n\n(3) What joint training approaches are used in the paper? \n\n(4) The empirical evaluation isn't very convincing to me. For instance, how many ensemble members are we using for \"Learner codependence\"? It is not surprising that the performance degrades to me if we drop ensemble members during testing, especially if our overall ensemble members is not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their constructive feedback. We have addressed the points raised below which have helped us to further improve our submissions clarity and contribution. \n  \n  $~$\n  \n* _What joint training approaches are used_ - Joint training refers to the case when the aggregated predictions of the entire ensemble are optimized directly. Formally (using the notation from our paper): given an ensemble of learners $f_1, \\ldots, f_M$, we might consider optimizing the _joint objective_ $\\mathcal{L}(\\frac{1}{M} \\sum_{i=1}^{M} f_i , y)$ ($= \\text{ERR}$) where $\\mathcal{L}$ is a given loss function such as mean squared error or negative log-likelihood. This is in contrast to the commonly observed _independent training_ where each ensemble member is optimized directly resulting in an optimization objective given by $\\frac{1}{M} \\sum_{i=1}^{M} \\mathcal{L}(f_i , y)$ ($= \\bar{\\text{ERR}}$). \n  \n  $~$\n  \n  **Action taken**: This definition is an important foundation for the contributions of our paper. We have therefore added this formal definition earlier to the introduction section to ensure it is established up front and without ambiguity.   \n  \n  $~$\n  \n* _What does \"easier to train\" refer to?_ - We did not make any claims regarding any method being “easier to train”. In fact, this quote does not appear in our paper. A point that we did make (e.g. L30-38) is that – despite the joint objective being the “true objective of interest” – throughout the literature, we typically observe the ensemble being trained independently in practice (occasionally with some regularization on the ensemble). Again, let us make this point more formally using the same notation as the previous point. The key question our paper addresses is: given that when we evaluate the performance of an ensemble (e.g. at test time or in production) using the loss term $\\text{ERR}$, why do we not optimize for that objective during training rather than the commonly observed proxy of $\\bar{\\text{ERR}}$? Of course, the conclusion is that this is for good reason: the joint training of deep ensembles (i.e. optimizing $\\text{ERR}$) fails due to learner collusion.\n  \n  $~$\n  \n  **Action taken**: We have integrated this point more formally into the introduction as part of the additional formalism from our previous action point. Thank you for encouraging us to further improve our clarity. \n  \n  $~$\n  \n* _Empirical evaluation_ - We agree that extending the empirical evaluation of the practical limitations of joint training can further strengthen this work. We have therefore **extensively broadened our evaluation on ImageNet** (from Table 1 and Figure 6 RHS) with eight additional architectures and various ensemble sizes. All models are trained from scratch and **we include these new results in the attached PDF document** (Table 5) where we consistently find that joint training performs significantly worse than independent training, thus reinforcing the claims of our work. \n  \n  $~$\n  \n  **Action taken**: We appreciate this suggestion and have updated our manuscript by including these additional results in Sec. 3. \n\n  \n  $~$\n  \n* _It is not surprising that the performance degrades to me if we drop ensemble members during testing (regarding Figure 4)_ - Indeed, it is correct to assume that dropping ensemble members at test time will likely result in a degradation in performance. Our intention in this experiment was to analyze the extent of that degradation for different training methods. If joint training results in learner collusion as we hypothesize, we would expect that the individual learners would become more codependent in that setting. To illustrate this, consider the trivial case of two identical learners that reasonably accurately predict the label $f_1 = f_2 \\approx y$. Should they collude by biasing their regression predictions in opposing directions by some constant $k \\in \\mathbb{R}$ (i.e. $f_1^\\text{bias} = f_1 + k$ and $f_2^\\text{bias} = f_2 - k$), this would artificially inflate diversity ($\\text{DIV} = \\frac{1}{2} \\sum_{j=1}^2(\\bar{f} - f_j^\\text{bias})^2 = k^2$) but result in a codependence such that dropping either learner at test time would be catastrophic to the ensemble performance as the remaining learner would be highly biased by exactly the term $k$. If these learners were independently trained and, therefore, not colluding, this bias can not be learned and the resulting drop in performance should be significantly less. In Figure 4 this is exactly what we observe. For all values of $\\beta$ we consider the relative increase in test loss upon dropping a subset of the learners at test time. As $\\beta \\to 1$ (i.e. as we approach joint training) we observe a very large increase in that test error while elsewhere the relative increase in error is far more modest. \n  \n  $~$\n  \n  **Action taken**: We have added a short interpretation of the results of this experiment directly in the caption of Figure 4. Again, thank you for encouraging us to further clarify exposition. \n  \n  $~$\n  \n* _Formatting issue_ -  We wish to sincerely apologize to all four reviewers for a (silent) latex compiling error in the final version of our manuscript that resulted in the titles of the papers not appearing in the PDF. This was an unfortunate inconvenience for the reviewers. We do wish to highlight that several reviewers made positive comments about the writing and organization of this work – thus, the issues regarding presentation are limited to simple **formatting errors which we have now rectified**. Should the paper be accepted, we would ensure careful attention is paid during further polishing of the manuscript for a camera-ready version."}, {"review_id": "1nouT5009z", "reviewer": "Reviewer_m1Sy", "summary": "This paper explores the joint training of deep ensembles, wherein the ensemble error is directly optimized during training. The authors find that joint training leads to poor performance, which they posit it due to phenomenon they call \"learner collusion\", where base learners artificially inflate their diversity, at the expense of test performance. The authors provide a new decomposition of a broad class of loss functions in terms of the average error rate, the ensemble error rate, and a generalized notion of diversity. The authors then perform an experimental evaluation into the prevalence of learner collusion during joint training, using an augmented objective function to probe the effect of explicitly encouraging diversity during training.  ", "questions": "I find the distinct change in behavior of the various metrics at $\\beta=1$ very interesting; it seems the behavior is relatively benign for $\\beta < 1$, and yet there appears to be a distinct transition at $\\beta = 1$. Do the authors have any hypotheses as to why this occurs?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Overall, the paper is very well-written and easy to follow, and in my view the topic is very relevant to the community. \n\nTo me, the most compelling experimental evidence for the learner collusion phenomenon are those presented under \"learner codependence\"/in Figure 4. From these results it is very clear that something distinct is happening in the joint training regime, and that the individual models are exploitating the joint loss. The definition of the augmented objective $\\mathcal{L}^\\beta$ seems like a very useful tool for probing these and other related phenomena. ", "weaknesses": "Unless I am misunderstanding, the conclusions drawn from Figure 2 do not seem to strongly support the claim that joint training significantly harms ensemble performance. Indeed, for the Facebook task, it seems the method actually helps, while for others it does not. This is in contrast to the CIFAR results in Table 1/Figure 6, where the degredation in performance is significant. Could the authors provide some clarification as the intended conclusion from these experiments?\n\nThere are a number of other papers in the literature that study some form of the gap (in the authors' notation) $\\bar{\\text{ERR}} - \\text{ERR}$, and express and/or bound it in terms of various forms of \"diversity\", depending on the given loss function. For example Ortega et al., 2022, Abe et al., 2022 and Masegosa et al., 2020 give explicit expressions for the average - ensemble error gap in terms of diversity metrics that are easily interpretable. It's unclear to me that the form in Theorem 4.5 adds much to this literature, and indeed the stated expression for diversity doesn't seem to be used in the remainder of the paper.\n\nOverall, while I think the paper is well-written and the topic is relevant, I think there is 1) insufficient use/novelty in the theoretical results and 2) somewhat incomplete conclusions to be drawn from the empirical results to recommend accepting at the the current stage. In particular, while it certainly appears like some type of \"learner collusion\" is happening, it seems very important to understand why this only appears at $\\beta=1$, and exactly what impact this could have on the ensemble error.   \n\nAs a minor point, there seems to be an issue with the references -- they don't seem to contain any titles of the referred papers. \n\n**References:**\n\nTaiga Abe, E Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. The best deep\nensembles sacrifice predictive diversity, 2022.\n\nLuis A. Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural\nnetwork ensembles, 2022.\n\nAndres Masegosa, Stephan Lorenzen, Christian Igel, and Yevgeny Seldin. Second order PAC-\nBayesian bounds for the weighted majority vote, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their constructive feedback. We have addressed the points raised below which have helped us to further improve our submissions clarity and contribution. \n  \n$~$\n  \n* _Facebook dataset & Formatting_ - Please see our response in the general comment section.\n  \n$~$\n  \n* _On Thm. 4.5_ - We apologize for not sufficiently emphasizing the significance of this result. This theorem is an essential ingredient as it demonstrates that, for any at least twice differentiable loss function, the ensemble loss can be decomposed into the aggregate individual loss and a term that captures ensemble diversity. This generalizes specific examples of this decomposition (e.g. Krogh and Vedelsby, 1994) and unifies any analysis of joint training to practically any loss function. Indeed, we do not manually calculate diversity using the Hessian as it is more efficiently calculated as $\\bar{ERR} - ERR$, but having this theorem ensures that the term we obtain from this calculation can always be interpreted as ensemble diversity. Furthermore, this theorem also guarantees that for positive semi-definite losses (e.g. MSE, Cross-Entropy), the diversity is lower bounded by zero – an important property for a sensible definition of diversity.  We emphasize that without this theorem we could only make conclusions about specific loss functions whilst with it we can discuss diversity and joint training in the general setting – resulting in us being the first paper to do so. \n  \n  $~$\n  \n  **Action taken**: We have added further context on the significance of this result in the text.\n  \n  $~$\n  \n  \n* _Distinct transition at $\\beta = 1$_ - This is indeed a notable phenomenon that we only addressed in the appendix. Let us provide a more succinct discussion on this point. \n  \n  $~$\n  \n  Figures 2-4 do highlight a sudden jump in learner collusion as $\\beta \\to 1$ in the case of regression rather than a gradual increase.  Why does this occur? A useful explanation lies in the gradient-adjusted target (GAT) analysis in App D. To briefly summarize, this section considers applying individual training but with an adjusted target which is a function of the gradient of the ensemble error. Specifically, we adjust the targets to account for the errors of the ensemble. Formally, we optimize $\\frac{1}{M}\\sum_{j=1}^M (f_j - \\bar{y})^2$ where $\\bar{y}$ is the adjusted target given by $\\bar{y} = y - \\alpha \\cdot g$. Here $g$ is the ensemble loss gradient and $\\alpha$ is a step size parameter determining how big a step the target of an individual learner should take in order to account for the errors of the ensemble. In Thm. D.1 we show that this GAT objective is exactly equivalent to our augmented objective in Sec. 5 with $\\beta = (1 - \\frac{1}{(1 + \\alpha)^2})$. Given this relationship, we might notice that the gradient step $\\alpha$ grows exponentially as $\\beta \\to 1$ resulting in each individual learner having their targets biased excessively to account for the ensemble errors (**please see the new figure in the attached PDF visualizing this relationship**). Given this perspective, it is unsurprising that learner collusion does not grow linearly with $\\beta$. \n  \n  $~$\n  \n  **Action taken**: We have added a succinct summary of this point in the main text. We have also added this new plot and refined the discussion in App D.\n  \n$~$\n  \n* _Related works_ - Thank you for raising this point, we hope that extending our discussion to these works will further highlight our contribution. Firstly, we wish to reiterate that our work investigates the direct optimization of the loss of the ensemble which we refer to as joint training. Given our findings (i.e. learner collusion), it is not surprising that there have been several works that attempt to improve ensemble performance by other means (i.e. proposing alternative methods of encouraging diversity). We categorized this literature as “ensemble-aware individual training” in our background section. We believe that our investigation of the underlying issue with directly optimizing the ensemble will provide an important basis for future methods of this type. \n  \n  $~$\n  \n  Masegosa et al. (2020) is one such example in which the authors propose to optimize a PAC-Bayesian generalization bound in the case of cross-entropy. Whilst conceptually appealing, optimizing the bounds of model performance is generally not a standard approach in machine learning as it is unclear how minimizing a worst-case bound or maximizing a best-case bound directly affects the actual generalization performance under the metric of interest of a model. However, given the limitations we have identified in directly optimizing that metric of interest in our work, this approach might offer an effective alternative providing a fruitful direction for future work. Then Ortega et al. (2022) extended this approach to additional loss functions, analytical analysis and a more practical empirical evaluation. We hope that our contribution here will provide a useful foundation for alternative ensemble training approaches such as these.  \n  \n  $~$\n  \n  The recent workshop paper of Abe et al. (2022) is indeed relevant as it considers decomposing the objective into individual loss and diversity for two specific cases (MSE and CE with probability averaging) and provides a useful motivation for our investigation of joint training. Whilst this work does provide preliminary evidence of poor performance under joint training (an empirical observation that was also reported and misdiagnosed previously in the literature as discussed extensively in our related work and App. E), it does not investigate why this occurs - the primary contribution of our work. Furthermore, all of the contributions of our work (as summarized in Fig 7), are entirely novel with respect to this work. \n  \n  $~$\n  \n  **Action taken**: We have integrated these works into our related work discussion, thank you for encouraging us to extend our literature review."}, {"review_id": "YOUteLvh74", "reviewer": "Reviewer_pZUZ", "summary": "Ensembles are a simple but powerful way to improve model performance. Typically, ensembles are used to improve performance by training each model independently and then using them jointly. However, unlike previous ML methods that require ensemble members to be trained individually, Deep Ensemble, an ensemble of deep learning models, can also be trained jointly. Considering that joint performance is the true objective of Deep Ensemble, it seems more natural to train jointly rather than individually. However, in practice, joint training leads to poor performance and poor generalization. The authors show theoretically and empirically that this is due to ensemble diversity. From this, they propose an augmented objective $\\mathcal{L}^\\beta$ that allows for a linear interpolation between joint and independent training. Finally, they hypothesize that this is due to learner collusion, a phenomenon where each model has a bias when jointly training an ensemble, and show through experiments that this is indeed the case.", "questions": "- Do you think it would be beneficial to drop a different fraction of base learners each step during the joint training of the ensemble to avoid learner codependency?\n- Minor comments:\n  - As mentioned above, the formatting of the reference is incorrect. At least the title of the paper should be visible.\n  - NeurIPS style rules do not allow vertical lines in tables. I would recommend removing the background color as well.\n  - Overall, I like the paper, but you spend too much space organizing and formalizing previous research; I think it would be better to add the experiments in the appendix to the main text.\n  - In Figure 6, it looks better to add the ImageNet title to the two graphs on the right for consistency.", "limitations": "I see no potential negative societal impact from this paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The notation is well organized and the formulas are easy to follow.\n- The problems covered in previous studies are well summarized and theoretically explained in Sections 2, 4, and 5.\n- The reason why joint training of Deep Ensemble fails is effectively explained through *learner collusion* and is well supported experimentally.", "weaknesses": "- The authors do not propose a way to prevent learner collusion while doing joint training, so there is not much contribution in terms of practicality. The possibility of partial joint training by weighting between independent training and joint training using an augmented objective is presented as a future work, which is already analyzed in various ways in Webb et al. (2020).\n- (Minor) The overall formatting of the paper is less polished and a bit difficult to read.\n    - Figures are scattered throughout the page. If possible, the figure should be located at the top of the page.\n    - The text in the figure is too small to read the results. Increase the size of the text so that it is not too different from the size of the main text.\n    - The lack of titles in the references makes it very difficult to identify the relevant papers. **Please fix this.**\n-----\n(Webb et al., 2020) [To Ensemble or Not Ensemble: When does End-To-End Training Fail?](https://arxiv.org/abs/1902.04422)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their constructive feedback. We have addressed the points raised below which have helped us to further improve our submissions clarity and contribution. \n  \n  $~$\n  \n* _Practical contribution_ - While we appreciate that the focus of this work was not on proposing new methods, we do believe that identifying and characterizing the limitations of joint training will still have a significant practical impact. We have already discussed several previous works which have directly attempted joint training (see Sec. 2 “joint training” & a deeper exposition of the limitations of the analyses of these works in App E) but it is likely that this issue is regularly unknowingly rediscovered by practitioners as joint training is a natural objective to consider. For example: (a) [1] reports training an ensemble sequentially due to “unstable” training dynamics when training simultaneously, (b) [2] introduces a regularization term to prevent their ensemble from “collapsing to degenerate solutions” and (c) [3] which argues that training for both loss and diversity on the same data may \"render the convergence point of the training process uncontrollable\". We hope that, at a minimum, our work can act as a canonical reference for this issue such that future works might avoid this common pitfall.  \n  \n  $~$\n  \n  Regarding the augmented objective, we would point out that our objective is more general than that of Webb et. al as it is defined for all twice or more differentiable loss functions. However, we agree that as a proposed solution this would be similar in spirit. In our work, we intended the augmented objective to primarily be a tool for analysis of the limitations of joint training rather than a complete solution. We do hope that the result indicating that low levels of $\\beta$ still performing reasonably well will inspire future methods that can overcome the limitations we have described. \n  \n  $~$\n  \n  **Action Taken**: We have added a note on the consequences of learner collusion to future ensemble research in our conclusion.  \n  \n  $~$\n  \n  [1] Pagliardini, Matteo, et al. \"Agree to Disagree: Diversity through Disagreement for Better Transferability.\" The Eleventh International Conference on Learning Representations. 2022.\n\n  [2] Lee, Yoonho, Huaxiu Yao, and Chelsea Finn. \"Diversify and disambiguate: Out-of-distribution robustness via disagreement.\" The Eleventh International Conference on Learning Representations. 2022.\n\n  [3] Pang, Tianyu, et al. \"Improving adversarial robustness via promoting ensemble diversity.\" International Conference on Machine Learning. PMLR, 2019.\n  \n  $~$\n  \n* _dropping a fraction of base learners each step during the joint training_ - We agree that this is a very natural solution to learner collusion, thank you for the suggestion! The hypothesis being that if we drop a sufficiently large portion of base learners in, say, each batch but still perform joint training on the remainder, this should cause inflated diversity to be actively harmful even on the training data and therefore force the ensemble to avoid this degeneracy. We decided to try out this idea which we describe next. \n  \n  $~$\n  \n  We repeat the setup on CIFAR-10 with ResNet-18 architecture as described in the paper. We train each model using the joint objective, but at each batch we drop a proportion $p \\in [0, 0.2, 0.4, 0.6]$ of randomly selected learners from the ensemble. We then investigate if this (a) reduces collusion and (b) improves ensemble performance. **The results of this experiment are included in the attached pdf**. We find that dropping learners does indeed significantly reduce the diversity score indicating a reduction in learner collusion (although a reasonably large proportion of learners is required to be dropped). Unfortunately, the improvement in performance by reducing collusion is negated by a decrease in individual performance due to the base learners becoming weaker on average. The reason why is apparent once we notice that by dropping a base learner from some proportion of batches, this is exactly equivalent to bootstrapping. While bootstrapping has been effective for ensembles of weak learners (e.g. random forest) it has already been shown to be harmful for deep ensembles [4]. Unfortunately, despite being a sensible idea, this resolution can only reduce the effect of learner collusion by simultaneously harming the performance of the ensemble due to bootstrapping.\n  \n  $~$\n  \n  Proposing methods to overcome learner collusion is a valuable but non-trivial direction for future work that we and, we hope, others in the research community will pursue. We believe that the comprehensive diagnosis of the issue we have presented in this work will provide a foundation for future methodological progress.  \n  \n  $~$\n  \n  **Action taken**: We have added the results of this experiment as a negative result in order to aid future research. \n  \n  $~$\n  \n  [4] Nixon, Jeremy, Balaji Lakshminarayanan, and Dustin Tran. \"Why are bootstrapped deep ensembles not better?.\" ''I Can't Believe It's Not Better!''NeurIPS 2020 workshop. 2020.\n  \n  $~$\n  \n* _Formatting issues_ - We wish to sincerely apologize to all four reviewers for a (silent) latex compiling error in the final version of our manuscript that resulted in the titles of the papers not appearing in the PDF. This was an unfortunate inconvenience for the reviewers. We do wish to highlight that several reviewers made positive comments about the writing and organization of this work – thus, the issues regarding the presentation are limited to simple **formatting errors which we have now rectified**.  Furthermore, we have implemented the further formatting suggestions from this reviewer and, should the paper be accepted, would ensure careful attention is paid during further polishing of the manuscript for a camera-ready version. \n"}, {"review_id": "PE1jh48a8F", "reviewer": "Reviewer_a4xj", "summary": "The authors aim to answer why joint training of ensembles does not work as well as separately training the members of the ensembles, before ensembling them. This is a well-known empirical fact, but the authors attempt to give a theoretical understanding of it, which is novel to my knowledge.\n\nTo reach their objective, they consider an expression for diversity (DIV) for probability averaging, which they motivate thru a (known) regression example. Then they generalize that expression to any twice differentiable loss function. \n\nThe authors study DIV in more depth by (upper) bounding it, then using it as a tunable regularization tool to interpolate between joint training and separate training of ensembles. \n\nThen the authors hypothesize that jointly training the ensemble may artificially bump up DIV without yielding proper generalization. They say that this happens because of \"learner collusion\" a phenomenon where two learners are displaced artificially by a constant bias term without changing the overall performance of the ensemble. They devise empirical measures to test that hypothesis (by studying generalization gaps, diversity explosion, learner codependence etc.)\n\n \n\n", "questions": "Q1: In Figure 2, why does Facebook seem to be different from the overall trend? \n\nQ2: If we get $\\beta \\approx 1$, then great, we have increased diversity and this gives us some hints as to why joint training might fail. However, as we vary $\\beta$, I do not gain any intuition. Is the collusion effect a monotonic one? It seems so from Figure 6. If there is some $\\beta > 0$ then do we always have that effect? Shouldn't intuition tell us that there should be a \"sweet spot\" for $\\beta$? If not, then why would we be interested in training learners jointly to any extent? For example, could we do something more explicit to prevent the learner's collusion?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1: The paper is really well written. Thanks to the authors for building my reading experience so well.\n\nS2: The explanation of the phenomenon (to my knowledge) is novel, and gives me a new way of thinking about the well-known failure case of jointly training ensembles. \n\nS3: It is great that the authors have theory that is paralleled with experiments on real-world data.", "weaknesses": "W1: Not sure I understand what happens when we vary $\\beta$. I think the paper can improve in a more detailed discussion about a more thorough study of the effect of collusion in the intermediate values of $\\beta$. See my Q2 in the questions below.\n\nW2: Typically, we expect to be able to devise better models after we have a better understanding of an underside phenomenon. The paper lacks a discussion about this. It's fine to leave it as future work, but it would be useful to discuss some directions a bit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank this reviewer for their constructive feedback. We have addressed the points raised below which have helped us to further improve our submissions clarity and contribution. \n  \n  $~$\n  \n* _Further discussion on the effects of varying $\\beta$_ - The non-linear relationship between $\\beta$ and the mean squared error is indeed a notable phenomenon that we only addressed in the appendix. Let us provide a more succinct discussion on this point. \n  \n  $~$\n  \n  Figures 2-4 do highlight a sudden jump in learner collusion as $\\beta \\to 1$ in the case of regression rather than a gradual increase.  Why does this occur? A useful explanation lies in the gradient-adjusted target (GAT) analysis in App D. To briefly summarize, this section considers applying individual training but with an adjusted target which is a function of the gradient of the ensemble error. Specifically, we adjust the targets to account for the errors of the ensemble. Formally, we optimize $\\frac{1}{M}\\sum_{j=1}^M (f_j - \\bar{y})^2$ where $\\bar{y}$ is the adjusted target given by $\\bar{y} = y - \\alpha \\cdot g$. Here $g$ is the ensemble loss gradient and $\\alpha$ is a step size parameter determining how big a step the target of an individual learner should take in order to account for the errors of the ensemble. In Thm. D.1 we show that this GAT objective is exactly equivalent to our augmented objective in Sec. 5 with $\\beta = (1 - \\frac{1}{(1 + \\alpha)^2})$. Given this relationship, we might notice that the gradient step $\\alpha$ grows exponentially as $\\beta \\to 1$ resulting in each individual learner having their targets biased excessively to account for the ensemble errors (**please see the new figure in the attached PDF visualizing this relationship**). Given this perspective, it is unsurprising that learner collusion does not grow linearly with $\\beta$. \n  \n  $~$\n  \n  The reviewer's question regarding a \"sweet spot\" of $\\beta$ for optimizing diversity is an interesting one. Considering that joint training only occurs at exactly $\\beta = 1$ we had the same intuition that low values of $\\beta$ may still be beneficial. Interpolating values of $\\beta$ sometimes appear to achieve the best performance (see e.g. ResNet-18 in Fig. 6 left and CNN in Fig. 14) but often this is not the case (e.g. Fig. 15). We hope that, building upon our findings, future work will be able to provide methods for optimizing for higher levels of diversity whilst avoiding learner collusion and thereby enable us to study the effects of increasing diversity _without_ this obscuring degeneracy. \n  \n  $~$\n    \n  **Action taken**: We have added a succinct summary of this discussion to the main text. We have also added this new plot and refined the discussion in App D.\n  \n  $~$\n  \n* _Discussion in improved methods based on our findings_ - We agree that our work could benefit from some additional discussion on future methodological work building upon ours. This is something we are happy to expand upon in our conclusion. Reviewer pZUZ suggested randomly dropping a subset of base learners during training. The hypothesis being that if we drop a sufficiently large portion of base learners in, say, each batch but still perform joint training on the remainder, this should cause inflated diversity to be actively harmful even on the training data and therefore force the ensemble to avoid this degeneracy. We decided to try out this idea which we describe next. \n  \n  $~$\n  \n  We repeat the setup on CIFAR-10 with ResNet-18 architecture as described in the paper. We train each model using the joint objective, but at each batch we drop a proportion $p \\in [0, 0.2, 0.4, 0.6]$ of randomly selected learners from the ensemble. We then investigate if this (a) reduces collusion and (b) improves ensemble performance. **The results of this experiment are included in the attached pdf**. We find that dropping learners does indeed significantly reduce the diversity score indicating a reduction in learner collusion (although a reasonably large proportion of learners is required to be dropped). Unfortunately, the improvement in performance by reducing collusion is negated by a decrease in individual performance due to the base learners becoming weaker on average. The reason why is apparent once we notice that by dropping a base learner from some proportion of batches, this is exactly equivalent to bootstrapping. While bootstrapping has been effective for ensembles of weak learners (e.g. random forest) it has already been shown to be harmful for deep ensembles [1]. Unfortunately, despite being a sensible idea at the outset, this resolution can only reduce the effect of learner collusion by simultaneously harming the performance of the ensemble due to bootstrapping.\n  \n  $~$\n  \n  Proposing methods to overcome learner collusion is a valuable but non-trivial direction for future work that we and, we hope, others in the research community will pursue. We believe that the comprehensive diagnosis of the issue we have presented in this work will provide a foundation for future methodological progress.  \n  \n  $~$\n  \n  **Action taken**: We have added the results of this experiment as a negative result in order to aid future research and expanded upon our discussion of future work in the conclusion. \n  \n  $~$\n  \n  [1] Nixon, Jeremy, Balaji Lakshminarayanan, and Dustin Tran. \"Why are bootstrapped deep ensembles not better?.\" ''I Can't Believe It's Not Better!''NeurIPS 2020 workshop. 2020.\n  \n  $~$\n  \n* _Facebook dataset_ - Please see our response in the general comment section."}]

Author rebuttal:
We thank all four reviewers for their constructive feedback. We have found their feedback to be instructive with their suggestions and questions helping us to further improve our submissions clarity and contribution.
  
  $~$
  
* _Formatting error_ - We wish to sincerely apologize to all four reviewers for a (silent) latex compiling error in the final version of our manuscript that resulted in the titles of the papers not appearing in the PDF. This was an unfortunate inconvenience for the reviewers. We do wish to highlight that several reviewers made positive comments about the writing and organization of this work – thus, the issues regarding the presentation are limited to **simple formatting errors which we have now rectified**.  Furthermore, we have implemented further formatting suggestions from pZUZ and, should the paper be accepted, would ensure careful attention is paid during further polishing of the manuscript for a camera-ready version.
  
  $~$
  
* _Clarification on Figure 2 (Facebook dataset)_ - We thank reviewers **m1Sy** & **a4xj** who highlighted that it would be beneficial to comment on this result, we certainly agree that this particular result requires some further clarification. Unlike all other experiments throughout this paper, the test error on this task is relatively flat across all values of $\beta$ (after accounting for the standard errors). Therefore, one might ask whether this result contradicts any of our claims. 
  
  $~$
  
  We begin by revisiting our analytical analysis of the regression case in App F. While this section mathematically uncovers how optimizing for diversity results in diversity-inflating bias terms, it does not guarantee that it will result in worse test performance. It is at least conceptually possible to construct a task in which diversity is not desirable and, therefore, learner collusion is not harmful. Further analysis indicates that this dataset is such a case (we also highlight that this is quite a popular regression task in the machine learning literature e.g. [1-3]). The goal in this task is to predict the number of impressions a post will obtain given several metrics. While this is a challenging task due to high variance in the response, the signal that does exist seems to be almost entirely contained within a single variable: “number of likes” resulting in no need for true diversity and, therefore, no substantial impact due to learner collusion. We verify this claim by matching the reported test performance with a single decision tree of depth 3 which (a) matches the performance of a neural network (MSE = 2.18 $\pm$ 0.66 over 5 runs) and (b) has feature importance dominated by this single “number of likes” variable (in all 5 runs the tree uses this feature as its primary split). 
  
  $~$
  
  Finally, we note that this dataset is still valuable as (1) this experiment was investigating the existence of learner collusion – which certainly still occurs on this task, and (2) selective inclusion of datasets based on nice characteristics should be avoided – indeed, we hope that this result and our accompanying clarification may be instructive for future readers. Readers might also be interested to note that we have **extensively broadened our evaluation on ImageNet** (from Table 1 and Figure 6 RHS) with eight additional architectures and various ensemble sizes. All models are trained from scratch and **we include these new results in the attached PDF document** (Table 5) where we consistently find that joint training performs significantly worse than independent training, thus reinforcing the claims of our work. 
  
  $~$
  
  **Action taken**: We have added a comment to the paper clarifying this result and significantly extended our experiments on ImageNet.
  
  $~$
  
  [1] Romano, Yaniv, Evan Patterson, and Emmanuel Candes. "Conformalized quantile regression." Advances in neural information processing systems 32 (2019).

  [2] Sesia, Matteo, and Yaniv Romano. "Conformal prediction using conditional histograms." Advances in Neural Information Processing Systems 34 (2021): 6304-6315.

  [3] Jeffares, Alan, et al. "TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization." The Eleventh International Conference on Learning Representations. 2022.

  
  $~$
  

Meta-review: While there was something of a split in scores on this paper, the balance of sentiment seems to me clearly in favor of acceptance, especially given that the lowest score was given by a reviewer who did not engage with the rebuttal (which, in my opinion and that of at least some of the other reviewers during later discussions, addresses many of that reviewer's concerns). While on one level I agree with Reviewer XMTw that it's not shocking that jointly trained deep ensembles learn to collude in a problematic way, demonstrating and explaining that phenomenon cleanly and clearly is a significant and valuable contribution.
Decision: Accept (poster)