SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alan Jefferes
Tennison Liu
University of Cambridge t1522@cam.ac.uk
University of Cambridge jc2133@cam.ac.uk
Ensembles of machine learning models have been well established as a powerful method of improving performance over a single model. Traditionally, assembling algorithms train their base learners independently or sequentially with the goal of optimizing their joint performance. In the case of deep ensembles of neural networks, we are provided with the opportunity to directly optimize the true objective: the joint performance of the ensemble as a whole. Surprisingly, however, directly minimizing the loss of the ensemble appears to rarely be applied in practice. Instead, most previous research trains individual models independently with assembling performed post hoc. In this work, we show that this is for good reason - joint optimization of ensemble loss results in degenerate behavior. We approach this problem by decomposing the ensemble objective into the strength of the base learners and the diversity between them. We discover that joint optimization results in a phenomenon in which base learners collude to artificially inflate their apparent diversity. This pseudo-diversity fails to generalize beyond the training data, causing a larger generalization gap. We proceed to comprehensively demonstrate the practical implications of this effect on a range of standard machine learning tasks and architectures by smoothly interpolating between independent training and joint optimization.<sup>1</sup>
Deep ensembles [1] have proven to be a remarkably effective method for improving performance over a single deep learning model. This success has been attributed to deep ensembles exploring multiple modes in the loss landscape [2], in contrast to variational Bayesian methods, which generally attempt to better explore a single mode [e.g. 3]. Recent work has proposed an alternative view suggesting that deep ensembles' success can be explained as being an effective method of scaling smaller models to match the performance of a single larger model [4]. Regardless of the specific mechanism, deep ensembles' empirical success has been at the heart of numerous state-of-the-art deep learning solutions in recent years [e.g. 5, 6, 7].
Historically, non-neural network ensembles such as a bagging predictor [8] or a random forest [9] were generally trained independently and aggregated at test time. Sequential training methods such as boosting [10] were also developed to produce more collaborative ensembles. Training individually and evaluating jointly was sensible for these methods, in which the base learners were typically
$^{1}$ Code is provided at https://github.com/alanjeffares/joint-ensembles.
decision trees that require individual training. Formally, for a single example  $\mathbf{x}$ , a loss function  $\mathcal{L}$  and denoting the prediction of learner  $j$  on  $\mathbf{x}$  as  $\mathbf{f}_j \in \mathbb{R}^d$  with target  $\mathbf{y} \in \mathbb{R}^d$ , we define the joint objective as  $\mathcal{L}\left(\frac{1}{M} \sum_{j=1}^{M} \mathbf{f}_j, \mathbf{y}\right)$  and the independent objective as  $\frac{1}{M} \sum_{j=1}^{M} \mathcal{L}(\mathbf{f}_j, \mathbf{y})$ . Interestingly, modern deep ensembles are also generally trained independently and evaluated jointly. This is less intuitive as the performance of the ensemble as a whole (i.e., the joint objective) is the true objective of interest and jointly training a deep ensemble is quite natural for the backpropagation algorithm. Indeed, one might expect there to be no need for the proxy objective of independent training in this case, as the joint objective should find a solution in which individual learners collaborate to minimize the ensemble's loss. In practice, however, independent training generalizes better than joint training in this context. A phenomenon that we demonstrate in Section 3 and investigate throughout this work.
In Section 4, we demonstrate that ensemble diversity is the foundation upon which understanding the limitations of joint training sits. We show that any twice differentiable joint loss function of an ensemble can be decomposed into the individual losses and a second term, which can be interpreted as ensemble diversity. This diversity term encourages some level of ambiguity among the base learners' predictions, a generalization of well-established work in the regression setting [11]. Mathematically, this term is exactly the difference between independent and joint training and, therefore, must encapsulate the disparity in performance. In Section 5, we introduce a scalar weighting on the diversity term allowing us to smoothly interpolate between the extremes of fully independent and fully joint training.
Then, in Section 6, we diagnose and empirically verify a precise cause of the sub-optimal performance of jointly trained ensembles. We show that the additional requirement for diversity in the training objective can be trivially achieved by an adverse effect we term learner collusion. This consists of a set of base learners artificially biasing their outputs in order to trivially achieve diversity while ensuring that these biases cancel in aggregate as illustrated in Figure 1. We perform extensive experiments verifying this hypothesis. We then show that although training loss can be driven down by learner collusion, this does not generalize beyond the training set, resulting in a larger generalization gap than independent training. Finally, we examine the practical implications of learner collusion on standard machine learning benchmarks.
Figure 1: Learner collusion. An illustrative example in which a subset of base learners  $(f_{1} \& f_{2})$  collude in output space by shifting their predictions in opposing directions such that the ensemble prediction  $(\bar{f})$  is unchanged, but diversity is inflated.
This work addresses the pervasive yet relatively understudied phenomenon that directly optimizing a deep ensemble fails. (1) We unify various strands of the ensemble literature to provide a deep, practical categorization of ensemble training and diversification. (2) We present and verify a comprehensive explanation for this counter-intuitive limitation of deep ensembles with both theoretical and empirical contributions. (3) We investigate the catastrophic consequences of this issue on practical machine learning tasks. As this investigation takes a somewhat non-standard format, we provide an abridged summary of this work and its technical contributions in Appendix A.
Independent training. This refers to ensembles<sup>3</sup> in which each base learner is optimized independently, with no influence from the other ensemble members either explicit or implicit. This includes many non-neural ensemble techniques in which diversity between the base learners results in ensemble performance better than any individual. This diversity is often achieved by sampling the input space [e.g. bagging in 8] and the feature space [e.g. random forest in 9]. In the case of deep ensembles of neural networks, independent training is also generally applied, with diversity achieved using random weight initialization and stochastic batching [1, 2]. Interestingly, bootstrap sampling of the inputs is ineffective in the case of deep ensembles [12]. Several notable state-of-the-art results
2In Section 4 we further generalize this definition to account for (a) non-equal weighting between the predictions of ensemble members and (b) aggregating ensemble outputs before normalization (i.e. score averaging).
3In this work we consider ensembles in the least prescriptive sense (i.e. any algorithm that aggregates multiple target predictions).
have been achieved by ensembling several independently trained but individually performant neural networks. For example, the groundbreaking results in the ImageNet challenge were often achieved by independently trained ensembles such as VGG [13] GoogLeNet [5], and AlexNet [14]. Seq2Seq ensembles of long short-term memory networks [15] achieved remarkable performance for neural translation. More recently in the tabular domain, ensembles of transformers have outperformed state-of-the-art non-neural methods [7].
Ensemble-aware individual training. This refers to the case where each base learner is still being optimized as an individual, but with some additional technique being applied to coordinate the ensemble members. A prominent approach is sequentially trained, boosting ensembles in which each new member attempts to improve upon the errors of the existing members (e.g. AdaBoost [10] & gradient boosting [16]). In the case of deep ensembles, an alternative approach consists of training each base learner with a standard individual loss with the addition of an ensemble-level regularization term designed to manage diversity. This often consists of an f-divergence between the learners' predictive distributions. Examples include the pairwise  $\chi^2$ -divergence [17],  $KL$ -divergence, but between the learner and the ensemble [18], and a computationally efficient, kernelized approximation of Renyi-entropy [19]. Another prominent regularization approach is based on the seminal idea of negative correlation learning (NCL) which penalizes individual learners through a correlation penalty on error distributions [20, 21]. This penalty weakens the relationship with other learners and controls the trade-off between bias, variance, and covariance in ensemble learning [22]. Recent works have adapted NCL for improved computational efficiency with large base learners [23] and attempt to generalize NCL beyond the regression setting [24]. [25] encourages ensemble diversity in weight and functional space using a "repulsive" term while other works show that a single learner can be viewed as an ensemble with diversity encouraged implicitly (e.g. [26, 27]). The efficient BatchEnsemble [28] can also be characterized as an ensemble-aware individual training approach.
Joint training. This refers to the case where the ensemble's predictions are optimized directly as if it were a single model. Intuitively, one might naturally expect joint training to outperform the previously discussed methods as it directly optimizes (at training time) the test metric of all three approaches - ensemble loss. Empirically, however, this is not the case as joint training achieves inferior performance in general. [29] reported poor results when joint training averages the base learners' output scores but excellent performance when they are averaged at the log-probability or loss levels. However, later work showed that the two alleged success cases were, in fact, implementing independent training rather than joint training unbeknownst to the authors [30]. This later work also reported poor generalization for score-averaged ensembles. We provide a detailed clarification and gradient analysis in Appendix E. Elsewhere, [31] experienced poor performance from joint training for both score and probability averaging. Concurrently to this work, [32, 33] also found that joint training consistently resulted in worse empirical performance. To put the matter to rest, we demonstrate these issues with joint training in the next section.
3 Empirically Evaluating Joint Training
In this section, we briefly demonstrate the effect that we investigate throughout this work. That is, when evaluating the ensemble loss at test time, optimizing for that same ensemble loss objective at training time (i.e. joint training) is less effective than optimizing each of the base learners independently (i.e. independent training). We do this by performing a straightforward comparison of the two methods. We compare the test set performance of independent and joint training on ImageNet [34]. We consider several architectures as base learners of an ensemble of various sizes determined by computational limitations. We report test set top-1 and top-5 accuracy for both independently and jointly trained ensembles. These results are included in Table 1. Complete experimental details for all experiments in this work are provided in Appendix B.
Here, we observe the counter-intuitive phenomenon that independently trained ensembles consistently outperform jointly trained ensembles, despite all models being evaluated using the joint objective. This is consistent with previous works that have attempted to apply joint training [30, 31]. In the following sections, we will proceed to investigate and explain this effect through the lens of ensemble diversity.
Summary: Joint training performs considerably worse than independent training in practice.
Table 1: Joint training on ImageNet. We compare independent to joint training across a range of architectures trained from scratch on the ImageNet dataset. Top 1 & 5 test accuracy is reported for both methods, where independent training consistently performs better.
Base learner	Number of learners	Independent	Joint
Top-1	Top-5	Top-1	Top-5
ResNet-18 [35]	5	68.10%	88.43%	62.74%	84.03%
ResNet-34 [35]	3	73.11%	91.25%	68.91%	88.01%
DenseNet [36]	3	70.62%	89.51%	60.81%	81.35%
ShuffleNet V2 (0.5×) [37]	10	61.00%	81.94%	53.41%	74.88%
ShuffleNet V2 (1×) [37]	5	68.56%	87.64%	61.96%	81.56%
SqueezeNet V1.1 [38]	5	56.25%	78.65%	42.69%	65.24%
MobileNet V3-small [39]	5	67.43%	86.82%	57.34%	78.30%
MobileViT [40]	3	61.29%	82.84%	54.69%	76.79%
4 Diversity in Ensemble Learning
To understand the limitations of joint training, we will need to examine the joint objective through the lens of diversity. In this section, we argue for a unified definition of diversity which generalizes much of the existing literature on the subject.
We begin with some notation. We consider ensembles made up of  $M$  individual learners. The  $j$ 'th individual learner maps an input  $\mathbf{x} \in \mathbb{R}^p$  to a  $d$ -dimensional output space such that  $\mathbf{f}_j: \mathbb{R}^p \to \mathbb{R}^d$  and attempts to learn some regression target(s)  $\mathbf{y} \in \mathbb{R}^d$  or, in the case of classification, labels which lie on a probability simplex  $\Delta^d$ . Without loss of generality, we can refer to  $\mathbf{y} \in \mathcal{V}$ . We primarily consider combining individual learner predictions into a single ensemble prediction as a weighted average  $\sum_{j=1}^{M} w_j \mathbf{f}_j(\mathbf{x}) = \bar{\mathbf{f}}(\mathbf{x})$  where  $w_j \in \mathbb{R}$  denotes the scalar weights. Although not strictly required for much of this work, we will generally assume that these weights satisfy the properties  $\sum_{j=1}^{M} w_j = 1$  and  $w_j \geq 0 \forall j$ , ensuring that each individual learner can be interpreted as predicting the target. Scalars are denoted in plain font and vectors in bold. We generally drop the dependence on  $\mathbf{x}$  in our notation such that individual and ensemble predictions are denoted  $\mathbf{f}_j$  and  $\bar{\mathbf{f}}$  respectively. Finally, we formalize the respective losses of the ensemble and the base learners in Definition 4.1.
Definition 4.1 (Ensemble and Individual Loss). For a given loss function  $\mathcal{L}:\mathcal{Y}^2\to \mathbb{R}^+$ , we define:
(a) The overall ensemble loss ERR  $\coloneqq \mathcal{L}(\bar{\mathbf{f}},\mathbf{y})$
(b) The weighted individual loss of the base learners  $\overline{\mathrm{ERR}}\coloneqq \sum_{j = 1}^{M}w_{j}\mathcal{L}(\mathbf{f}_{j},\mathbf{y})$
Intuitively, ERR measures the predictive performance of the ensemble, while  $\overline{\mathrm{ERR}}$  measures the aggregate predictive performance of the individual ensemble members.
4.2 A Generalized Diversity Formulation
Diversity in regression ensembles. The prevailing view of diversity in regression ensembles began in [11], which showed that the mean squared error of a regression ensemble can be decomposed into the mean squared error of the individual learners and an additional ambiguity term which measures the variance of the individual learners around the ensemble prediction showing
$$
\sum_ {j} w _ {j} \left(f _ {j} - \bar {f}\right) ^ {2} = \sum_ {j} w _ {j} \left(f _ {j} - y\right) ^ {2} - (\bar {f} - y) ^ {2}, \tag {1}
$$
$$
\mathrm {D I V} = \overline {{\mathrm {E R R}}} - \mathrm {E R R}.
$$
This decomposition provides an intuitive definition of diversity whilst also showing that a jointly trained ensemble, that is one that simply optimizes ERR, encourages diversity by default due to it being baked into the objective. Later work from the evolutionary computation literature proposed an apparently alternative measure of diversity called Negative Correlation Learning, which argues
that lower correlation in individual errors is equivalent to higher diversity [21]. The apparent disagreement between these two seemingly valid views on diversity was resolved when it was shown that they are in fact equivalent up to a scaling factor of the diversity term [22].
Generalized diversity. Motivated by the regression case, we will argue that the difference between the individual and ensemble errors has a natural interpretation as the diversity among the predictions of the ensemble members for any twice (or more) differentiable loss function. In fact, this generalizes several existing notions of diversity proposed throughout the literature for specific cases. Specifically, we define diversity as in Definition 4.2 and will proceed to justify this definition in what follows.
Definition 4.2 (Diversity). The diversity of an ensemble of learners (DIV) is the difference between the weighted average loss of individual learners and the ensemble loss:
$$
\operatorname {D I V} := \overline {{\operatorname {E R R}}} - \operatorname {E R R}
$$
Diversity in classification ensembles. It is natural to next verify that this definition is appropriate in the classification setting. We proceed by concretely deriving the diversity term in this setting. There are two standard methods for aggregating base learners' predictions into a single ensemble prediction in classification. Averaging at the score (preactivation) level and averaging at the probability level. We address each of these in turn.
① Score averaging. We consider a classification task with target distribution  $\mathbf{y} \in \mathcal{Y} \in \Delta^d$  with  $d$  classes, and consider ensembling by averaging the base learners' scores, before applying the normalizing map  $\phi$  (typically a softmax). We define  $\mathbf{p}_j: \mathcal{X} \to \Delta^d \equiv \phi \circ \mathbf{f}_j$  as the normalized predictive distribution for learner  $j$  and  $\bar{\mathbf{q}} = \phi \circ \bar{\mathbf{f}}$  the predictive distribution for the ensemble, where  $\bar{\mathbf{f}} = \sum_{j=1}^{M} w_j \mathbf{f}_j$  as before. In this setting, it can be shown [41] that
$$
\mathrm {D I V} = \sum_ {j = 1} ^ {M} w _ {j} D _ {K L} (\mathbf {y} | | \mathbf {p} _ {j}) - D _ {K L} (\mathbf {y} | | \bar {\mathbf {q}}) = \sum_ {j = 1} ^ {M} w _ {j} D _ {K L} (\bar {\mathbf {q}} | | \mathbf {p} _ {j}),
$$
where  $D_{KL}$  is the  $KL$ -divergence. Intuitively, this diversity term measures the weighted divergence between each ensemble member's predictive distribution and the full ensemble's predictive distribution, a sensible expression for diversity. It should be noted that this decomposition holds for normalizing functions beyond the softmax [30]. Specifically, this exact expression holds for all exponential family normalization functions with score averaging. This allows the diversity decomposition described in this work to apply to disparate settings as demonstrated by its recent application in modeling diversity among the Poisson spike train outputs of spiking neural networks [42].
$\Theta$  Probability averaging. This setting diverges from score averaging by setting the ensemble predictive distribution  $\bar{\mathbf{p}}$  to be the average of the base learners' probabilities such that  $\bar{\mathbf{p}} = \sum_{j=1}^{M} w_j (\phi \circ \mathbf{f}_j)$ . This is a more natural strategy, averaging the scale-invariant probabilities while also providing each base learner with unique gradients during backpropagation (see a detailed gradient analysis in Appendix E). The resulting expression for diversity is included in Theorem 4.3.
Theorem 4.3. For a probability averaged ensemble classifier, trained using cross-entropy loss  $\mathcal{L}_{CE}$  with  $k^{\star} \in \mathcal{K}$  denoting the ground truth class, diversity is given by
$$
\mathrm {D I V} = \sum_ {j = 1} ^ {M} w _ {j} \mathcal {L} _ {C E} (\mathbf {y}, \mathbf {p} _ {j}) - \mathcal {L} _ {C E} (\mathbf {y}, \bar {\mathbf {p}}) = \sum_ {j = 1} ^ {M} w _ {j} \cdot y (k ^ {\star}) \log \frac {\bar {p} (k ^ {\star})}{p _ {j} (k ^ {\star})}
$$
Proof. Please refer to Appendix C.3.
Remark 4.4. It is not immediately obvious that this expression has a natural interpretation as diversity. However, in Appendix C.3 we show it to be well aligned with the notion of diversity used throughout this work by analysis of the distribution of  $p_j(k^\star)$  around  $\bar{p}(k^\star)$ .
Beyond the KL-divergence family, alternative loss functions also result in sensible diversity expressions. Brier score, expressed as  $(\bar{\mathbf{p}} - \mathbf{y})^T (\bar{\mathbf{p}} - \mathbf{y})$ , can be shown to obtain  $\mathrm{DIV} = \sum_{j} w_j (\bar{\mathbf{p}}_j - \mathbf{p})^T (\bar{\mathbf{p}} - \mathbf{p}_j)$ , the variance of the base learner probabilities around the ensemble probabilities [4].
For 0-1 loss and the case of majority voting, diversity is also measured by the disagreement between the (categorical) predictions of the base learners and the ensemble prediction [43].
Diversity in a general setting. The previous paragraphs verified the diversity definition introduced in Definition 4.2 to be a sensible interpretation for several common loss functions. A more general question can be asked — Is DIV reasonable for any choice of loss functions? The answer is yes, and this turns out to be no coincidence. To demonstrate why, we analyze the properties of the diversity expression for any twice differentiable loss function and discover a general form that permits a clear interpretation as diversity. To do so, we present Theorem 4.5, a multidimensional generalization of one proposed in [44] which, in turn, generalizes our notion of diversity and analysis of joint training to practically any loss function.
Theorem 4.5. [Diversity for General Loss] For any loss function  $\mathcal{L}:\mathcal{Y}^2\to \mathbb{R}^+$  with  $\dim (\mathcal{V})\in \mathbb{N}^*$  that is at least twice differentiable, the loss of the ensemble can be decomposed into
$$
\mathrm {D I V} = \frac {1}{2} \sum_ {j = 1} ^ {M} w _ {j} \left(\mathbf {f} _ {j} - \bar {\mathbf {f}}\right) ^ {\intercal} \cdot \mathcal {H} _ {f} ^ {\mathcal {L}} \left(\mathbf {f} _ {j} ^ {*}, \mathbf {y}\right) \cdot \left(\mathbf {f} _ {j} - \bar {\mathbf {f}}\right)
$$
$$
w i t h \mathbf {f} _ {j} ^ {*} = \bar {\mathbf {f}} + c _ {j} (\mathbf {f} _ {j} - \bar {\mathbf {f}}),
$$
where  $\mathcal{H}_f^{\mathcal{L}}$  is the Hessian of  $\mathcal{L}$  with respect to its first argument and some  $c_{j} \in [0,1]$  for all  $j \in [M]$ .
Proof. Please refer to Appendix C.2.
Here, the measure of diversity is obtained by performing a Taylor expansion on the loss of the output of  $\mathbf{f}_i$  near the ensemble output  $\bar{\mathbf{f}}$  and using Lagrange's remainder theorem to obtain an exact equality. Additionally,  $\mathbf{f}_i^*$  is an uncertain number, which approaches a constant in the limit. This diversity admits the same interpretation as the variance of learners around the ensemble, but is now weighted by a term dependent on the second derivative of the loss function.