SlowGuess/NeurIPS_2023_OCR huggingface dataset:
One can easily verify that this generalized definition encompasses the uni-dimensional MSE ambiguity decomposition in Equation (1), where  $\mathcal{L}''(f_i^*, y) = 2$ . Similarly, the multidimensional MSE  $\mathcal{L}(\mathbf{f}, \mathbf{y}) = \| \mathbf{f} - \mathbf{y} \|_2^2$  leads to the Hessian  $\mathcal{H}_f^{\mathcal{L}} = 2\mathbf{I}$ , where  $\mathbf{I}$  is the identity matrix on  $\mathcal{V}$ . From this, we get the multi-dimensional generalization  $\mathrm{DIV} = \sum_{j} w_j \| \mathbf{f}_j - \bar{\mathbf{f}} \|^2 \geq 0$ . Importantly, diversity has the same non-negative property (DIV  $\geq 0$ ) for any loss function with semi-positive definite Hessian  $\mathcal{H}_f^{\mathcal{L}} \succeq 0$ , such that ensemble performance is always better than individual learner performance. Another typical example is the classification loss  $\mathcal{L}(\mathbf{f}, \mathbf{y}) = -\mathbf{y}^\top \log \mathbf{f}$ . Here, it is trivial to show that  $\mathcal{H}_f^{\mathcal{L}}(\mathbf{f}, \mathbf{y}) = \mathrm{diag}(\mathbf{y} \oslash \mathbf{f}^2)$ , where  $\oslash$  denotes the component-wise division. Again, this Hessian matrix is positive semi-definite for  $\mathbf{y}$ ,  $\mathbf{f} \in \Delta^d$  so that  $\mathrm{DIV} \geq 0$ . Revisiting our previous question, we verify that, for at least twice-differentiable loss functions (e.g. logistic loss:  $\mathcal{L} = \log (1 + \exp^{-y f})$  & exponential loss:  $\mathcal{L} = \exp^{-y f}$ ), DIV admits a generalized interpretation as the variance of individual predictors around ensemble prediction. This makes a robust case for Definition 4.2, which provides a principled estimate of diversity (as opposed to heuristic measures) and can be easily obtained by calculating the difference between ensemble loss and weighted average learner loss.
Summary: The difference between independent loss and joint loss has a natural interpretation as diversity, providing a robust generalization of much of the existing literature.
5 An Augmented Objective
Given this generalized definition of diversity, one might naturally consider placing a scalar weighting on its contribution, resulting in an augmented training objective. In fact, such an approach has already been applied for special cases of diversity including regression [22], score averaging [30], and, concurrently to this work, in [33, 32]. Throughout our experiments, we will consider this augmented objective in Equation (2), allowing us to analyze the effects of smoothly transitioning between independent and joint training.
$$
\begin{array}{l} \mathcal {L} ^ {\beta} (\bar {\mathbf {f}}, \mathbf {y}) := \overline {{\operatorname {E R R}}} - \beta \cdot \operatorname {D I V} \tag {2} \\ = (1 - \beta) \cdot \overline {{\mathrm {E R R}}} + \beta \cdot \mathrm {E R R}. \\ \end{array}
$$
When  $\beta = 0$ , this amounts to training each learner independently (i.e. independent training) and  $\beta = 1$  corresponds to optimizing the loss of the ensemble as a whole (i.e. joint training). Clearly, intermediate values of  $\beta$  can be interpreted as interpolating between the two.
When examining Equation (2), one might be tempted to encourage more diversity by setting  $\beta > 1$ . Previous work by Brown et al. [45] showed that, in the case of regression, optimization can become degenerate for larger values of  $\beta$ , proving an upper bound of  $\beta < \frac{M}{M - 1}$ . More recently, [30] showed that in the case of score-averaged ensembles using their modular loss, the ensemble loss ERR was unbounded from below for  $\beta > 1$  – i.e. DIV can trivially be increased faster than its corresponding increase in  $\overline{\mathrm{ERR}}$ , thus (superficially) minimizing the loss. In Theorem 5.1 we generalize this same bound on  $\beta$  to all loss functions and further include the case of probability-averaged ensembles.
Theorem 5.1. [Pathological Behavior for  $\beta > 1$ ] We consider a target set  $\mathcal{Y} = \mathbb{R}^d$  or  $\mathcal{Y} \subset \mathbb{R}^d$  being a compact and convex subset of  $\mathbb{R}^d$ . Let  $\mathcal{L}: \mathcal{Y}^2 \to \mathbb{R}^+$  be a continuous and finite function on the interior  $\operatorname{int}(\mathcal{Y}^2)$ . We assume that the loss is not bounded from above on  $\mathcal{Y}^2$  in such a way that there exists  $\mathbf{y} \in \operatorname{int}(\mathcal{Y})$  and a sequence  $(\mathbf{y}_t)_{t \in \mathbb{N}} \subset \mathcal{Y}$  such that  $\mathcal{L}(\mathbf{y}_t, \mathbf{y}) \to +\infty$  for  $t \to \infty$ . If  $\beta > 1$ , then the augmented loss  $\mathcal{L}^\beta$  defined in Equation (2) is not bounded from below on  $\mathcal{Y}^2$ .
Proof. Please refer to Appendix C.4.
Remark 5.2. Note that we consider the case where  $\mathcal{V}$  is a compact and convex subset of  $\mathbb{R}^d$  to include classifiers, for which the target set is a probability simplex  $\Delta^{d - 1}$ .
As a consequence of the above theorem, one needs to impose the tighter bound  $\beta \leq 1$  in order to have a well-defined optimization objective. Furthermore, intermediate values where  $\beta \in (0,1)$  may have an additional interpretation beyond interpolating between independent and joint training. In Appendix D we show that, in the regression setting,  $\mathcal{L}^{\beta}$  is exactly equivalent to applying independent training while adjusting the targets to account for the errors of the ensemble as measured by its gradient. We derive an exact equivalence between  $\beta$  in  $\mathcal{L}^{\beta}$  and the magnitude of the adjustment of the target in this gradient-adjusted target setting.
Summary: An interpolation between independent training and joint training is a well-motivated and useful objective but, to avoid pathological behavior, one should not extrapolate to higher diversity.
6 Why Does Joint Training Fail?
Given the universal interpretation of diversity developed in the previous sections paired with its exclusive impact on the joint training objective, we now turn our attention to answering the question - why does joint training of deep ensembles perform worse than independent training? Some previous works have attempted to partially address this question. [29] suggested that score averaging fails as the softmax function "distorts" the averages. However, little evidence exists to support this claim. When faced with the same issue, [31] suggested the issue could be due to score averaged ensemble members receiving identical gradient values. We address this point in a gradient analysis in Appendix E, where we (a) point out that the optimal solution to joint objective would still achieve equal or lower loss for score averaging, and (b) show that probability averaged ensemble members receive variable gradient expressions and still suffer from poor performance. Finally, [30] noticed vastly different test set performances among the base learners which they referred to as model dominance. Indeed, this is a symptom of the underlying issue with joint training which we describe and verify throughout the rest of this section. The explanation we propose for the poor empirical results achieved by joint training consists of two claims:
Claim 1: the additional diversity term in the joint learning objective (Equation (2)) results in a dual objective. The DIV term can always be artificially inflated if multiple learners collude to bias their predictive distributions in opposing directions which cancel out in aggregate and, therefore, is easily maximized across the training data. We refer to this phenomenon as learner collusion.
Claim 2: while learner collusion can still minimize the loss on the training data, this solution may contain little genuine diversity resulting in a solution that achieves worse generalization than independent training. In Appendix F, we provide a detailed illustrative example of learner collusion for the regression case from an optimization perspective.
In Section 6.1 we empirically demonstrate this learner collusion effect is emergent. Then, in Section 6.2, we show that the low training loss achieved by learner collusion generalizes poorly onto unseen data. Finally, in Section 6.3, we establish the practical implications of learner collusion on standard tasks.
6.1 Diagnosing Learner Collusion
In this section, we focus on claim 1 and empirically investigate the extent to which learner collusion does occur. Recall that the effect being investigated is that a subset of non-diverse base learners can trivially adjust their similar, or even identical, predictions to appear diverse (i.e. Figure 1). If learner collusion does indeed take place we suggest that the following three effects should be observable empirically: (1) the relative magnitude of diversity should be excessive relative to the ensemble loss, (2) if we remove constant output biases from the individual learners, the ensemble diversity should decrease significantly, and (3) learners should become highly dependent on each other such that removing a small subset of base learners is detrimental to the ensemble performance. We investigate each of these effects in turn.
We perform these experiments on four tabular datasets of the style upon which ensemble methods typically excel [46]. Here we focus on regression as it offers the most controlled setting to isolate the learner collusion phenomenon (see Appendix F), but in later experiments we demonstrate its effects in the classification setting too. We smoothly interpolate between independent and joint training using the augmented objective controlled by  $\beta$  from Section 5. We report mean  $\pm$  one standard deviation evaluated on a test set throughout. An extended description of the experimental protocol for these experiments is provided in Appendix B.
Diversity explosion. We begin with the most simple expected effect of learner collusion, that the relative magnitude of diversity is excessive with respect to the ensemble loss. The results are included in Figure 2 where we note a sharp increase in diversity for the joint objective. Specifically, we note that mean test set diversity remains relatively low for most values of  $\beta$  but quickly explodes as  $\beta \rightarrow 1$  to levels that are often significantly greater than the mean squared error of the ensemble predictions. This indicates that the base learners' variation around their own mean is much greater than the ensemble's variation around the label.
Figure 2: Diversity explosion. Test set diversity explodes relative to MSE across four datasets. We note two further empirical phenomena: (1) the non-linear relationship between  $\beta$  and diversity which we discuss further in Appendix D and (2) the minimal effect diversity has on the test MSE on the Facebook dataset which is discussed in Appendix F.
Debiased diversity. We now proceed to the second postulated effect of learner collusion. Although learners could perform learner collusion on a per-example basis, the most straightforward form of this behavior would be learners that bias their predictions without any dependence on the example at hand. To investigate the extent to which this trivial form of collusion does occur we apply a post hoc debiasing to each of the individual learners in the ensemble which affects the trade-off between  $\overline{\mathrm{ERR}}$  and DIV without affecting ERR. Specifically, we adjust learner  $j$ 's predictions such that
$$
\begin{array}{l} f _ {j} ^ {\text {d e b i a s e d}} \left(\mathbf {x} _ {i}\right) = f _ {j} \left(\mathbf {x} _ {i}\right) - b _ {j} \\ = f _ {j} (\mathbf {x} _ {i}) - \left(\frac {1}{N} \sum_ {i = 1} ^ {N} f _ {j} (\mathbf {x} _ {i}) - \frac {1}{N} \sum_ {i = 1} ^ {N} \bar {f} (\mathbf {x} _ {i})\right). \\ \end{array}
$$
In Appendix C.1 we prove that the debiasing term  $b_{j}$  is the optimal diversity minimizer that keeps the ensemble predictions unchanged. In Figure 3 we show that applying this technique significantly reduces the ensemble's apparent diversity to far more reasonable values. In general, we find that this typically removes around half of the ensemble's diversity at the joint training region  $(\beta = 1)$ . While example-dependent collusion could also emerge, these results already indicate that a significant proportion of the apparent diversity in jointly trained ensembles is due to an artificial biasing of outputs.
Figure 3: Bias removal. The percentage of diversity explained by the most simple form of learner collusion across each dataset. In the region where learner collusion appears to occur  $(\beta \rightarrow 1)$ , a large proportion of the diversity can be explained by simple additive bias in the predictions.
Learner codependence: The final effect we predicted to emerge due to learner collusion is an excessive codependence among the base learners. If individual learners are engaging in learner collusion, the removal of a subset of these learners at test time should be catastrophic to the performance of the ensemble. This is because a learner producing excessive errors needs to be counterbalanced by other learners performing similar errors in the opposite direction, otherwise the ensemble predictions will be influenced by the poorly performing base learners. To test this we consider dropping a fraction of the individual learners at test time and evaluate the resulting ensemble performance. This is similar to experiments performed in [30], although from a very different perspective. In Figure 4 we present the results of randomly applying various levels of test time base learner dropout 100 times for each model and report the average resulting increase in test set error. These results clearly show that such a procedure is detrimental to ensemble performance at precisely the levels of  $\beta$  at which we have hypothesized learner collusion to occur, while having only minimal impact elsewhere.
Figure 4: Learner codependence. Applying various levels of base learner dropout at test time and observing the relative increase in MSE. As  $\beta \to 1$ , dropping a subset of learners causes a greater increase in test error suggesting the occurrence of learner collusion.
Summary: Extensive empirical evidence confirms that training with the joint objective results in learner collusion.
6.2 Learner Collusion Generalizes Poorly
Given that we have shown that learner collusion emerges from joint training, we now proceed to examine the second claim. That is, although learner collusion does not prevent the training loss from being minimized, the solution it obtains will generalize more poorly than an independently trained ensemble with non-trivial diversity. This claim is more straightforward to investigate and only requires a comparison of the difference between the training and testing loss curves over the course of training. The difference between the two, which we refer to as the generalization gap, captures the drop in performance when a model transitions from training to testing data.
For this experiment, we return to the more standard machine learning benchmark of classification on the CIFAR tasks as introduced in Section 3. On each dataset we train a joint model (ERR) and an independent model (ERR) and report the generalization gap of ensemble loss for both methods over the course of training. We repeat this six times and include a shaded region representing  $\pm 2$  standard errors. The results are included in Figure 5 where we consistently find that joint training displays a significantly larger generalization gap as training proceeds. This is strongly suggestive of a much greater tendency towards overfitting in the jointly trained model while the independently trained model's generalization gap plateaus at a much lower level. We provide
further analysis of the distinct training and testing curves in Appendix G.
Figure 5: Generalization gaps. Joint training results in a larger generalization gap  $(\mathcal{L}^{\mathrm{test}}(\bar{\mathbf{f}},\mathbf{y}) - \mathcal{L}^{\mathrm{train}}(\bar{\mathbf{f}},\mathbf{y}))$  than independent training, despite both methods reporting test performance using the joint objective.
Summary: Learner collusion caused by joint training (ERR) results in greater overfitting exhibited by a larger generalization gap.
6.3 Practical Implications
We have now described the phenomenon of learner collusion caused by the joint training of deep ensembles, empirically verified its existence, and demonstrated how it results in poorer generalization. We complete this analysis with an empirical investigation of how learner collusion manifests in the ensemble performance on standard machine learning tasks. We use the augmented objective,  $\mathcal{L}^{\beta}$  as described in Section 5, to examine the effects of smoothly increasing the level of joint training in the objective on CIFAR-10/CIFAR-100 with ResNet architectures. We also examine the validation performance of the ensemble and base learners throughout training on ImageNet. Results are provided in Figure 6. We also provide extensive additional results in Appendix G confirming the generality of this degeneracy. There we consider: larger ensembles, score-averaging, alternative models, additional datasets, and  $\overline{\mathrm{ERR}}$  as a metric. Complete experimental details are provided in Appendix B.
Level of diversity  $\beta$
Figure 6: Practical implications of learner collusion. Left: Test accuracy with increasing diversity in the objective on CIFAR-10/100 with ResNets. Right: Validation accuracy throughout training ResNet-18 base learners on ImageNet. Jointly trained ensembles  $(\beta = 1)$  perform significantly worse than independently trained ensembles  $(\beta = 0)$ .
Training iteration
Consistently across all experiments, we find that joint training  $(\beta = 1)$  results in the worst test set performance. We also note the poor performance among base learners in this regime, highlighting that inflating diversity comes at the cost of degenerate individual performance. Interestingly, across all of our experiments learner collusion typically emerges as  $\beta \rightarrow 1$  implying that partial joint training may be a feasible direction for future work. However, in general, independent training emerges as a primary choice of training strategy for deep ensembles.
In this work, we have provided a detailed account explaining the unexpected observation that jointly trained deep ensembles are generally outperformed by their independently trained counterparts despite joint training being the objective of interest at test time. We presented a comprehensive view of diversity in machine learning ensembles through which we analyzed the joint training objective. From this perspective, we uncovered a learner collusion effect in which non-diverse base learners collude to artificially inflate their apparent diversity. Empirically, we both isolated this phenomenon and demonstrated its catastrophic effect on model generalization. Finally, the practical consequences of this limitation were exposed for standard machine learning settings. Note that we also provide a visual summary of this work and its technical contributions in Appendix A. We hope that future work will investigate how to optimize deep ensembles to operate collaboratively whilst avoiding the degeneracies of learner collusion. In Appendix G.7 we include a negative result of our initial attempts in this direction. However, several promising avenues remain open (e.g. jointly optimizing worst-case bounds instead [47, 48]).
We would like to thank Alicia Curth, Fergus Imrie, and the anonymous reviewers for very insightful comments and discussions on earlier drafts of this paper. AJ gratefully acknowledges funding from the Cystic Fibrosis Trust. TL is funded by AstraZeneca. JC is funded by Aviva. This work was supported by Azure sponsorship credits granted by Microsoft's AI for Good Research Lab.
[1] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[2] Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757, 2019.
[3] Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
[4] Taiga Abe, E Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? arXiv preprint arXiv:2202.06985, 2022.
[5] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[6] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-23998. PMLR, 2022.
[7] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34: 18932-18943, 2021.
[8] Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
[9] Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
[10] Yoav Freund, Robert E Schapire, et al. Experiments with a new boosting algorithm. In icml, volume 96, pages 148-156. Citeseer, 1996.
[11] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994.
[12] Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't Believe It's Not Better!" NeurIPS 2020 workshop, 2020.
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[14] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[15] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27, 2014.
[16] Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[52] Dongjin Cho, Cheolhee Yoo, Jungho Im, and Dong-Hyun Cha. Comparative assessment of various machine learning-based bias correction methods for numerical weather prediction model forecasts of extreme air temperatures in urban areas. Earth and Space Science, 7(4): e2019EA000740, 2020.
[53] Sérgio Moro, Paulo Rita, and Bernardo Vala. Predicting social media performance metrics and evaluation of the impact on brand building: A data mining approach. Journal of Business Research, 69(9):3341-3351, 2016.
[54] Samuel George Waugh. Extending and benchmarking Cascade-Correlation: extensions to the Cascade-Correlation architecture and benchmarking of feed-forward supervised artificial neural networks. PhD thesis, University of Tasmania, 1995.
[55] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[56] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[57] Milton Abramowitz, Irene A Stegun, and Robert H Romer. Handbook of mathematical functions with formulas, graphs, and mathematical tables, 1988.
[58] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
A Paper Summary
Figure 7: A condensed description of this work and its main contributions.