SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Empirically Evaluating Joint Training (Section 3). This section compares training using cross-entropy loss as the objective applied to each learner's predictions independently (independent training) and to the ensemble's aggregated predictions (joint training). As described in the main text, we compare the test set performance of these two training methods using various popular architectures on ImageNet [34] evaluating the joint objective on the validation set. Specifically, we use the architectures cited in Table 1 with ensemble size determined by computational limitations. We use implementations from [49] except for MobileViT where the implementation is from [40]. Base learners' are aggregated using their probability outputs (i.e. probability averaging in Section 4.2). We use the standard ImageNet training-validation splits. For all models, we adjusted the following hyperparameters to optimize performance: learning rate, learning rate scheduler (and corresponding decay factor, gamma and frequency), momentum, and weight decay. Batch size was typically set to 128 but was divided by a multiple of 2 for larger base learners. We train models with early stopping and patience of 15 test set evaluations which occur after every 2000 batches. We apply stochastic
gradient descent as our optimizer. We apply standard preprocessing and augmentations consisting of input standardization, random crops, and random horizontal flips.
Diagnosing Learner Collusion (Section 6.1). These experiments investigate the performance of ensembles of multi-layer perceptrons on four regression datasets from the UCI machine learning repository [50]. These are Bioconcentration [51], Weather [52], Facebook [53], and Abalone [54]. Standard preprocessing is applied to each dataset including standardization of the features and targets, one hot encoding of categorical features, and log transformation of skewed features. Due to the smaller size of these tabular datasets, these experiments are less computationally expensive and, therefore, we perform hyperparameter optimization using Bayesian optimization for each run. For each run, we apply 20 iterations of Bayesian optimization searching over the following parameters. Learning rate  $\in \{0.01, 0.005, 0.001\}$ , batch size  $\in \{16, 32, 64\}$ , weight decay  $\in \{0, 0.01, 0.001\}$ , number of base learners  $\in \{5, 10, 15, 20\}$ , hidden layer neurons  $\in \{10, 20, 30\}$ , and number of hidden layers  $\in \{1, 2, 3\}$ . Additionally, the ReLU activation function is used with mean squared error loss. At each run,  $20\%$  of the data is randomly used for testing with the remaining data being further split into training and validation with an  $80/20$  ratio. Each ensemble is trained with early stopping with patience of 5 epochs. The best-performing hyperparameters are selected based on validation set performance and then retrained on the train/validation data for all values of  $\beta \in \{0, 0.1, \dots, 0.9, 1\}$  and evaluated on the test set. This is repeated for 10 runs. The mean  $\pm$  a standard deviation is provided in our results.
In Figure 2 we report test set ERR and DIV as MSE and Diversity respectively. In Figure 3 we report the percentage decrease in test set DIV for each value of  $\beta$  after the debiasing process described in the main text. In Figure 4, for each model being evaluated on the test set we randomly drop a percentage of the base learners  $p \in \{0.2, 0.4, 0.6\}$  and report the mean increase in test set ERR (ensemble mean squared error).
Learner Collusion Generalizes Poorly (Section 6.2). This section uses a similar experimental setting as Section 6.3 on the CIFAR datasets, as described in detail in the next paragraph. The key difference is using Adam optimizer [55] without learning rate decay to evaluate performance smoothly over the course of training. We train for a fixed period of 100 training epochs. We note that this number of training epochs is only for our analysis as the best validation performance generally occurs significantly earlier. We also only use the standard training/testing split as a third split is not required. In Figure 5 we report the difference in ERR measured on the training set vs the test set over the course of training and refer to this metric as the generalization gap. This can be expressed as  $\mathcal{L}^{\mathrm{test}}(\bar{\mathbf{f}},\mathbf{y}) - \mathcal{L}^{\mathrm{train}}(\bar{\mathbf{f}},\mathbf{y})$  where the loss function is cross-entropy loss. Further results from this experiment are provided in Appendix G.
Practical Implications (Section 6.3). In Figure 6 (right) we use the same experimental setup as in Section 3. In the two left-side plots we train and evaluate on CIFAR-10 and CIFAR-100 [56]. We train ensembles of either ResNet-18 or ResNet-34 [35] as base learners with each ensemble consisting of five learners. Base learners are again aggregated using their probability outputs (i.e. probability averaging in Section 4.2). We use the standard training and testing splits with an additional 80/20 split on the training data to provide a validation set in order to ensure our test set performance is independent of implicit bias from the training process. We use a similar learning schedule as proposed in [35]. Specifically, we train for 80 epochs using stochastic gradient descent with momentum of 0.9. The learning rate is initialized at 0.1 and exponentially decays at a rate of 0.2 at epochs [30, 50, 70]. We apply weight decay of  $5e - 4$ . Validation performance is evaluated at each epoch with the best-performing model across all 80 epochs evaluated on the test set. A batch size of 128 was used. We repeat each experiment for a total of six runs and report the mean and two standard errors in our results. We apply standard preprocessing and augmentations consisting of input standardization, random crops, random horizontal flips, and random rotations. The key difference in the left figure is that instead of only training deep ensembles using independent training and joint training, we now also consider a hybrid objective using the augmented objective for Section 5. We train this objective for  $\beta \in \{0,0.2,0.4,0.6,0.8,1\}$  and report accuracy evaluated on the test set in the figure. In the right figure, we provide the validation set accuracy of both ImageNet models throughout training. We provide both the performance of the individual learners and the aggregate ensemble performance.
Additional experiments from this section are provided in Appendix G. Unless otherwise stated, results reported in the appendix are aggregated over three runs.
Compute. All experiments are run on Azure NCv3-series VMs powered by NVIDIA Tesla V100 GPUs with 16GB of GPU VRAM.
C.1 Optimal Debiasing
We hypothesize that the predictions of each learner in the ensemble is associated with a bias term  $b_{j}$  which cancel in aggregate but significantly affect our measure of diversity. This is the most simple conceivable form of learner collusion and in this section we prove that the debiasing term used in Section C.1 is optimal for this purpose.
We begin by noting that for these biasing terms to not affect the aggregated ensemble prediction we require that  $\sum_{j}(f_{j} - b_{j}) = \sum_{j}f_{j}$  and therefore we require that  $\sum_{j}b_{j} = 0$ . Then we wish to find the set of such bias terms that minimize the standard diversity, DIV, of the ensemble over the data. Combining these two requirements we arrive at the optimization objective in Equation (3) for an ensemble of size  $M$  with  $N$  examples.
$$
\min  _ {b _ {1}, \dots , b _ {M}} \frac {1}{N \cdot M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} \left(f _ {j} \left(\mathbf {x} _ {i}\right) - b _ {j} - \bar {f} \left(\mathbf {x} _ {i}\right)\right) ^ {2} \quad \text {s u b j e c t t o} \quad \sum_ {j = 1} ^ {M} b _ {j} = 0. \tag {3}
$$
By the method of Lagrange multipliers we obtain  $M + 1$  equations for the  $M$  values  $b_{j}$  and the Lagrange multiplier  $\lambda$
$$
\nabla_ {b _ {j}} \frac {1}{N \cdot M} \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {M} \left(f _ {j} \left(\mathbf {x} _ {i}\right) - b _ {j} - \bar {f} \left(\mathbf {x} _ {i}\right)\right) ^ {2} - \lambda \nabla_ {b _ {j}} \sum_ {j = 1} ^ {M} b _ {j} = 0 \quad \forall \quad j, \tag {4}
$$
$$
\sum_ {j = 1} ^ {M} b _ {j} = 0. \tag {5}
$$
Solving equation 4 for an arbitrary  $b_{j}$ , we obtain
$$
\lambda = - \frac {2}{N \cdot M} \sum_ {i = 1} ^ {N} (f _ {j} (\mathbf {x} _ {i}) - b _ {j} - \bar {f} (\mathbf {x} _ {i}))
$$
$$
\Longrightarrow b _ {j} = \frac {\lambda \cdot M}{2} + \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {j} \left(\mathbf {x} _ {i}\right) - \bar {f} \left(\mathbf {x} _ {i}\right)\right). \tag {6}
$$
And subbing into equation 5
$$
\sum_ {j = 1} ^ {M} \left(\frac {\lambda \cdot M}{2} + \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {j} \left(\mathbf {x} _ {i}\right) - \bar {f} \left(\mathbf {x} _ {i}\right)\right)\right) = 0
$$
$$
\frac {\lambda \cdot M ^ {2}}{2} + \frac {M}{N} \sum_ {i = 1} ^ {N} (\bar {f} (\mathbf {x} _ {i}) - \bar {f} (\mathbf {x} _ {i})) = 0
$$
$$
\lambda = 0.
$$
Finally, we can substitute  $\lambda = 0$  back into equation 6 to obtain the solution
$$
b _ {j} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(f _ {j} \left(\mathbf {x} _ {i}\right) - \bar {f} \left(\mathbf {x} _ {i}\right)\right) \quad \forall \quad j.
$$
C.2 General Diversity Expression for Multidimensional Outputs
Proof. We follow the same approach taken in [44] but extend their proof to account for a multidimensional output space. For each  $j \in [M]$ , we define the curve  $\gamma_j : [0,1] \to \mathcal{V}$  by  $\gamma_j(t) = \mathcal{L}[\overline{\mathbf{f}} + t \cdot (\mathbf{f}_j - \overline{\mathbf{f}}), \mathbf{y}]$  for all  $t \in [0,1]$ . We note that  $\mathcal{L}(\mathbf{f}_j, \mathbf{y}) = \gamma_j(1)$ . Since the loss  $\mathcal{L}$  is at least twice differentiable, we can perform a first-order Taylor expansion around  $t = 0$  with Lagrange remainder:
$$
\gamma_ {j} (1) = \mathcal {L} [ \bar {\mathbf {f}}, \mathbf {y} ] + \nabla_ {f} ^ {\intercal} \mathcal {L} [ \bar {\mathbf {f}}, \mathbf {y} ] \cdot (\mathbf {f} _ {j} - \bar {\mathbf {f}}) + \frac {1}{2} (\mathbf {f} _ {j} - \bar {\mathbf {f}}) ^ {\intercal} \cdot \mathcal {H} _ {f} ^ {\mathcal {L}} (\mathbf {f} _ {j} ^ {*}, \mathbf {y}) \cdot (\mathbf {f} _ {j} - \bar {\mathbf {f}}),
$$
with  $\mathbf{f}_j^*$  defined as in the theorem statement. We note that this expression for  $\gamma_j(1)$  is exact for any twice differentiable loss thanks to the Lagrange's remainder theorem (see e.g. p.878 of [57]). We deduce the following formula for the average loss over the learners:
$$
\begin{array}{l} \overline {{\operatorname {E R R}}} = \sum_ {j = 1} ^ {M} w _ {j} \mathcal {L} (\mathbf {f} _ {j}, \mathbf {y}) \\ = \sum_ {j = 1} ^ {M} w _ {j} \gamma_ {j} (1) \\ = \underbrace {\mathcal {L} [ \bar {\mathbf {f}} , \mathbf {y} ]} _ {\text {E R R}} + \nabla_ {f} ^ {\intercal} \mathcal {L} [ \bar {\mathbf {f}}, \mathbf {y} ] \cdot \sum_ {j = 1} ^ {M} w _ {j} \cdot (\mathbf {f} _ {j} - \bar {\mathbf {f}}) + \frac {1}{2} \sum_ {j = 1} ^ {M} (\mathbf {f} _ {j} - \bar {\mathbf {f}}) ^ {\intercal} \cdot \mathcal {H} _ {f} ^ {\mathcal {L}} (\mathbf {f} _ {j} ^ {*}, \mathbf {y}) \cdot (\mathbf {f} _ {j} - \bar {\mathbf {f}}), \\ \end{array}
$$
where the second term vanishes due to the identity  $\sum_{j=1}^{M} w_{j} \mathbf{f}_{j} = \bar{\mathbf{f}}$ . Since  $\mathrm{DIV} = \overline{\mathrm{ERR}} - \mathrm{ERR}$ , this proves the theorem.
C.3 Probability Averaged Diversity Term for Classification
$$
\begin{array}{l} \mathrm {D I V} = \sum_ {j = 1} ^ {M} w _ {j} \mathcal {L} _ {C E} (\mathbf {y}, \mathbf {p} _ {j}) - \mathcal {L} _ {C E} (\mathbf {y}, \bar {\mathbf {p}}) \\ = \sum_ {j = 1} ^ {M} w _ {j} D _ {K L} (\mathbf {y} | | \mathbf {p} _ {j}) + H (\mathbf {y}) - D _ {K L} (\mathbf {y} | | \bar {\mathbf {p}}) - H (\mathbf {y}) \quad \text {(w h e r e} H \text {i s e n t r o p y)} \\ = \sum_ {j = 1} ^ {M} w _ {j} \sum_ {k \in \mathcal {K}} y (k) \log \frac {y (k)}{p _ {j} (k)} - \sum_ {k \in \mathcal {K}} y (k) \log \frac {y (k)}{\bar {p} (k)} \\ = \sum_ {j = 1} ^ {M} w _ {j} \sum_ {k \in \mathcal {K}} y (k) \log \frac {\bar {p} (k)}{p _ {j} (k)} \\ = \sum_ {j = 1} ^ {M} w _ {j} \cdot y (k ^ {\star}) \log \frac {\bar {p} (k ^ {\star})}{p _ {j} (k ^ {\star})} \\ \end{array}
$$
Interpretation. This expression has a sensible interpretation as the diversity of a probability-averaged ensemble. Examining this expression of diversity:
$$
\mathrm {D I V} = \sum_ {j = 1} ^ {M} w _ {j} \cdot y (k ^ {\star}) \log \frac {\bar {p} (k ^ {\star})}{p _ {j} (k ^ {\star})},
$$
where  $k^{\star}$  is the true class, and we note that this expression's only non-zero contribution comes from the predictions of the ground-truth class. We now view  $p_j(k^{\star})$  as coming from a distribution of probabilities and assume that  $p_j(k^{\star})$  is distributed around  $\bar{p}(k^{\star})$  following a Beta distribution with its mean at  $\bar{p}(k^{\star})$ , i.e.  $p_j(k^{\star}) \sim \mathrm{Beta}(\alpha, \beta)$ , where  $\mathbb{E}[p_j(k^{\star})] = \bar{p}(k^{\star})$ . Consequently, we can examine the expected value of the term with respect to ensemble learners as follows:
$$
\begin{array}{l} \mathbb {E} _ {p _ {j} (k ^ {\star})} \left[ \log \frac {\bar {p} (k ^ {\star})}{p _ {j} (k ^ {\star})} \right] = \mathbb {E} _ {p _ {j} (k ^ {\star})} [ \log \bar {p} (k ^ {\star}) ] - \mathbb {E} _ {p _ {j} (k ^ {\star})} [ \log p _ {j} (k ^ {\star}) ] \\ \approx \mathbb {E} _ {p _ {j} (k ^ {\star})} [ \log \bar {p} (k ^ {\star}) ] - \log \mathbb {E} _ {p _ {j} (k ^ {\star})} [ p _ {j} (k ^ {\star}) ] + \frac {\mathbb {V} [ p _ {j} (k ^ {\star}) ]}{2 \mathbb {E} _ {p _ {j} (k ^ {\star})} [ p _ {j} (k ^ {\star}) ] ^ {2}} \\ = \log \bar {p} (k ^ {\star}) - \log \bar {p} (k ^ {\star}) + \frac {\mathbb {V} [ p _ {j} (k ^ {\star}) ]}{2 \bar {p} (k ^ {\star}) ^ {2}} \\ = \frac {\mathbb {V} \left[ p _ {j} \left(k ^ {\star}\right) \right]}{2 \bar {p} \left(k ^ {\star}\right) ^ {2}} \\ = \frac {\alpha \beta}{(\alpha + \beta) ^ {2} (\alpha + \beta + 1) 2 \bar {p} (k ^ {\star}) ^ {2}} \\ = \frac {\mu (1 - \mu)}{(1 + \phi) 2 \bar {p} (k ^ {\star}) ^ {2}} \\ = \frac {\bar {p} (k ^ {\star}) (1 - \bar {p} (k ^ {\star}))}{(1 + \phi) 2 \bar {p} (k ^ {\star}) ^ {2}} \\ \end{array}
$$
where the approximation is from taking a second order Taylor expansion around  $p_{j_0}(k^\star) = \mathbb{E}_{p_j(k^\star)}[p_j(k^\star)]$  to approximate  $\mathbb{E}_{p_j(k^\star)}[\log p_j(k^\star)]$ . The fourth line is due to a reparameterization of the Beta distribution in terms of mean  $\mu$  and precision  $\phi$  parameters with  $\alpha = \phi \cdot \mu$  and  $\beta = \phi (1 - \mu)$ . Using this reparameterization, we arrive at an expression that emits an intuitive interpretation—that for a fixed  $\bar{p}$ , the smaller the value of  $\phi$  (conversely, the larger the variance), the larger the expected value of diversity.
Furthermore, we might ask if this expression of diversity in Theorem 4.3 is connected to an information-theoretic quantity as in the score averaging case. While this expression does not take the form of any divergence term that the authors of this work are aware of, the following link can be made.
$$
\begin{array}{l} \sum_ {j = 1} ^ {M} w _ {j} \sum_ {k \in \mathcal {K}} y (k) \log \frac {\bar {p} (k)}{p _ {j} (k)} \\ = \sum_ {j = 1} ^ {M} w _ {j} \sum_ {k \in \mathcal {K}} y (k) \frac {\bar {p} (k)}{\bar {p} (k)} \log \frac {\bar {p} (k)}{p _ {j} (k)} \\ = \sum_ {j = 1} ^ {M} w _ {j} \mathbb {E} _ {\bar {p} (k)} \left[ \frac {y (k)}{\bar {p} (k)} \log \frac {\bar {p} (k)}{p _ {j} (k)} \right] \\ = \sum_ {j = 1} ^ {M} w _ {j} \mathbb {E} _ {\bar {p} (k)} \left[ \gamma (k) \log \frac {\bar {p} (k)}{p _ {j} (k)} \right] \quad \text {w h e r e} \gamma (k) = \frac {y (k)}{\bar {p} (k)}, \\ \rightarrow \sum_ {j = 1} ^ {M} w _ {j} D _ {K L} (\bar {\mathbf {p}} | | \mathbf {p} _ {j}) \quad \text {a s} \boldsymbol {\gamma} \rightarrow \mathbf {1}. \\ \end{array}
$$
Therefore, as the ensemble predictive distribution approaches the label distribution, the diversity term approaches the weighted average of the KL-divergence between the ensemble predictive distribution and each of the base learner predictive distributions, analogous to the equivalent expression of the score averaged ensemble.
C.4 Pathological loss for  $\beta > 1$
Proof. Case 1. We first consider the case where  $\mathcal{V} = \mathbb{R}^d$ . Without loss of generality, we consider an ensemble such that  $w_{1}, w_{2} > 0$ . We fix  $\mathbf{f}_j = 0 \in \mathcal{V}$  for all  $2 < j \leq M$ . We then consider the following sequence of learners:  $(\mathbf{f}_{1,t})_{t \in \mathbb{N}} = (\mathbf{y}_t)_{t \in \mathbb{N}}$  and  $(\mathbf{f}_{2,t})_{t \in \mathbb{N}} = (-w_1 w_2^{-1} \mathbf{y}_t)_{t \in \mathbb{N}}$ . We note that for all  $t \in \mathbb{N}$ , the ensemble prediction equals zero  $\overline{\mathbf{f}}_t = (w_1 \mathbf{y}_t - w_2 w_2^{-1} w_1 \mathbf{y}_t + 0) = 0$ . The ensemble loss is therefore constant and finite  $\mathrm{ERR}_t \to \mathcal{L}(0, \mathbf{y}) < +\infty$  as  $t \to \infty$  since  $(0, \mathbf{y}) \in \operatorname{int}(\mathcal{V}^2)$ . On the other hand, we have that the average loss diverges as  $\overline{\mathrm{ERR}}_t \geq w_1 \mathcal{L}(\mathbf{y}_t, \mathbf{y}) \to +\infty$  as  $t \to \infty$ . We deduce that the augmented loss is not bounded from below when  $\beta > 1$  as  $\beta \cdot \mathrm{ERR}_t + (1 - \beta) \cdot \overline{\mathrm{ERR}}_t \to -\infty$  for  $t \to \infty$ .
Case 2. We now consider the case where  $\mathcal{Y} \subset \mathbb{R}^d$  is a compact and convex subset of  $\mathbb{R}^d$ . Without loss of generality, we consider an ensemble such that  $0 < w_1 < 1$ . Since  $\mathcal{Y}$  is compact, it is closed and admits a topological boundary that we denote  $\partial \mathcal{Y}$ . Since the loss is finite on the interior  $\operatorname{int}(\mathcal{Y}^2)$ , our assumption implies that the loss admits a singular point  $(\mathbf{y}_{\infty}, \mathbf{y})$  with  $\mathbf{y}_{\infty} \in \partial \mathcal{Y}$ . We fix  $\mathbf{f}_j = \mathbf{y}$  for all  $1 < j \leq M$ . For the first learner, we consider a sequence  $(\mathbf{f}_{1,t})_{t \in \mathbb{N}^+}$  with  $\mathbf{f}_{1,t} = t^{-1} \cdot \mathbf{y} + (1 - t^{-1}) \cdot \mathbf{y}_{\infty}$ . We trivially have that  $\mathbf{f}_{1,t} \to \mathbf{y}_{\infty}$  for  $t \to \infty$ , so that  $\mathcal{L}(\mathbf{f}_{1,t}, \mathbf{y}) \to \infty$ . Hence, once again, we have that the average loss diverges as  $\overline{\mathrm{ERR}}_t \geq w_1 \mathcal{L}(\mathbf{f}_{1,t}, \mathbf{y}) \to +\infty$  as  $t \to \infty$ . It remains to show that the ensemble loss ERR is finite. We note that the ensemble prediction also defines a sequence  $(\bar{\mathbf{f}}_t)_{t \in \mathbb{N}^+}$  such that for all  $t \in \mathbb{N}^+$
$$
\begin{array}{l} \bar {\mathbf {f}} _ {t} = w _ {1} \cdot [ t ^ {- 1} \cdot \mathbf {y} + (1 - t ^ {- 1}) \cdot \mathbf {y} _ {\infty} ] + \mathbf {y} \cdot \sum_ {j = 2} ^ {M} w _ {j} \\ = [ 1 + w _ {1} \cdot (t ^ {- 1} - 1) ] \cdot \mathbf {y} + [ w _ {1} \cdot (1 - t ^ {- 1}) ] \cdot \mathbf {y} _ {\infty}. \\ \end{array}
$$
We deduce that  $\bar{\mathbf{f}}_t\in \mathcal{V}$  as it is a convex combination of  $\mathbf{y}$  and  $\mathbf{y}_{\infty}$ . Furthermore, we trivially have that  $\bar{\mathbf{f}}_t\rightarrow \bar{\mathbf{f}}_{\infty} = (1 - w_1)\cdot \mathbf{y} + w_1\cdot \mathbf{y}_{\infty}$  as  $t\to \infty$ . We shall now demonstrate that  $\bar{\mathbf{f}}_{\infty}\in \mathrm{int}(\mathcal{V})$  and conclude. We note that, since  $\mathbf{y}\in \mathrm{int}(\mathcal{V})$ , there exists an open ball  $B[\mathbf{y},r]$  of radius  $r\in \mathbb{R}^{+}$  centered at  $\mathbf{y}$  such that  $B[\mathbf{y},r]\subseteq \mathcal{V}$ . Let us now consider the open ball  $B[\bar{\mathbf{f}}_{\infty},(1 - w_1)\cdot r]$ . We note that any  $\mathbf{y}'\in B[\bar{\mathbf{f}}_{\infty},(1 - w_1)\cdot r]$  is of the form
$$
\begin{array}{l} \mathbf {y} ^ {\prime} = \overline {{\mathbf {f}}} _ {\infty} + (1 - w _ {1}) \cdot \mathbf {v} \\ = (1 - w _ {1}) \cdot (\mathbf {y} + \mathbf {v}) + w _ {1} \cdot \mathbf {y} _ {\infty}, \\ \end{array}
$$
where  $\mathbf{v} \in B[0, r]$ . Clearly,  $(\mathbf{y} + \mathbf{v}) \in B[\mathbf{y}, r] \subseteq \mathcal{V}$ , hence  $\mathbf{y}' \in \mathcal{V}$  as it is a convex combination of two elements of  $\mathcal{V}$ . We deduce that  $B[\mathbf{f}_{\infty}, (1 - w_1) \cdot r] \subseteq \mathcal{V}$ , which implies that  $\mathbf{f}_{\infty} \in \operatorname{int}(\mathcal{V})$ . We conclude that  $\mathrm{ERR}_t \to \mathcal{L}(\overline{\mathbf{f}}_{\infty}, y)$  by continuity of the loss on  $\operatorname{int}(\mathcal{V}^2)$ . We also know that  $\mathcal{L}(\overline{\mathbf{f}}_{\infty}, \mathbf{y})$  is finite as  $(\overline{\mathbf{f}}_{\infty}, \mathbf{y}) \in \operatorname{int}(\mathcal{V}^2)$ . We deduce that the augmented loss is not bounded from below when  $\beta > 1$  as  $\beta \cdot \mathrm{ERR}_t + (1 - \beta) \cdot \overline{\mathrm{ERR}_t} \to -\infty$  for  $t \to \infty$ .
D Gradient Adjusted Targets
Throughout this work we have discussed the two extremes of training neural network ensembles. Independent training, where each ensemble member is trained independently and joint training where the predictions of the ensemble as a whole is optimized. We have also shown that interpolating between these two extremes via the parameter  $\beta$  can result in an effective hybridization. In this section, we provide an alternative interpretation of these intermediate values of  $\beta$  as an optimization objective. In particular, we show that the hybrid objective can in fact be interpreted as an independently trained ensemble in which the targets  $y$  are adjusted by the gradient of the ensemble.
Specifically, in the case of mean squared error we can adjust the true target  $y$  by a step in the opposite direction to the gradient. Then each individual learner  $f_{i}$  will have its target adjusted to compensate for the errors of the full ensemble. The size of this step is determined by the parameter  $\alpha$ . This results in the gradient adjusted target (GAT) objective in Equation (7). Note that throughout this section we consider scalar regression but, of course, these results would hold for multiple outputs by applying the same reasoning to each output.
$$
\mathcal {L} ^ {\mathrm {G A T}} \left(f _ {i}, \tilde {y}\right) = \frac {1}{M} \sum_ {j = 1} ^ {M} \left(f _ {i} - \tilde {y}\right) ^ {2}, \tag {7}
$$
where  $\tilde{y} = y - \alpha \cdot g$
and  $g = \frac{\partial}{\partial\bar{f}}\left(\frac{1}{2} (\bar{f} -y)^2\right).$
Conveniently, it can be shown that applying individual training with this GAT objective is equivalent to using the hybrid objective described in equation 2 where the step size  $\alpha$  has a similar effect to the interpolation parameter  $\beta$ .
Theorem D.1 (Residual adjusted mean squared error). The ensemble gradient adjusted mean squared error objective
$$
\frac {1}{M} \sum_ {i = 1} ^ {M} \left(f _ {i} - \left(y - \frac {\alpha}{M} \sum_ {j = 1} ^ {M} \left(f _ {j} - y\right)\right)\right) ^ {2}, \tag {8}
$$
is proportional to  $\overline{ERR} -\left(1 - \frac{1}{(\alpha + 1)^2}\right)\cdot DIV.$
$$
\begin{array}{l} \frac {1}{M} \sum_ {i = 1} ^ {M} \left(f _ {i} - \left(y - \frac {\alpha}{M} \sum_ {j = 1} ^ {M} \left(f _ {j} - y\right)\right)\right) ^ {2} \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(f _ {i} - y + \alpha \cdot \bar {f} - \alpha \cdot y\right) ^ {2} \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(\left(f _ {i} - \bar {f}\right) + (1 + \alpha) (\bar {f} - y)\right) ^ {2} \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left[ \left(f _ {i} - \bar {f}\right) ^ {2} + (1 + \alpha) ^ {2} (\bar {f} - y) ^ {2} + 2 \cdot (1 + \alpha) \left(f _ {i} - \bar {f}\right) (\bar {f} - y) \right] \\ = \quad \left[ \begin{array}{c} \text {D I V} + (1 + \alpha) ^ {2} \underbrace {\text {E R R}} _ {= \overline {{\text {E R R}}} - \text {D I V}} + 0 \end{array} \right] \\ \propto \overline {{\mathrm {E R R}}} - \left(1 - \frac {1}{(1 + \alpha) ^ {2}}\right) \cdot \mathrm {D I V} \\ \end{array}
$$