SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When we take no gradient step at  $\alpha = 0$ , this is equivalent to  $\beta = 0$  or independent training. As we take a larger gradient step we approach joint training noting that as  $\alpha \to \infty$  we find  $\beta \to 1$ . Should we choose to take a step of exactly the gradient itself (i.e.  $\alpha = 1$ ), it is equivalent to hybrid training with  $\beta = 0.75$ .
Alternatively, we might wish to only consider the  $M - 1$  other members of the ensemble in our gradient step  $g$ . In this case we might consider  $\bar{f}^{-i} = \frac{1}{M - 1}\sum_{j\neq i}^{M - 1}f_j$  rather than  $\bar{f}$ . Again, we find that this can be expressed in terms of  $\overline{\mathrm{ERR}}$  and DIV. We begin with a prerequisite in Lemma D.2 and then proceed in Theorem D.3 to show that there is also an exact equivalence in the  $M - 1$  case to augmented objective  $\mathcal{L}^{\beta}$  described in Section 5.
Lemma D.2.
$$
\sum_ {j} ^ {M} \left(\bar {f} - f _ {j}\right) \left(y - f _ {j}\right) = M \cdot \text {D I V}. \tag {9}
$$
$$
\begin{array}{l} \sum_ {j} ^ {M} (\bar {f} - f _ {j}) (y - f _ {j}) \\ = - \sum_ {j} ^ {M} f _ {j} (\bar {f} - f _ {j}) \\ = - M \cdot \bar {f} ^ {2} + \sum_ {j} ^ {M} f _ {j} ^ {2} \\ = - M \cdot \bar {f} ^ {2} + \sum_ {j} ^ {M} (f _ {j} - \bar {f}) ^ {2} + M \cdot \bar {f} ^ {2} \\ = \sum_ {j} ^ {M} \left(f _ {j} - \bar {f}\right) ^ {2} \\ = M \cdot \operatorname {D I V}. \\ \end{array}
$$
Theorem D.3 ( $M - 1$  residual adjusted mean squared error). The  $M - 1$  ensemble gradient adjusted mean squared error objective
$$
\frac {1}{M} \sum_ {i = 1} ^ {M} \left(f _ {i} - \left(y - \frac {\alpha}{M - 1} \sum_ {j \neq i} ^ {M} \left(f _ {j} - y\right)\right)\right) ^ {2}, \tag {10}
$$
is equivalent to  $\overline{ERR} - \gamma \cdot DIV$  up to a scalar constant where  $\gamma = \frac{\alpha}{(\alpha + 1)^2} \cdot \frac{M((\alpha + 2)M - 2(\alpha + 1))}{(M - 1)^2}$ .
$$
\begin{array}{l} \frac {1}{M} \sum_ {i = 1} ^ {M} \left(f _ {i} - \left(y - \frac {\alpha}{M - 1} \sum_ {j \neq i} ^ {M} \left(f _ {j} - y\right)\right)\right) ^ {2} \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(f _ {i} - y + \frac {\alpha}{M - 1} \sum_ {j \neq i} ^ {M} f _ {j} - \alpha \cdot y\right) ^ {2} \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(\frac {\alpha \cdot M}{M - 1} \bar {f} + \frac {M - 1 - \alpha}{M - 1} f _ {i} - (1 + \alpha) \cdot y\right) ^ {2} \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(\frac {\alpha \cdot M}{M - 1} (\bar {f} - f _ {i}) + (1 + \alpha) f _ {i} - (1 + \alpha) \cdot y\right) ^ {2} \quad (\text {u s i n g :} 1 + \alpha - \frac {\alpha \cdot M}{M - 1} = \frac {M - 1 - \alpha}{M - 1}) \\ = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(\frac {\alpha \cdot M}{M - 1} (\bar {f} - f _ {i}) - (1 + \alpha) \cdot (y - f _ {i})\right) ^ {2} \\ = \left(\frac {\alpha \cdot M}{M - 1}\right) ^ {2} \operatorname {D I V} + (1 + \alpha) ^ {2} \overline {{\operatorname {E R R}}} - 2 \cdot \frac {\alpha}{M - 1} (1 + \alpha) \sum_ {i = 1} ^ {M} (\bar {f} - f _ {i}) (y - f _ {i}) \\ = \left(\frac {\alpha \cdot M}{M - 1}\right) ^ {2} \mathrm {D I V} + (1 + \alpha) ^ {2} \overline {{\mathrm {E R R}}} - 2 \cdot \frac {\alpha}{M - 1} (1 + \alpha) \cdot M \cdot \mathrm {D I V} \quad (\text {u s i n g L e m m a D . 2}) \\ = (1 + \alpha) ^ {2} \cdot \overline {{\mathrm {E R R}}} - \frac {\alpha M ((\alpha + 2) M - 2 (\alpha + 1))}{(M - 1) ^ {2}} \cdot \mathrm {D I V}. \\ \end{array}
$$
While the scalar  $\gamma$  might initially appear unwieldy, at closer inspection it performs a similar function to the full ensemble gradient equivalent in Proposition D.1. The key difference being that this term also accounts for the ensemble size  $M$  and ensures that smaller ensembles have a higher weighting on diversity. This relationship is visually depicted in Figure 8 where we again observe a sensible connection to hybrid training at various values of  $\beta$ .
Figure 8: M - 1 gradient adjusted targets.  $\gamma$  (equivalent to  $\beta$  in the augmented objective  $\mathcal{L}^{\beta}$ ) as a function of ensemble size  $M$  and gradient step size  $\alpha$ . Again we observe a sensible relationship between  $\alpha$  and  $\gamma$ .
Investigating the relationship of  $\beta$  and diversity via GAT. In the regression experiments in Figure 2 one might intuitively have expected diversity to increase linearly with  $\beta$ . Instead, we observe a sharp increase in diversity as  $\beta \rightarrow 1$ . The GAT perspective described in this section may provide some insight into why this relationship takes this form. Recall that the  $\beta$  term from the augmented objective in Equation (2) is related to the  $\alpha$  term the GAT objective in Theorem D.1 according to  $\beta = (1 - \frac{1}{(1 + \alpha)^2})$ . We include a visualization of this relationship in Figure 9. This illustrates that the gradient step  $\alpha$  grows exponentially as  $\beta \rightarrow 1$  resulting in each individual learner having their targets biased excessively (given that the targets are standardized in these experiments) to account for the ensemble errors. This exponential increase in the bias in the targets would appear to be related to the simultaneous increase in diversity at the same point.
Figure 9: Understanding the non-linear increase in diversity with  $\beta$ . Increasing  $\beta$  in our augmented objective is exactly equivalent to individual training with the targets adjusted to account for the errors of the ensemble by a step size of  $\alpha$ . The relationship between these two terms is given by  $\beta = (1 - \frac{1}{(1 + \alpha)^2})$  which we visualize in this plot. Thus the gradient step  $\alpha$  grows exponentially as  $\beta \to 1$  resulting in each individual learner having excessive biases dominate their targets which coincides with an exponential increase in diversity.
E Gradient Analysis of Classification Ensemble Averaging Methods
On the claims of [31]. We begin by taking a closer look at the claims of [31] where, using their notation, they presented the following equation for an equally weighted ensemble using either output averaging or probability averaging.
$$
\frac {\partial \ell}{\partial \mathbf {x} _ {\mathbf {i}}} = \frac {\partial \ell}{\partial \mu} \frac {\partial \mu}{\partial \mathbf {x} _ {\mathbf {i}}} = \frac {\partial \ell}{\partial \mu} \frac {1}{N},
$$
where  $\mu (\mathbf{x_1},\dots ,\mathbf{x_N}) = \frac{1}{N}\sum_{i = 1}^{N}\mathbf{x_i}$  represents the averaging of a set of  $N$  model outputs and  $\ell$  is some loss function.
The authors claimed that, because this expression does not depend on  $i$ , the "gradients backpropagated into all ensemble members are identical". On this basis, they argued that this "lack of gradient diversity" combined with numerical stability issues explains the poor performance of joint training. We wish to make two points on this claim. (1) Contrary to what is claimed, this constant gradient effect is only true in the case of score averaging and is not true for probability averaging which also performs poorly under joint training. (2) Even in the case of score averaging, although all ensemble members receive the same gradient signal, that signal is a function of the ensemble performance under joint training and the individual performance under independent training. Therefore the joint training optimal solution would still maximize ensemble performance if we could obtain the optimal solution.
We begin with the first point (1). If each  $\mathbf{x_i}$  represents the scores of each ensemble member and we presume that the softmax activation  $\phi$  is baked into  $\ell$ , then this is the score-averaged classification setting, and the author's expression holds. However, if we wish to consider the probability averaged setting, then the averaging function should in fact be  $\mu (\mathbf{x_1},\dots ,\mathbf{x_N}) = \frac{1}{N}\sum_{i = 1}^{N}\phi (\mathbf{x_i})$  and the derivative takes the following form.
$$
\frac {\partial \ell}{\partial \mathbf {x _ {i}}} = \frac {\partial \ell}{\partial \mu} \frac {\partial \mu}{\partial \phi (\mathbf {x _ {i}})} \frac {\partial \phi (\mathbf {x _ {i}})}{\partial \mathbf {x _ {i}}} = \frac {\partial \ell}{\partial \mu} \frac {1}{N} \frac {\partial \phi (\mathbf {x _ {i}})}{\partial \mathbf {x _ {i}}}.
$$
It is clear that the final term in this expression is indeed a function of  $\mathbf{x_i}$  and therefore each learner can receive unique gradients in the case of probability averaging.
With respect to the second point (2), we can simply note that the loss term  $\ell$  is a function of just  $\mathbf{x_i}$  in independent training and all  $\mathbf{x_1},\dots ,\mathbf{x_N}$  in joint training. Concretely, we note that the gradient with respect to learner  $i$  should be expressed as  $\frac{\partial\ell(\mathbf{x_i})}{\partial\mathbf{x_i}}$  for independent training and as  $\frac{\partial\ell(\mu(\mathbf{x_1},\cdots,\mathbf{x_N}))}{\partial\mathbf{x_i}}$  for joint training. This should clarify that although the learners could potentially have constant gradients in the case of joint training, they are still optimizing the correct objective and, therefore, the constant nature of these gradients would not in itself explain why the joint training objective results in poorer ensemble performance. We argue that a better explanation lies in the observation that the additional diversity term in the joint objective surprisingly results in poorer solutions is the observation that these multivariate entangled objectives result in an optimization problem that is significantly more challenging to solve in practice. In other words, individual training performs better as it results in a better solution to a slightly sub-optimal objective.
A gradient analysis of cross-entropy loss. Here we extend the previous gradient analysis by comparing the exact gradients obtained for independent training, probability-averaged joint training and score-averaged joint training in the case of cross-entropy loss. We use the following notation.  $f_{j,k}$  is the pre-activation output, or score, for learner  $j$  of class  $k$ .  $\phi$  is the softmax function with  $\phi(f_j)_k$  referencing the the  $k$ 'th element of the softmax function applied to the vector  $f_j$ .  $\mathcal{L}$  is the loss function.
We summarize the objectives for the three cases in Table 2. The columns correspond to the training method (independent vs joint) paired with the output averaging method (probability vs score). The first row corresponds to a general loss function, while the others are the specific cases of cross-entropy loss. The third row is simply re-expressing the second row by considering the true class  $k^{\star}$  such that  $y_{k^{\star}} = 1$ .
Table 2: Ensemble loss functions. A summary of primary loss functions used throughout this work. The first row corresponds to the expressions for a general loss function, while the following rows contain equivalent expressions for cross-entropy loss. Note that the final row is just expressing the previous row with the correct class  $k^{\star}$  replacing the sum.
Independent training	Joint training
Output aggregation	Both	Probability averaging	Score averaging
General loss	Lind = 1/M ∑j L(φ(fj),y)	Lpa = L(1/M ∑j φ(fj),y)	Lsa = L(φ(1/M ∑j f_j),y)
Cross-entropy loss	-1/M ∑k ∑j yk log (φ(fj)k)	-∑yk log (1/M ∑j φ(fj)k)	-∑yk log (φ(1/M ∑j f_j)k)
Cross-entropy loss (k=k*)	-1/M ∑j log (φ(fj)k*)	- log (1/M ∑j φ(fj)k*)	- log (φ(1/M ∑j f_j)k*)
Now we evaluate the gradient of each training method and output averaging combination with respect to an arbitrary learner  $i$  for an arbitrary class  $k$ . We begin with the case of independent training.
$$
\begin{array}{l} \frac {\partial \mathcal {L} ^ {\mathrm {i n d}}}{\partial f _ {i , k}} = - \frac {1}{M} \frac {\partial}{\partial f _ {i , k}} \sum_ {j} \log \left(\phi (f _ {j}) _ {k ^ {\star}}\right) = - \frac {1}{M} \frac {\frac {\partial}{\partial f _ {i , k}} \phi (f _ {i}) _ {k ^ {\star}}}{\phi (f _ {i}) _ {k ^ {\star}}} \\ = \left\{ \begin{array}{l l} - \frac {1}{M} \cdot (1 - \phi (f _ {i}) _ {k ^ {*}}) & \text {i f} k = k ^ {*}, \\ \frac {1}{M} \cdot \phi (f _ {i}) _ {k} & \text {i f} k \neq k ^ {*}. \end{array} \right. \\ \end{array}
$$
Next we examine the equivalent gradients of the joint training objective with probability averaged predictions.
$$
\begin{array}{l} \frac {\partial \mathcal {L} ^ {\mathrm {p a}}}{\partial f _ {i , k}} = - \frac {\partial}{\partial f _ {i , k}} \log \left(\frac {1}{M} \sum_ {j} \phi (f _ {j}) _ {k ^ {\star}}\right) = - \frac {\frac {\partial}{\partial f _ {i , k}} \phi (f _ {i}) _ {k ^ {\star}}}{\sum_ {j} \phi (f _ {j}) _ {k ^ {\star}}} \\ = \left\{ \begin{array}{l l} - \frac {\phi (f _ {i}) _ {k ^ {\star}} (1 - \phi (f _ {i}) _ {k ^ {\star}})}{\sum_ {j} \phi (f _ {j}) _ {k ^ {\star}}} & \text {i f} k = k ^ {*}, \\ \frac {\phi (f _ {i}) _ {k} \phi (f _ {i}) _ {k ^ {\star}}}{\sum_ {j} \phi (f _ {j}) _ {k ^ {\star}}} & \text {i f} k \neq k ^ {*}. \end{array} \right. \\ \end{array}
$$
Finally, we calculate the same gradients for the case of score averaging.
$$
\begin{array}{l} \frac {\partial \mathcal {L} ^ {\mathrm {s a}}}{\partial f _ {i , k}} = - \frac {\partial}{\partial f _ {i , k}} \log \left(\phi \left(\frac {1}{M} \sum_ {j} f _ {j}\right) _ {k ^ {\star}}\right) = - \frac {\frac {\partial}{\partial f _ {i , k}} \phi \left(\frac {1}{M} \sum_ {j} f _ {j}\right) _ {k ^ {\star}}}{\phi \left(\frac {1}{M} \sum_ {j} f _ {j}\right) _ {k ^ {\star}}} \\ = \left\{ \begin{array}{l l} - \frac {1}{M} \cdot \Big (1 - \phi (\frac {1}{M} \sum_ {j} f _ {j}) _ {k ^ {\star}} \Big) & \text {i f} k = k ^ {*}, \\ \frac {1}{M} \cdot \phi (\frac {1}{M} \sum_ {j} f _ {j}) _ {k} & \text {i f} k \neq k ^ {*}. \end{array} \right. \\ \end{array}
$$
From these gradients, we can draw the following conclusions for a given ensemble member  $f_{i}$ :
- Independent training - Each member receives a variable gradient that is a function of their own predictions but not the predictions of the rest of the ensemble. The gradient for learner  $f_{i}$  takes the form  $g(f_{i})$ .
- Joint training with probability averaging - Each member receives a variable gradient that is a function of their own predictions and the predictions of the other ensemble members. The gradient for learner  $f_{i}$  takes the form  $g(i, f_{1}, \dots, f_{M})$ .
- Joint training with score averaging - Each member receives a constant gradient that is a function of their own predictions and the predictions of the other ensemble members. The gradient for learner  $f_{i}$  takes the form  $g(\bar{f})$ .
F Learner Collusion in Regression
In this section, we provide an illustrative example of the learner collusion effect in the case of regression with mean squared error. We begin by recalling the relevant decomposition from Equation (1) for an equally weighted ensemble.
$$
\operatorname {E R R} = \overline {{\operatorname {E R R}}} - \operatorname {D I V},
$$
$$
(\bar {f} - y) ^ {2} = \frac {1}{M} \sum_ {j} (f _ {j} - y) ^ {2} - \frac {1}{M} \sum_ {j} (f _ {j} - \bar {f}) ^ {2}. \tag {11}
$$
Without loss of generality, we consider each base learner to predict some ensemble prediction  $f \in \mathbb{R}$  and a learner-dependent scalar adjustment  $c_{j} \in \mathbb{R}$  such that  $f_{j} = f + c_{j}$ . We also write  $\frac{1}{M}\sum_{j}c_{j} = \bar{c}$ . It is straightforward to check that Equation (11) can then be expressed as
$$
\begin{array}{l} \operatorname {E R R} = \frac {1}{M} \sum_ {j} (f - y + c _ {j}) ^ {2} - \frac {1}{M} \sum_ {j} (c _ {j} - \bar {c}) ^ {2}, \\ = \underbrace {(f - y) ^ {2} + \frac {1}{M} \sum_ {j} c _ {j} ^ {2} + \frac {1}{M} \sum_ {j} c _ {j} (f - y)} _ {\overline {{\text {E R R}}}} - \underbrace {\frac {1}{M} \sum_ {j} (c _ {j} - \bar {c}) ^ {2}} _ {\text {D I V}}. \\ \end{array}
$$
Setting aside  $f$ , we focus on the optimization of the  $c_{j}$ 's. We note that in the setting of independent training where we only minimize  $\overline{\mathrm{ERR}}$ , the unique minimum is achieved when  $c_{j} = 0 \forall j$ . However, when we add the additional DIV term, this changes. When optimizing the joint objective, ERR, any solution satisfying  $\bar{c} = y - f$  will minimize this loss. This is exactly learner collusion. We
note that as we optimize on training data, loss typically approaches zero, and  $\bar{c} \to 0$ . Furthermore, if we were to require that the two terms  $\overline{\mathrm{ERR}}$  and DIV should be relatively, approximately equal, we note that this would only occur as the  $c_{j}$  terms become large. Specifically,  $\overline{\mathrm{ERR}} \approx \mathrm{DIV}$  as  $|c_{j}| \to \infty$ . While this analysis illustrates how artificial diversity may emerge in the regression setting, we note that this is still not guaranteed to result in poor test performance in all cases. One such example is the Facebook dataset in Figure 2 where the test mean squared error is largely unaffected by this excessive diversity. In this case, it appears that almost the entire predictive is contained within a single feature resulting in no benefit from true diversity and, therefore, no substantial impact due to learner collusion
Finally, we highlight that during independent training, the learners are not aware of each other's predictions and, therefore, the premise that there would be a single shared ensemble prediction can not hold. In this case, all diversity will be due to external factors such as random initialization and improved performance over a single model can be explained by standard ensemble theory [e.g. 1, 2]. In the case of joint training, the premise that the ensemble produces a single prediction disguised as multiple predictions (i.e.  $f_{j} = f + c_{j}$  where  $\frac{1}{M}\sum_{j}c_{j} = 0$ ) is plausible and even likely. However, if this is the case, then clearly there is no genuine diversity among the learners implying that overfitting and poor generalization may become more likely.
G.1 Generalization Gap Loss Plots
This section provides an extended analysis of the experiments investigating the generalization consequences of learner collusion discussed in Section 6.2. Figure 10 provides the same generalization gap plots from Figure 5, but decomposed into the training and testing loss. For analysis purposes, we trained these models for 100 epochs, but this plot highlights that convergence occurs much earlier. These results reveal that the generalization gap is caused by both a slower training set convergence in addition to a greater test set loss. This slower convergence may in part be due to the two terms of the joint objective  $(\beta = 1)$  competing against each other in the direction of their gradient updates. More concerning is the larger test set loss. This indicates that the solution achieved as training loss eventually becomes low, generalizes much more poorly than the solution achieved by the independent training objective  $(\beta = 0)$ .
Figure 10: Train & test loss. The jointly trained model  $(\beta = 1)$  converges more slowly on the training set and reaches a solution that generalizes more poorly than the independent objective  $(\beta = 0)$ .
We also provide a decomposition of the loss into individual error (ERR) and diversity (DIV) on both the training set (Figure 11) and the test set (Figure 12). From the training plots, it appears that the joint objective  $(\beta = 1)$  maximizes diversity early in training before beginning to minimize individual error later. This aligns with the learner collusion effect we describe in Section 6.1 as it is trivial to inflate diversity while minimizing the individual error requires solving the true objective of the task. The test set decomposition in Figure 12 reinforces that the solutions obtained with joint training generalize poorly beyond the training data.
Figure 11: Train loss decomposition. The training loss decomposed into diversity (DIV) and individual error (ERR) highlights that diversity is trivially inflated early in the training process when joint training is applied ( $\beta = 1$ ).
Figure 12: Test loss decomposition. The test loss decomposed into diversity (DIV) and individual error (ERR) reinforces that optimizing for learner collusion does not generalize effectively onto a test set.
G.2 Larger Ensembles
In Figure 13 we reproduce the experiments with ResNet-18 base learners from Figure 6 (left) but with a larger ensemble such that  $M = 10$ . We observe that, while there is a small improvement in test accuracy, the shape of the curves with increasing  $\beta$  are consistent with those of the smaller ensemble.
Figure 13: Larger ensembles. Identical experiments with larger ensembles ( $M = 10$ ) result in a similar trend in the accuracy curves.
G.3 Score Averaging
Figure 14 includes the results of repeating the experiments in the top row of Figure 6 using score-averaging rather than probability averaging as described in Section 4.2. Similarly to previous works, we find that joint training performs poorly in this setting too. We also note that test set performance is similar to that of probability averaging on both datasets.
G.4 CNN Ensembles
We repeat the experiments described in Section 6.3 with ensembles consisting of base learners with a vanilla convolutional neural network architecture. The architecture of each of these base learners is provided in Table 3. Dropout with probability 0.1 is applied throughout the network during training. Furthermore, we apply Adam optimizer [55] with learning rate 0.001 and early stopping
Figure 14: Score averaging. This objective also performs significantly worse when applying joint training  $(\beta = 1)$ .
with patience of 5 epochs. Otherwise, all experimental parameters are consistent with those previous experiments. We apply this ensemble to CIFAR-10 with ensembles of size  $M = 5$  and  $M = 10$  with results provided in Figure 15. The results of this experiment are consistent with those of previous architectures and we again note the poor test set performance of the jointly trained model  $(\beta = 1)$ . Interestingly, we find in both cases that the lowest test loss is achieved at interpolating values of  $\beta$  as achieved by the augmented objective  $\mathcal{L}^{\beta}$  as described in Section 5.
Table 3: Vanilla Convolutional Neural Network Architecture.
Layer Type	Hyperparameters	Activation Function
Conv2d	Input Channels:3 ; Output Channels:10 ; Kernel Size:5 ; Stride:2 ; Padding:1	ReLU
Conv2d	Input Channels:10 ; Output Channels:20 ; Kernel Size:5 ; Stride:2 ; Padding:1	ReLU
Flatten	Start Dimension:1
Linear	Input Dimension: 500 ; Output Dimension: 50	ReLU
Linear	Input Dimension: 50 ; Output Dimension: 10
Figure 15: CNN ensembles. Standard architectures achieve similarly poor results for joint training  $(\beta = 1)$  but obtain their lowest test loss at intermediate values  $(\beta \in (0,1))$  using  $\mathcal{L}^{\beta}$ .
G.5 SVHN with VGG architecture
In this experiment we repeat the experiments in Section 6.3 on the SVHN dataset [58] using a VGG style architecture [14]. We report the exact architecture in Table 4. Again, we use ensembles of size 5. We use the same optimization protocol as applied to the ResNet-based ensembles as reported in Appendix B. Consistent with all previous experiments we observe that joint training again results in degenerate performance in this setting.
G.6  $\overline{\mathbf{ERR}}$  as a metric
For completeness we report the  $\overline{\mathrm{ERR}}$  test set performance corresponding to the experiments in Figure 6 (left). We note that this is exactly the ensemble loss which is optimized at  $\beta = 1$ . This is useful to report as it is the exact quantity being optimized during training. Consistent with all previous results, performance is worst as  $\beta \rightarrow 1$  (i.e. as we approach optimizing for the metric being reported).