SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sigma_ {\eta} ^ {L} (\tilde {x}, z) := \mathfrak {m l p} _ {\eta} ([ \tilde {x}, z ]) ^ {\top} z \in \mathbb {R} ^ {C}, \tag {6}
$$
where  $\mathfrak{mlp}_{\eta}$  is an MLP with outputs in  $\mathbb{R}^{D_Z\times C}$  and  $[\tilde{x},z]$  is a flattened concatenation of  $\tilde{x}$  and  $z$ . Depending on the choice of hidden units,  $\sigma_{\eta}^{L}$  can represent highly nonlinear functions in  $\tilde{x},z$  and thus allow for expressive joint predictions in high level features. The design, initialization and scaling the prior network  $\sigma^P$  allows an algorithm designer to encode prior beliefs, and is essential for good performance in learning tasks. Typical choices might include  $\sigma^P$  sampled from the same architecture as  $\sigma^L$  but with different parameters.
4.2 Training loss function
While the epinet's novelty primarily lies in its architecture, the choice of loss function for training can also play a role in its performance. In many classification problems, the standard regularized log loss suffices:
$$
\ell_ {\lambda} ^ {\mathrm {X E N T}} \left(\theta , z, x _ {i}, y _ {i}, i\right) := - \ln \left(\operatorname {s o f t m a x} \left(f _ {\theta} \left(x _ {i}, z\right)\right) _ {y _ {i}}\right) + \lambda \| \theta \| _ {2} ^ {2}. \tag {7}
$$
Here,  $\lambda$  is a regularization penalty hyperparameter, while other notation are as defined in Sections 3 and 4.1. This is the loss function we use for the experiments with the Neural Testbed (Section 5) and ImageNet (Section 6).
Image classification benchmarks often exhibit a very high signal-to-noise ratio (SNR); identical images are almost always assigned the same label. As demonstrated by Dwaracherla et al. (2022), when the SNR is not so high and varies significantly across inputs, it can be beneficial to randomly perturb the loss function via versions of the statistical bootstrap (Efron and Tibshirani, 1994). For example, a Bernoulli bootstrap omits each data pair with probability  $p \in [0,1]$ , giving rise to a perturbed loss function:
$$
\ell_ {p, \lambda} ^ {\mathrm {X E N T}} (\theta , z, x _ {i}, y _ {i}, i) := \left\{ \begin{array}{l l} \ell_ {\lambda} ^ {\mathrm {X E N T}} (\theta , z, x _ {i}, y _ {i}, i) & \text {i f} c _ {i} ^ {T} z > \Phi^ {- 1} (p) \\ 0 & \text {o t h e r w i s e ,} \end{array} \right. \tag {8}
$$
where each  $c_{i}$  is an independent random vector sampled uniformly from the unit sphere, and  $\Phi (\cdot)$  is the cumulative distribution function of the standard normal distribution  $N(0,1)$ .
Note that the loss functions defined in equation 7 and 8 only explicitly state the loss for a single input-label pair  $(x_i, y_i)$  and a single epistemic index  $z$ . To compute a stochastic gradient, one needs to sample a batch of input-label pairs and a batch of epistemic indices, average the losses defined above, and then compute the gradient.
4.3 How can this work?
The epinet is designed to produce effective joint predictions. As such, one might expect the training loss to explicitly reflect this and be surprised that we use standard marginal loss functions such as  $\ell_{\lambda}^{\mathrm{XENT}}$ . Recall that a prediction  $f_{\theta}(x,z)$  produced by an epinet is given by a trainable component  $\mu_{\zeta}(x) + \sigma_{\eta}^{L}(\tilde{x},z)$  perturbed by the prior function  $\sigma^{P}(\tilde{x},z)$ , which has no trainable parameters. Minimizing  $\ell_{\lambda}^{\mathrm{XENT}}$  can therefore be viewed as optimizing the learnable component  $\mu_{\zeta}(x) + \sigma_{\eta}^{L}(\tilde{x},z)$  with a perturbed loss function. Previous work has established that learning with prior functions can induce effective joint predictions (Osband et al., 2018; He et al., 2020; Dwaracherla et al., 2020, 2022), we extend this to the epinet.
To show this marginal loss function can lead to effective joint predictions, Theorem 4 proves that this epinet training procedure can mimic exact Bayesian linear regression. Although this paper focuses on classification, in this subsection we consider a regression problem because its analytical tractability facilitates understanding. To establish this result, we introduce a regularized squared loss perturbed by Gaussian bootstrapping:
$$
\ell_ {\sigma , \lambda} ^ {\mathrm {L S G}} (\theta , z, x _ {i}, y _ {i}, i) := (f _ {\zeta , \eta} (x _ {i}, z) - y - \sigma c _ {i} ^ {T} z) ^ {2} + \lambda \left(\| \zeta \| _ {2} ^ {2} + \| \eta \| _ {2} ^ {2}\right). \tag {9}
$$
Here, each  $c_{i}$  is a context vector sampled uniformly from the unit sphere in  $D_Z$  dimensions, in the same manner as  $\ell_{p,\lambda}^{\mathrm{XENT}}$  (8).
We say that a dataset  $\mathcal{D}$  is generated by a linear-Gaussian model if  $g_{\nu_{*}}(x) = \nu_{*}^{\top}x$ ,  $\nu_{*} \sim N(0, \sigma_{0}^{2}I)$ , and each data pair  $(x_{i}, y_{i}) \in \mathcal{D}$  satisfies  $y_{i} = g_{\nu_{*}}(x_{i}) + \epsilon_{i}$  where  $\epsilon_{1:N}$  are i.i.d. according to  $N(0, \sigma^{2})$ . We say an ENN is a linear-Gaussian epinet with parameters  $\theta = (\zeta, \eta)$  if its trainable component is comprised of a linearly parameterized functions  $\mu_{\zeta}(x) = \zeta^{T}x$  and  $\sigma_{\eta}^{L}(\tilde{x}, z) = z^{T}\eta\tilde{x}$ , with epinet input  $\tilde{x} = x$ , the prior function takes the form  $\sigma^{P}(x, z) = \sigma_{0}z^{T}P_{0}x$ , where each column of the matrix  $P_{0}$  is independently sampled uniformly from the unit sphere, and the reference distribution is taken to be  $P_{z} \sim N(0, I_{D_{Z}})$ .
Theorem 4. Let data  $\mathcal{D} = \{(x_i,y_i,i)\}_{i = 1}^N$  be generated by a linear-Gaussian model and  $f$  be a linear-Gaussian epinet. Let  $\hat{\theta}\in \mathrm{argmin}_{\theta}\sum_{i = 1}^{N}\int_{z}P_{Z}(dz)\ell_{\sigma ,\lambda}^{\mathrm{LSG}}(\theta ,z,x_{i},y_{i},i)$  with parameter  $\lambda = \sigma^2 /(N\sigma_0^2)$ . Then, conditioned on  $(\mathcal{D},c_{1:N},P_0)$ ,  $f_{\hat{\theta}}(\cdot ,z)$  converges in distribution to  $g_{\nu_*}$  as  $D_Z$  grows, almost surely.
This result, proved in Appendix D, serves as a 'sanity check' that an epinet trained with a standard loss function can approach optimal joint predictions as the epistemic index dimension grows. Although our analysis is limited to linear-Gaussian models, the loss function applies more broadly. Indeed, we next demonstrate that epinets and standard loss functions scale effectively to large and complex models.
Table 1: Summary of benchmark agents, full details in Appendix G.
agent	description	hyperparameters
mlp	vanilla MLP	L2decay
ensemble	deep ensembles (Lakshminarayanan et al., 2017)	L2decay, ensemble size
dropout	Dropout (Gal and Ghahramani, 2016)	L2decay, network, dropout rate
bbb	Bayes by backprop (Blundell et al., 2015)	prior mixture, network, early stopping
hypermodel	hypermodel (Dwaracherla et al., 2020)	L2decay, prior, bootstrap, index dimension
ensemble+	ensemble + prior functions (Osband et al., 2018)	L2decay, ensemble size, prior scale, bootstrap
sgmcmc	stochastic gradient MCMC (Welling and Teh, 2011)	learning rate, prior, momentum
epinet	MLP + MLP epinet (this paper)	L2decay, network, prior, index dimension
5 The neural testbed
The Neural Testbed is an open-source benchmark that evaluates the quality of joint predictions in classification problems using synthetic data produced by neural-network-based generative models (Osband et al., 2022a). We use this as a unit test to sanity-check learning algorithms in a controlled environment and compare the epinet against benchmark approaches.
Table 1 lists the agents that we study as well as hyperparameters that we tune via grid search. For our epinet agent, we use base network  $\mu_{\zeta}(x)$  that matches the baseline mlp agent. We take the features  $\phi(x)$  to be a concatenation of the input  $x$  and the last hidden layer of the base network. We initialize the learnable epinet  $\sigma^L$  according to (6) with 2 hidden layers of 15 hidden units. The prior network  $\sigma^P$  is initialized as an ensemble of  $D_Z = 8$  networks each with 2 hidden layers of 5 hidden units in each layer, and combine the output by dot-product with index  $z$ . We push the details, together with open source code, to Appendix G.
Figure 5 examines the trade-offs between statistical loss and computational cost for epinet against benchmark agents. The bars indicate standard error, estimated over the testbed's random seeds. After tuning, all agents perform similarly in marginal prediction, but the epinet is able to provide better joint predictions at lower computational cost. We first compare the performance of the epinet (blue) against that of an ensemble (red) as we grow the number of particles. We show the epinet is able to perform much better than an ensemble with 100 particles, with a model less than twice the size of a single particle. These results are still compelling when we compare against ensemble+, which includes prior functions, and which is necessary for good performance in low data regimes included in testbed evaluation. Included in this plot are the other agents bbb, dropout and hypermodel agents. The dashed line indicates performance of sgmcmc agent, which can asymptotically obtain the Bayes optimal solution, but at a much higher computational cost.
6 ImageNet
Benefits of the epinet scale up to more complex datasets and, in fact, become more substantial. This section focuses on experiments involving the ImageNet dataset (Deng
model size (number of parameters)
Figure 5: Quality of marginal and joint predictions across models on the Neural Testbed.
et al., 2009); qualitatively similar results for both CIFAR-10 and CIFAR-100 are presented in Appendix H. We compare our epinet agent against ensemble approaches as well as the uncertainty baselines of Nado et al. (2021). Even after tuning to optimize joint log-loss, none of these agents match epinet performance. We assess joint log-loss via dyadic sampling (Osband et al., 2022b), as explained in Appendix F.
For our experiments, we first train several baseline ResNet architectures on ImageNet. We train each of the ResNet-  $L$  architectures for  $L\in \{50,101,152,200\}$  in the Jaxline framework (Babuschkin et al., 2020). We tune the learning rate, weight decay and temperature rescaling (Wenzel et al., 2020) on ResNet-50 and apply those settings to other ResNets. The ensemble agent only uses the ResNet-50 architecture. After tuning hyperparameters, we independently initialize and train 100 ResNet-50 models to serve as ensemble particles. These models are then used to form ensembles of sizes 1, 3, 10, 30, and 100.
The epinet takes a pretrained ResNet as the base network with frozen weights. We fix the index dimension  $D_Z = 30$  and let the features  $\phi$  be the last hidden layer of the ResNet. We use a 1-layer MLP with 50 hidden units for the learnable network (6). The fixed prior  $\sigma^P$  consists of a network with the same architecture and initialization as  $\sigma_{\eta}^{L}$ , together with an ensemble of small random convolutional networks that directly take the image as inputs. We push details on hyperparameters and evaluation, with open-source code, to Appendix H.
Figure 6: Marginal and joint predictions on ImageNet.
Figure 2 presents our key result of this paper: relative to large ensembles, epinets greatly improve joint predictions at orders of magnitude lower compute cost. The figure plots performance of three agents with respect to three notions of loss as a function of model size in the spirit of Kaplan et al. (2020). The first two plots assess marginal prediction performance in terms of classification error and log-loss. Performance of the ResNet scales similarly whether or not supplemented with the epinet. Performance of the ensemble does not scale as well as that of the ResNet. The third plot pertains to performance of joint predictions and exhibits dramatic benefits afforded by the epinet. While joint log-loss incurred by the ensemble agent improves with model size more so than the ResNet, the epinet agent outperforms both alternatives by an enormous margin.
Figure 6 adds evaluation for the best single-model agents from uncertainty baselines (Nado et al., 2021): epinet also outperforms all of these methods. We tuned each uncertainty baseline agent to minimize joint log-loss, subject to the constraint
that their marginal log-loss does not degrade relative to published numbers. We can see that the epinet offers substantial improvements in joint prediction compared to sngp (Liu et al., 2020), dropout (Gal and Ghahramani, 2016), mimo (Havasi et al., 2020) and het (Collier et al., 2020). As demonstrated in Appendix H, these qualitative observations remain unchanged under alternative measures of computational cost, such as FLOPs.
This paper introduces ENNs as a new interface for uncertainty modeling in deep learning. We do this to facilitate the design of new approaches and the evaluation of joint predictions. Unlike BNNs, which focus on inferring unknown network parameters, ENNs focus on uncertainty that matters in the predictions. The epinet is a novel ENN architecture that cannot be expressed as a BNN and significantly improves the tradeoffs in prediction quality and computation. For large models, the epinet enables joint predictions that outperform ensembles consisting of hundreds of particles at a computational cost only slightly more than one particle. Importantly, you can add an epinet to large pretrained models with modest incremental computation.
ENNs enable agents to know what they do not know, thereby unlocking intelligent decision making capabilities. Specifically, ENNs allow agents to employ sophisticated exploration schemes, such as information-directed sampling (Russo and Van Roy, 2014), when tackling complex online learning and reinforcement learning problems. A recent paper (Osband et al., 2023) has demonstrated efficacy of the epinet in several benchmark bandit and reinforcement learning problems. While ENNs can also be integrated into large language models, we defer this to future work.
Babuschkin, I., Baumli, K., Bell, A., Bhupatiraju, S., Bruce, J., Buchovsky, P., Budden, D., Cai, T., Clark, A., Danihelka, I., Fantacci, C., Godwin, J., Jones, C., Hennigan, T., Hessel, M., Kapturowski, S., Keck, T., Kemaev, I., King, M., Martens, L., Mikulik, V., Norman, T., Quan, J., Papamakarios, G., Ring, R., Ruiz, F., Sanchez, A., Schneider, R., Sezener, E., Spencer, S., Srinivasan, S., Stokowiec, W., and Viola, F. (2020). The DeepMind JAX Ecosystem.
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015). Weight uncertainty in neural networks. In International Conference on Machine Learning, pages 1613-1622. PMLR.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs.
Charpentier, B., Zügner, D., and Gunnemann, S. (2020). Posterior network: Uncertainty estimation without ood samples via density-based pseudo-counts. Advances in Neural Information Processing Systems, 33:1356-1367.
Collier, M., Mustafa, B., Kokiopoulou, E., Jenatton, R., and Berent, J. (2020). A simple probabilistic method for deep classification under input-dependent label noise. arXiv preprint arXiv:2003.06778.
Collier, M., Mustafa, B., Kokiopoulou, E., Jenatton, R., and Berent, J. (2021). Correlated input-dependent label noise in large-scale image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1551-1560.
de Finetti, B. (1929). Funzione caratteristica di un fenomeno aleatorio. Atti del Congresso Internazionale dei Matematici: Bologna, (1):1979 - 190.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. IEEE.
Der Kiureghian, A. and Ditlevsen, O. (2009). Aleatory or epistemic? does it matter? Structural safety, 31(2):105-112.
Dwaracherla, V., Lu, X., Ibrahimi, M., Osband, I., Wen, Z., and Van Roy, B. (2020). Hypermodels for exploration. In International Conference on Learning Representations.
Wenzel, F., Roth, K., Veeling, B. S., Swiatkowski, J., Tran, L., Mandt, S., Snoek, J., Salimans, T., Jenatton, R., and Nowozin, S. (2020). How good is the Bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405.
Wilson, A. G. (2020). The case for Bayesian deep learning. arXiv preprint arXiv:2001.10995.
A Open source code
Two related github repositories complement this paper:
1. enn: https://anonymous.4open.science/r/enn-55BC
2. neural_testbed: https://anonymous.4open.science/r/neural_testbed-8961
These libraries contain the code necessary to reproduce the key results in our paper, divided into repositories based on focus. Together with each repository, we include several 'tutorial colabs' - Jupyter notebooks that can be run in a browser without requiring any local installation. Each of these libraries is written in Python, and relies heavily on JAX for scientific computing (Bradbury et al., 2018). We view this open-source effort as a major contribution of our paper.
The first library, enn, focuses on the design of epistemic neural networks and their training. This includes all of our network definitions and loss functions. Our library is built around Haiku (Babuschkin et al., 2020). The library provides the basis for all of the computational work reported in this paper.
The second library, neural_testbed, was introduced as part of The Neural Testbed (Osband et al., 2022a). We add the epinet agent, suitable for comparison with the existing agent implementations in that library.
B From predictions to decisions
Suppose a neural network has been trained on a dataset  $\mathcal{D}$  and, then, for a random input  $x$  with label  $y$  generates a distributional prediction  $\hat{P}$ . The expected marginal log loss is  $-\mathbb{E}[\ln \hat{P}(y)|\mathcal{D}]$ . Note that this expectation is over  $x, y,$  and  $\hat{P}$ . Results we establish point out that, while minimizing this produces optimal marginal predictions, that does not generally lead to performant downstream decisions.
To formulate a generic decision problem, consider a reward function  $r$  that maps an action  $a \in \mathcal{A}$  and  $\tau$  labels  $y_{1:\tau}$  to a reward  $r(a,y_{1:\tau}) \in [0,1]$ . Conditioned on training data  $\mathcal{D}$  and random inputs  $x_{1:\tau}$ , an action  $a$  generates expected reward
$$
\mathbb {E} \left[ r (a, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} \right] = \sum_ {y _ {1: \tau}} P _ {1: \tau} (y _ {1: \tau}) r (a, y _ {1: \tau}),
$$
where  $P_{1:\tau}(y_{1:\tau}) = \mathbb{P}(y_{1:\tau} = \cdot | \mathcal{D}, x_{1:\tau})$  is the posterior predictive distribution. An optimal action  $a_*$  can be determined by maximizing this conditional expectation. If an approximation  $\hat{P}_{1:\tau}$  is used instead of  $P_{1:\tau}$  then the objective becomes  $\sum_{y_{1:\tau}} \hat{P}_{1:\tau}(y_{1:\tau}) r(a, y_{1:\tau})$ .
Let  $P_{t} = \mathbb{P}(y_{t} = \cdot |\mathcal{D},x_{t}) = \sum_{y_{1:t - 1},y_{t + 1,\tau}}P_{1:\tau}(y_{1:\tau})$  be the marginal posterior predictive. Let  $\hat{P}_t = \sum_{y_{1:t - 1},y_{t + 1,\tau}}\hat{P}_{1:\tau}(y_{1:\tau})$  be the approximate marginal. A joint prediction  $\hat{P}_{1:\tau}$  minimizes marginal cross-entropy when  $\hat{P}_t = P_t$  for  $t = 1,\dots ,\tau$ . The following result establishes that such predictions that minimize marginal log loss can result in decisions no better than random guesses. Note that when it is clear from context, we use notation for a set, like  $\mathcal{A}$ , to express cardinality of the set. The exchangeability requirement – that the distribution of any set of data pairs be exchangeable – ensures that we are in a standard supervised learning setting.
Theorem 1. [formal] For all  $\tau$ , there exists an action set  $\mathcal{A}$  with  $|\mathcal{A}| = 2^{\tau}$ , an exchangeable data distribution, and a reward function  $r$  with range  $[0,1]$  such that if  $\hat{P}_{1:\tau}(y_{1:\tau}) = \prod_{t=1}^{\tau} \mathbb{P}(y_t | \mathcal{D}, x_{1:\tau})$  and  $\hat{a} \sim \mathrm{unif}(\arg \max_{a \in \mathcal{A}} \sum_{y_{1:\tau}} \hat{P}_{1:\tau}(y_{1:\tau}) r(a, y_{1:\tau}))$  then
$$
\max _ {a \in \mathcal {A}} \mathbb {E} [ r (a, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} ] = 1 \qquad a n d \qquad \mathbb {E} [ r (\hat {a}, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} ] \leq \mathbb {E} [ r (\tilde {a}, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} ] <   1,
$$
where, conditioned on  $\mathcal{D}$  and  $x_{1:\tau}$ ,  $\tilde{a} \sim \mathrm{unif}(\mathcal{A})$ .
Proof. Let  $\tau$  be even; extending our argument to odd  $\tau$  is straightforward. Without loss of generality, let the input space be a singleton. Hence,  $\mathbb{P}(y_{1:\tau} = \cdot |\mathcal{D},x_{1:\tau}) = \mathbb{P}(y_{1:\tau} = \cdot |\mathcal{D})$ .
Let labels  $y_{t}$  be in  $\{-1,1\}$  so that  $y_{1:\tau} \in \{-1,1\}^{\tau}$ . Let  $\mathcal{A} = \{-1,1\}^{\tau}$ . Let  $r(a,y_{1:\tau}) = 1 - (\sum_{t=1}^{\tau} a_{t} y_{t} / \tau)^{2}$ . Let  $\mathbb{P}(y_{1} = \dots = y_{\tau} | \mathcal{D}) = 1$  and, for all  $t$ ,  $\mathbb{P}(y_{t} = 1 | \mathcal{D}) = 1 / 2$ . Clearly, this data distribution is exchangeable, with uniform marginals. Hence, for any  $a \in \mathcal{A}$ ,
$$
\begin{array}{l} \sum_ {y _ {1: \tau}} \hat {P} _ {1: \tau} (y _ {1: \tau}) r (a, y _ {1: \tau}) = \sum_ {y _ {1: \tau}} \hat {P} _ {1: \tau} (y _ {1: \tau}) \left(1 - \left(\frac {1}{\tau} \sum_ {t = 1} ^ {\tau} a _ {t} y _ {t}\right) ^ {2}\right) \\ = 1 - \sum_ {y _ {1: \tau}} \hat {P} _ {1: \tau} \left(y _ {1: \tau}\right) \left(\frac {1}{\tau} \sum_ {t = 1} ^ {\tau} y _ {t}\right) ^ {2} \\ = 1 - \frac {1}{\tau^ {2}} \left(\sum_ {t = 1} ^ {\tau} \sum_ {y _ {1: \tau}} \hat {P} _ {t} (y _ {t}) y _ {t} ^ {2}\right) \\ = 1 - 1 / \tau . \\ \end{array}
$$
It follows that any action  $\hat{a}$  maximizes  $\sum_{y_{1:\tau}}\hat{P}_{1:\tau}(y_{1:\tau})r(a,y_{1:\tau})$ , and therefore, selecting randomly from this set results in expected reward  $\mathbb{E}[r(\hat{a},y_{1:\tau})|\mathcal{D},x_{1:\tau}] = 1 - 1 / \tau$ . Hence,
$$
\begin{array}{l} \mathbb {E} [ r (\hat {a}, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} ] = \mathbb {E} [ r (\tilde {a}, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} ] \\ = 1 - \mathbb {E} \left[ \left(\frac {1}{\tau} \sum_ {t = 1} ^ {\tau} \tilde {a} _ {t} y _ {t}\right) ^ {2} \mid \mathcal {D}, x _ {1: \tau} \right] \\ = 1 - \mathbb {E} \left[ \left(\frac {1}{\tau} \sum_ {t = 1} ^ {\tau} \tilde {a} _ {t}\right) ^ {2} \right] \\ = 1 - \frac {1}{\tau^ {2}} \sum_ {t = 1} ^ {\tau} \mathbb {E} [ \tilde {a} _ {t} ^ {2} ] \\ = 1 - 1 / \tau . \\ \end{array}
$$
For any action  $a$  such that  $\sum_{t=1}^{\tau} a_t = 0$ ,
$$
\begin{array}{l} \mathbb {E} [ r (a, y _ {1: \tau}) | \mathcal {D}, x _ {1: \tau} ] = \mathbb {P} (y _ {1: \tau} = - \vec {1} | \mathcal {D}) \left(1 - \left(\frac {1}{\tau} \sum_ {t = 1} ^ {\tau} - a _ {t}\right) ^ {2}\right) \\ + \mathbb {P} \left(y _ {1: \tau} = \vec {1} | \mathcal {D}\right) \left(1 - \left(\frac {1}{\tau} \sum_ {t = 1} ^ {\tau} a _ {t}\right) ^ {2}\right) \\ = 1. \\ \end{array}
$$
The result follows.
The first displayed equation of this theorem asserts that the optimal expected reward is one, while the second asserts that an action selected based on  $\hat{P}_{1:\tau}$ , which minimizes marginal log loss, does no better than an action chosen uniformly at random, which earns expected reward less than one.
While optimizing marginal predictions does not necessarily lead to performant downstream decisions, minimizing joint log loss does. Our next result formalizes this by bounding performance shortfall by a function of the KL-divergence:
$$
\mathbf {d} _ {\mathrm {K L}} \left(P _ {1: \tau} \| \hat {P} _ {1: \tau}\right) = \sum_ {y _ {1: \tau}} P _ {1: \tau} \left(y _ {1: \tau}\right) \ln P _ {1: \tau} \left(y _ {1: \tau}\right) - \sum_ {y _ {1: \tau}} P _ {1: \tau} \left(y _ {1: \tau}\right) \ln \hat {P} _ {1: \tau} \left(y _ {1: \tau}\right).
$$