The Neural Testbed: Evaluating Joint Predictions
Authors: Ian Osband
Zheng Wen
Seyed Mohammad Asghari
Vikranth Dwaracherla
Xiuyuan Lu
Morteza Ibrahimi
Dieterich Lawson
Botao Hao
Brendan O'Donoghue
Benjamin Van Roy
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Learning
Bayesian
Uncertainty
Testbed
open source

Abstract: 
Predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Neural Testbed: an open source benchmark for controlled and principled evaluation of agents that generate such predictions. Crucially, the testbed assesses agents not only on the quality of their marginal predictions per input, but also on their joint predictions across many inputs. We evaluate a range of agents using a simple neural network data generating process.

Our results indicate that some popular Bayesian deep learning agents do not fare well with joint predictions, even when they can produce accurate marginal predictions. We also show that the quality of joint predictions drives performance in downstream decision tasks. We find these results are robust across choice a wide range of generative models, and highlight the practical importance of joint predictions to the community.

The Neural Testbed: Evaluating Joint Predictions
Ian Osband; Zheng Wen, Seyed Mohammad Asghari, Brendan O'Donoghue, Botao Hao, Dieterich Lawson, Morteza Ibrahimi, Xiuyuan Lu, Vikranth Dwaracherla, and Benjamin Van Roy DeepMind, Efficient Agent Team, Mountain View
Abstract
Predictive distributions quantify uncertainties ignored by point estimates. This paper introduces The Newral Testbed: an open-source benchmark for controlled and principled evaluation of agents that generate such predictions. Crucially, the testbed assesses agents not only on the quality of their marginal predictions per input, but also their joint predictions across many imputs. We evaluate a range of agents using a simple neural network data generating process. Our results indicate that some popular Bayesian deep learning agents do not fare well with joint predictions, even when they can produce accurate marginal predictions. We also show that the quality of joint predictions drives performance these results are robust across choice wide range of generative models, and highlight the practical importance of joint predictions to the community.
downstream decision tasks. We find
Introduction
Most work on supervised learning has focused on marginal predictions. Marginal predictions predict one label given one input, but do not model the dependence between multiple predictions. For decision making, it not enough have good marginal predictions; the quality of join!. predictions drives decision performance (Wen et al. 2022). Joint predictions predict multiple labels given multiple imputs, and may capture some correlation between outcomes. This distinction can be particularly important in learning settings where joint predictions allow an agent to distinguish what it knows from what does not know
ct al.
2011
Lu et al. 2021
coin
bies
marginal prediction
joint prediction
f
0.5
26 0.25
0.5
0 or
0.5
05
Figure 1: Two coins with identical marginal predictions, but distinguished by joint predictions.
"Contact iosband@deepmind com
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

Figure 1 presents stylized example designed to highlight the importance of joint predictions in decision making. Consider two coins 'f' and &s' with different bro=bbprobaiility of heads'. Coin & has a known bias of 2. whereas coin $ has unknown bias of either 0 or 1, and which are both equally likely. Examining the marginal prediction over a single flip: the two coins present identical outcomes 50:50. However, if we consider the outcome over two successive flips, which can be modeled as a two-by-two grid, then the difference between these coins is evident in their joint predictions. If you want to maximize the cumulative heads through time, then it's important to know the difference between these two settings. In this case, learning agent should first choose $ and then, depending the outcome of that flip heads/ /tails, employ a fixed policy of $/ f. forward. Marginal predictions alone cannot drive this sort of policy, since they do not distinguish the two coins (Wen et al. 2022
Our research is motivated by the grand challenges artificial intelligence, and the great progress that has been made in deep learning systems (Krizhevsky et al. 2012 Brown et al. 2020 However, as these systems move beyond prediction and towards actually making decisions we have very little understanding of how and where popular deep learning approaches are suitable for joint predictions and hence decision making (Mnih et al. 2015 Silver et al. 2016). To this end, we introduce The Newral Testbed as simple and clear benchmark for evaluating the quality of joint predictions in deep learning systems. This work is meant to be 'sanity check' for popular deep learning approaches in simple setting, and one that can help guide future research.
The Neural Testbed works by generating random classification problems using neural- network-based generative process. The testbed splits data into a training set and testing set, allows deep learning agent to train on the training set, and then evaluates the quality of the predictions on the testing set. is worth noting that the problem framed by the Testbed is a computational one. Optimal performance would be attained by carrying out exact Bayesian inference: given infinite compute time, an agent could calculate the posterior distribution, which maximizes performance. However, due to the complexity of the data generating process, this is infeasible. The agents we study serve as approximate inference algorithms, and we compare their performance purely through the quality of their predictions, without worrying 'is XYZ Bayesian?' [Izmallvv et al. 2021).
Figure@offers offeers preview of our results in Section where we compare benchmark approaches to Bayesian deep learning. This plot shows the KL loss when making simultancous predictions. We compare the quality of marginal (T = 1) and joint (r = 10) predictions, normalized so that and MLP has loss=1. We see that, after tuning, most Bayesian deep learning approaches do not significantly outperform a single MLP in marginal predictions.
However, once we examine joint predictive distributions of order T = 10, there clear difference in performance among benchmark agents. In particular, some of the most popular benchmark approaches
al.] 2017), dropout (Gal and Chahramani] 2016), bbb [Blundell et al. 2015)) do not outperform the baseline MLP when evaluated joint predictions. At the same time, there are other approaches that perform much better in terms of joint predictions in this simple synthetic challenge. We will go on to show that these same agents perform better in docision making, and that these observations are robust to choice of generative model.
Bayesian deep learning (ensemble (Lakshminarayanan
normalized KL loss
tau
0.8.
10
0.6
mip
ensemble
drepout
bbb agent
hypermedel
ensemble+
sgmcmc
Figure 2: Quality of marginal and joint predictions on Neural Testbed (Section 4.2

---

1.1 Key contributions
We introduce The Neural Testbed, a simple benchmark for the field that involves making predictions in a neural-network- to bridge theory and practice, and provide an objective metric to assess the quality of approximate posterior inference in neural networks. We are the first paper to propose a concrete evaluation procedure for the quality of joint predictions in neural network classification.
generative model. This work helps
Together with this conceptual contribution, we open-sourec code in Appendix A This consists of highly optimized evaluation code, reference agent implementations and automated reproducible analysis. The lestbed uses JAX internally (Bradbury et al. 2018), but can be used lo evaluate any python agent. We believe that this library will be major contribution to researchers and, due to ils low computational cost, a boon to accessibility.
We use this new benchmark to obtain some important new experimental results. We discover that several of the most popular approaches to Bayesian deep learning do not perform well at joint prediction, and highlight this issuc to the community. Further, we show that there are alternative approaches that do perform well in terms of joint prodiction. Prior work has suggested that, in theory, the quality of joint predictions can drive decision performance (Wen et al.] 2022). In this paper we provide empirical evidence that this elfect occurs in praclical deep learning systems. We observe that performance in a neural bandit is highly correlated with performance in joint prediction, and that it is not significantly correlated with the quality of marginal predictions.
Finally, we show that the results in this paper are robust to the variations in the data generating model. Although we focus on a 2-layer ReLU MLP with 50 hidden units for most of our experiments, the results we obtain are highly correlated across wide range of alternative activation functions or network widths. This robustness supports the view that the field should be aware of these issues joint prediction, and may help to stimulate future rescarch in this area. Follow-up work has gone on to show that these results also carry over to challenge datasets popular in the community (Osband et al., 2022
1.2 Related work
There is a rich literature around uncertainty estimation in deep learning. Much of this work has focused agent development, with wide variety of approaches including variational inference (Blundell et al., 2015). dropout (Gal and Ghahramani 2016), ensembles (Osband and Van Roy 2015 lakshmmaaaaaannn et al.) 2017), and MCMC (Welling and Teh 2011 Hoffman et al. 2014). However, even when approaches become popular within particular research communities, there are still significant disagreements over the quality of the resultant
uncertainty estimates (Osband 2016;
Hron et al.] 2017).
Bayesian deep learning has largely relied on benchmark problems to guide agent development and measure agent progress. These typically include classic deep learning datasets but supplement the usual goal of classification accuracy to include evaluation of the probablistic predictions via negative log likelihood (NLL) and expected calibration error (ECE) (Nado) et al. 2021). More recently, several efforts have been made to supplement these datasets with challenges tailored towards Bayesian deep learning, and explicit Bayesian inference Wilson al. 2021). This literature has largely focused on evaluating marginal predictions, paired with evaluation on downstream tasks (Riquemee et al. 2018 Our work is motivated by the importance of joint predictions in driving good performance
sequential decisions (Wen et al.) )022). We share motivation with the work of |Wan et al. (2021), but show that directly measuring joint likelihoods can provide new information beyond marginals. Follow up work has built upon the research in our paper, to extend the analysis of joint distributions to higher-order joint distributions, and empirical datasets (Osband et al. 2022).
2 Evaluating predictive distributions
In this section, we introduce notation for the standard supervised learning we consider as well as our evaluation metric: KT-loss. We review the distinction between marginal and joint predictions, and numerical schemes to estimate KL divergence via Monte Carlo sampling.
3

---

Algorithm 1 KT-Loss Estimation
Generative process
for j= :.2.....JJ do sample environment. and training data train agent on training data for .,,,...,A do
Agent
Training data
Testing data
sample T test data pairs compute unvironmennlikellhood Pi,
Agent
compute agent likelihood fj.m end for end for return JN E/= E
Performance
matric
m=1 log (Pjn/Ps.nl
Figure 3: The Neural Testbed
2.1 Environment and predictions
Consider a sequence of pairs ((X,, =t= where each X, is a feature vector and each Y(+1 its target label. Each target label Ye+1 is produced by an environment &, which we formally take to be conditional distribution E()(Xe). The environment & is random variable; this reflects the agent's uncertainty about how labels are generated. Note that P(Y++ E X,) = E(IX,) and P(Y++1 E (I,)) E[E(-(X,)
We consider an agent that learns about the environment from training data Dr = ((x,, Y(+1) : t 0,1,..., - 1). After training, the agent predicts test- ing class labels YT+IT+ = Pr+1....,YT++: from unlabeled feature vectors X
We describe the agent's predictions in terms of generative model, parameterized by vector 0r that the agent learns from the training data Dr- For any inputs N7.r+r-1, Or determines predictive distribution, which could be used to sample imagined outcomes 11+17+T. Hence, the agents -th-order predictive distribution is given by
PT+LT4T = P(YT+1.+7 E 100,, XT.T+r-1).
which represents an approximation to what would be obtained by conditioning on the environment:
P = P((Y+11T++ E (,, -E,
If= 1, this represents a marginal prediction; that is a prediction of label for a single input. For 7 > this is a joint prediction over labels for different inputs.
2.2 Kullback Leibler loss
We use expected KL-loss to quantify the error between an agent's predictive distribution Pr+1T++ and the prescient prediction P++1T++ that would be made given full knowledge of the environment:
di.. E |KL (1
(1)
The expectation is taken over all random variables, including the environment E, the parameters 0r, Xr.T+r-1, and YI+1TTT. Note that dja, is equivalent to the widely used notion of cross-entropy loss, though offset by quantity that is independent of Or.
In contexts we will consider, it is not possible to compute dir,, exactly. As such, we will approximale dk. via Monte Carlo simulation, described by Algorithm [1 First, set of environments is sampled. Then, for each sampled environment, a training dataset is sampled. For sampled environment and corresponding training data set, the agent is re-initialized, trained, and then tested on N independent test data 7-samples. Note that each test data 7-sample includes data pairs. For each test data 7-sample, the likelihood of the environment Pjn is computed exactly, but that of the agent's prodictive distribution is approximated via another Monte Carlo simulation, and we use Py.m denote this approximation. The estimate of dkL, is taken to be the sample mean of these log- log-likeld ratios.

---

The Neural Testbed
In this section we introduce the Neural Testbed. We believe that a simple, clear and accessible testbed can provide significant value to community. We provide a high-level overview of the open-source code which we release in Appendix A We then provide more details on the underlying generative model, together with an extensive selection of benchmark agents that we have tuned to perform well in this setting.
3.1 Synthetic data generating processes
By data generating process, we do not mean only the conditional distribution of data pairs (Xt, (41)lE but also the distribution of the environment E. The Testbed considers 2-dimensional imputs and binary classification problems. The logits are sampled from 2-hidden-layer ReLU MLP with (50,50) hidden units and Xavier initialization (Glorot and Bengio 2010). We choose this process to be maximally simple and canonical in the deep learning world. However, we will on to show that the key findings of this paper are not particularly sensitive to the exact choice of generative model.
The Neural Testbed estimates KL-loss, with T E {1, 10), for three temperature settings and several training dataset sizes. The lemperature p controls the signal to noise ratio the class probabilities are given by boftmax(logits/// p)) For each value of т, the KL-losses are averaged produce an aggregate performance measure. Further details concerning data generation and agent evaluation are offered in Appendix B
3.2
Why do we need synthetic testbed?
The Neural Testbed is designed to be a maximally simple problem that investigates the key properties of uncertainty modeling in deep learning. Progress in deep learning has been driven both by challenge datasets that stretch agent capabilities (Deng et al. 2009 Krizhevsky et al.y 2012), together with foundational work that builds understanding (Bartlett et al.) 2021). ). In this work, we provide a benchmark designed to improve our understanding of probabilistic predictions beyond marginals. Doing well in the testbed is not necessarily impressive grand success in AI, although doing poorly in such simple setting may reveal fundamental flaws in algorithm design.
key property of the test that it is specified by probabilistic model, rather than a finite collection of datasets. Benchmarks that rank performance datasets vulnerable to overfitting through iterative hill-climbing on the data included in the benchmark (Russo and Zou 2016), which may not generalize to data outside of the benchmark (Recht et al. 2018). In contrast, access to a generative model means that we can produce an unlimited amount of testing data from problem of interest. We can avoid the dangers of overfitting to any specific choices of benchmark dataset simply by generating more samples.
3.3 Benchmark agents
Table T lists agents that we study and compare as well as hyperparameterss that we tune. In our experiments, we optimize these hyperparameters via grid search. Our implementations, which aim to match 'canonical' versions, are available Appendix A
In addition to these agent implementations, our open-source offerings include all the evaluation code to reproduce the results of this paper. Our experiments make extensive use of parallel computation to facilitate hyperparamcter sweeps. Nevertheless, the overall computational cost is relatively low by modern deep learning standards and relics only on standard CPUs. For reference, evaluating the mlp agent across all the problems in our testbed requires less than CPU-hours. We view our open-source effort as a substantial contribution of this work.
4 Results
We evaluate the benchmark agents of Section 3.3 across the Neural Testbed. We begin with an analysis of marginal predictions where, alter agent luning, all approaches are able to make reasonably good predictions. However, when we examine joönt predictions we find that agent performance can vary drastically, even for well-tuned agents. If an agent cannot
5

---

Table Summary of benchmark agents, full details in Appendix
agent
description
by ayperparameters
mip
Vamila MLP
La decay
ensemble
DDeep Ensemble' Lakshminaravanan et al,
2017
La docay, ensemble size
dropout
Dropout Gaal and Ghairamaa 2016
Lg deray, network, dropout rate
bbb
Bayes by Backprop [Bundel et 2015
prior mixture, network, early stopping
hypernodel
Hypermodel Dwatacherja of au 2120
La decay, prior, bootstrap, index dimension
ensenble+
Ensemble prior finctions Oshind
2018
L2 doray, ensemble size; prior scale, bootstrap learning prior, momentum
sgacnc
Stochastic Langevin MCMC Welling
2011
output accurate joint predictions in the testbed, we should question if expect that same agent to perform better other settings. These results provide significant new insights to the the design of effective learning agents, and are major contribution of this paper.
4.1 Performance in marginal predictions
We begin our evaluation of benchmark approaches to Bayesian deep learning in marginal predictions (T= 17=1), 1). One of the first questions one might consider is whether the generative model as outlined in Section [..1 represents meaningful challenge for deep learning systems. Figure compares the performance of naive uniform class probabilities, logistic regression, and tuned 2-layer MLP. This simple comparison demonstrates that the Neural Testbed is not trivially solved by agents without deep learning architocturcs.
AGENT
ACCURACY
ECE
dka
dj
0.4
MLPP
0.793
0.078
0.129
1.367
0.3
agent
ENSEMBLE
0.079
0.128
1.356
DROPOUT
0.793
0.080
0.128
1.347
BIB
0.792
0.079
0.129
1.375
RYPERMODRI,
0.793
0.081
0.130
1.107
ENSEMBLES
0.790
0.085
0.129
1.015
SCMCMC
0.796
0.082
0.122
0.917
Table 2: Agent performance, deviation
10c 2000
number of traininng points
from MLP greater than 2 stderr in bold.
Figure 4: Performance with growing data.
Marginal predictions have been the focus of the Bayesian deep Icarning literature. Despite this focus, Figure shows that none of the benchmark methods significantly outperform well-tuned MLP baseline in terms of dj.. This observation is mirrored when examine the average classification accuracy and expected calibration error (ECE) across the lestbed (Table 2 These results are different from the empirical observations in other challenge datasets, where much agent development has focused on improving ECE, and present an interesting new observation in the Bayesian deep learning literature (Nado et al., 2021). We have two main hypothesis for this discrepancy. First, our agents are tuned for Toes dkr.+ 10 ki. not ECE (see Appendix C) Socond, the generative model of Section 3.1 matches the agent architecture, with inputs sampled i.i.d. NO, I). Investigating the conditions in which these results hold more generally is an exciting area for future research.
4.2 Performance beyond marginals
One of the key contributions of this paper is to evaluate predictive distributions beyond marginals. Figure 2 shows that sgmcme is the top-performing agent, overall. This should be reassuring to the Bayesian deep learning community and beyond. In the limit of large compute this agent should recover the "gold-standard' of Bayesian inference, and it does indeed perform best (Welling and Teh (001I). However, some of the most popular approaches in this field (ensemble, dropout) do not actually provide good approximations to the predictive distributions of order r 10. In fact, we even see that ensemble+ and hypermodels can provide much better approximations to the Bayesian posterior than 'fully Bayesian' VI approaches like bbb (Wilson and Ixmailov (220)). We note too that while sgmemc performs best, it also requires orders of magnilude more computation than competitive methods even in this toy setting (see Appendix (.3). As we scale more complex environments, may therefore be worthwhile to consider alternative approaches.

---

To see where some agents are able to outperform, we compare ensemble and ensemble+ under the medium SNR regime. These agents are identical, except for the addition of randomized prior function (Osband et al. 2018). Figure 5 shows that, although these methods perform similarly the quality of their marginal predictions (T (T=1), 1), the addition of a prior function greatly improves the quality of joint predictive distributions (r = 10) in the low data regime. Note that, since the testbed considers 2D inputs, 100 training points may already be considered as in the high data regime. Figure provides some insight for how this benefit scales with the order т of the predictive distribution. We can see a clear trend that T increases does the separation between agents ensemble and ensemble+. For more intuition on how prior functions are able to drive this benefit, see Appendix D.1
tou:
Lau: 10
12
0.4
average KL loss
agent ensemble ensemble+
norallzed kL.
agert eecemble
crsemble-
0.1
10
100
1000
10
100
1000
number training points
joint predictive distribution
Figure 5: Prior functions help with joint predictions.
Figure 6: Benefit grows with T.
5 Sequential decisions
In this section, we will form a sequential decision problem based on the Neural Testbed, and show that it is the quality in joint predictions that is essential to driving good performance in sequential decision making. Further, we show that the insights gained from the simple 2D Neural Testbed can extend to high-dimensional decision problems.
5.1 Neural bandit
We use the generative model of the Neural Testbed to define a class of bandit problems Gittinn 1979)- First, we sample set of N actions X' (x1,-.., i.i.d. from a d- dimensional standard normal distribution. We then sample an environment E, which specifies the conditional probability &(Y,+1 E -(X;), according to the class of generative models described in Section 3.1 We pick the temperature, which controls the SNR, to be 0.1. At each timestep 1, an agent selects an action X, X and receives reward Rc+1 Y441- Let R2 E [R++1] E X, x] denote the expected reward of action æ conditioned on the environment, and let X. = arg max P2 denote the optimal action. We assess agent performance through regret(T) = E'- Rxx Rx.]. which measures the shortfall in expected cumulative rewards relative to an optimal decision maker.
We evaluate the testbed agents these bandit problems through actions selected by Thompson sampling, varying only the posterior predictive distributions that TS samples from. A TS agent. requires approximate posterior distribution over the environment, which is supplied by the testbed agents. At each timestep, TS samples an environment from the approximate posterior and selects action that optimizes for the sampled environment 1933 Russo et al. 2018). A complete algorithm is presented in Appendix
Thompson
5.2 Agent performance
We present empirical results of testbed agents on these random bandit problems with ==1000 actions drawn from d = 50 dimensional space. Figure shows the average regret through time for each of the agents as selected by the Neural Testbed, averaged over 20 random sceds We can see that for each learning agent, the quality of decisions improves through time. However, the quality of decisions greatly affected by the choice of agent.
2 We omit sgmcuc as the computational demands are several orders of magnitude too large to consider in ouline learning.

---

agent
average regret
ensemble
0.03
drapout
onsemble
0.01
hypermadel
0.003
10000
20000
30000
40000
50000
timestep (t)
Figure 7: Learning agent impaact TS regret in neural bandits.
To investigate the relationship between predictions and decisions repeat the experiment of Figure with 10 independent random initialization over all the testbed and bandit problems. We then empirically investigale the correlation between dic and total regret at T = 50, 000 for both - == and r = 10. We use bootstrap sampling to estimate confidence intervals on the correlation coefficient on logarithmic scale the 5th and 95th percentiles. Figures and support our claim that performance in dk is highly correlated with performance in sequential decision problems, whereas correlation to marginals is significant. We would not expect a perfect correlation the particular TS action selection strategy may introduce confounding factors, together with natural variability in scods.
Corralatinn 1..34, 0.991
coreeation 0.75)
agant mlp ensemblc boo
agent mip
total regre
1000
total regres.
1000
@ onsomble
03 0
onsembic
onsembler
hypernon model
hyperooal
0.11
0.32
0.13
0.14
0.9
1.1
1.2
1.4
marginal KL lass (T= 1)
joint KL loss (r= 10)
Figure 8: Testbed marginal performance not significantly correlated with regret.
Figure 9: Testbed joint performance highly correlated with regret.
6 Robustness of generative model
The experiments of Sections and 5 are all performed with the generative model as described in Scction 3.1 One natural concern is that these results might be sensitive to this choice of model, and be less transferable to general deep learning rescarch. In this section we repeal these analyses under different generalive models. We find that the quality of joint predictions and bandit. performance is extremely robust across choice generative models.
For these experiments we take the tuned agents Section and then evaluate these agents under different generative models. Whereas these agent hyperparameters were tuned for the 2-layer ReLU MLP with 50-50 hidden units, we will also these agents over alternative environments varying:
activation-Itanh, swish, sigmoid, selu, relu, leaky relu, gelu] (Figure 10 hidden units-(5, 10, 20, 50, 100] (Figure 11..
Evaluation for each these environments & proceeds as before: the agent is trained on data generated by & and then evaluated on the quality of predictions on testing data from & If the qualitative results under different environments are similar, then we know that our results are somewhat robust to the exact generative model we choose.
8

---

Dill
Ealt
bonid regget
8:98
0.45
1:21
0.35
0.31
101
1.3
0.50
0.
0.04
1.92
0.76
1.0
0:96
3.0
0.99
0.62
6.99
1.5
0.50
<4.9
4.9-0.77
activation
agmodd
4.76.04
92ku
0.01
3:0
1.0
0.76
0.00
0.08
0.
0.08
1.0
6082
0.52
0.96
6:46
0.4
6:3
1.5
0.28
6:92
0.30
C+0.
0.00
630
0:09
0.40
0.48
0:49
0:44
6440,77
Ioaky
4:20
0,7x0.9
6.0c
geNu
1.
0.00
3:03
6:47
0.01
30
94
04
AM
0:71
0:50
04
1.
0.44
0.
6.
8:48
0:41
leaty rél
1ala
digmold
Lami
gela
Kakly 185
1800
Lani
gold
kroliy reta
argmosid
Feston
Lane
activation
Figure 10: Correlation of agent performance across different activation functions.
KL [au--
XL teu-10
bandit regret
100
6:39
0.51
8.17
0.34
3.0
0.95
0.93
0.9
0.98
1.C
0.84
0.98
0.59
0.85
1.0
correlation
hidden units
10
81
0:44
0.
-0.4<0
10+
2:45
1.0
0.8
0.51
0.99
1.0
0:35
8
9.91
0:91
1.0
0.65
0.95
C.4x0.7
C.7x0.9
150
100
hidden units
Figure 11: Correlation agent performance across different hidden units.
Figure 10 and examine the empirical correlation coefficient between the vector of agent evaluations ions, under the metrics dkL, d and bandit regret. We see that, the marginal evaluations are highly correlated for 'similar' generative models (e.g. ReLU and leaky ReLU) but can even be anti-correlated when the models stray too far. However, the correlations are very high across wide range of generative models when look at either the quality of joint predictions or the regret in the bandit problems. These results help to build confidence in the key observations of agents in terms of performance on joint prediction (Figure 2 is not too sensitive to the choice of generative model, and so may hold some wider insight relevant the community. Follow up work has confirmed that these results are also highly correlated with performance on benchmark datasets (Osband al. 2022)
make in this paper. Notably, they suggest that the separation
7 Conclusion
The Neural Testbed investigates the quality of predictive uncertainty in joint predictions, as well as marginals. With this simple and clear 2D challenge we aim to build understanding that can inform the field's wider efforts in deep learning. We have shown that results on the testbed can offer new insights to agent development. Further, we establish that the insights gained in the testbed can scale up to complex and high-dimensional decision problems.
Beyond the results this paper, we believe this work can provide a base for future research:
Can design better learning algorithms for joint predictions, as well marginals?
Are there analogous results to Figure 2 on large-scale challenge datasets?
How can effective joint predictions drive better decisions?
We believe that studying these simple lestbed problems can help foster interplay between theory and practice, improve accessibility in the field, and complement existing research. We hope that this will accelerate the growth underslanding the field and, ultimately, drive forward the design of better learning agents.

---

References
Bartlett, P. L., Montanari, A., and Rakhlin, A. (2021). Deep learning: statistical viewpoint. Acto
numerrica,
Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. (2015), Weight uncertainty in neural PMLR.
network. In International Conference on Machine Learning, pages
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G.,
Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q (2018). JAX: composable transformations of Wython-NumPyy programs.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. arXiv preprint N.i:2005.14465.
Cover, T. and Hart, P. (1967). Nearest neighbor pattern classification. IEFE transactions on information Uheory, 13(1):21-27.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale
hierarchical image database. In 2009 JEEE conference computer vision and pattern recognition, pages 248 255. Teee.
Dwaracherla, V., Lu, X., Ibrahimi, M., Osband, I., Wen, Z., and Van Roy, B. (2020). Hypermodels for exploration. In International Conference on Learning Representations.
Friedman, J. H. (2017). The elements of slatistioal learning: Dala mining, inference, and prediction. springer open.
Gal, Y. and Ghahramani, Z. (2016). Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning.
Gittins, J. C. (1979). Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B (Methodological), 41(2):148-164
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the 18th international conference on artificial intelligence and statistics, pages 249 256.
He, B., Lakshminarayanan, B., and Teh, Y. W. (2020). Bayesian deep ensembles via the neural langent kernel. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Ad- wonces in Neural Information Processing Systems, volume 33, pages 1010-1022. Curran Associates, Inc.
Hoffman, M. D., Gelman, A., et al. (2014). The no-u-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res., 15(1):1593-166) -6223.
Hron, J., Matthews, A. G. d. G., and Ghahramani, Z. (2017). Variational Gaussian dropout is not Bayesian. arXiv preprint arXiv:1711.000 129999.
Izmailov, P., Vikram, S., Hoffman, M. D., and Wilson, A. G. (2021). What are Bayesian neural network posteriors really like? arXio preprint 1rXi::210..144:
Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep con- volutional neural networks. In Advances in Newral Information Processing Systems 25, pages 1097-1105.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Newral Information Processing Systems, pages 64105 6416.
Li, L., Littman, M. L., Walsh, T. J., and Strehl, A. L. (2011). Knows what it knows: a framework for self-aware learning. Machine learning, 82(3):399 443.
Lu, X., Van Roy, B., Dwaracherla, V., Tbrahimi, M., Osband, I., and Wen, Z. (2021). Reinforcement learning, bit by bit. arXiv preprint 2r0i::1103. 203.0000.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al, (2015). Human-leve Control through Deep Reinforcement Learning. Nature, 518(7540):529 533.
10

---

Murphy, K. P. (2012). Muchine Learning: A Probubilistic Perspective. MIT Press.
Nado, Z., Band, N., Collier, M., Djolonga, J., Dusenberry, M., Farquhar, S., Filos, A., Havasi, M., Jenatton, R., Jerfel, G., Liu, J., Mariet, Z., Nixon, J., Padly,, S., Ren, J., Rudner, T., Wen, Y., Wenzel, F., Murphy, K., Sculley, D., Lakshminarayanan, B., Snock, J., Gal, Y., and Tran, D. (2021). Uncertainty Baselines: Benchmarks for uncertainty & robustness deep learning. arXiv preprint or 106.0415.
Osband, I. (2016). Risk versus uncertainly in deep learning; Bayes, bootstrap and the dangers of dropout. In NIPS Workshop Bayesian Deep Learwing, volume 192.
Osband, 1., Aslanides, J., and Cassirer, A. (2018). Randomized prior functions for deep reinforcement learning. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Aduances in Neural Information Processing Systems 81, pages 8617-8629. Curran Associates, Inc.
Oshand, I. and Van Roy, B. (2015). Bootstrapped Thompson sampling and deep exploration. arXiv preprint arXiv:1507.00000.
Osband, I., Wen, Z., Asghari, M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B. (2021).
Epistemic neural networks. arXiv preprint arXio:2107.08924;
Osband, I,, Wen, Z., Asghari, S. M., Dwaracherla, V., Lu, X., and Van Roy, B. (2022). Evaluating high-order predictive distributions in deep learning.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R.,, Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine learning in Python, Journal of Machine Learning Reseanch, 12:2825-2888 2830.
Rasmussen, C. F. (2003). Gaussian processes in machine learning. In Summer school on machine learning, pages 63 63-71. Springer.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2018). Do cifar-10 classifiers generalize to cifar-10? Xiv preprint or 1rXiv:1866.000 00451.
Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International conference machine learning, pages 1530- 1538. PMI.R.
Riquelme, C., Tucker, G., and Snoek, J. (2018). Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arxi 10::::180..91111 09127.
Russo, D. and Lou, J. (2016). Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics, pages 1232-1240 PMLR
Russo, D. J., Van Roy, B., Kazeroumi, A., Osband, I., and Wen, (2018). A tutorial on Thompson sampling. Found. Trends Mach. Learn., 11(1):1 96.
Schölkopf, and Smola, A. (2018). Learning with kernels: Support vector machines, regularization, optimization, and beyond. MIT press.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016). Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):181 489.
Sun, S., Zhang, G., Shi, J., and Grosse, R. (2019). Functional variational Bayesian neural networks. Xid preprint arXie:1903.055 197700977....
Thompson, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-22
Wang, C., Sun, S., and Grosse, R. (2021). Beyond marginal uncertainty: How accurately can Bayesian regression models estimate posterior predictive correlations? In International Conference on Artificial Intelligence and Statistics, pages 2476 2484. PMLR.
Welling, M. and Teh, Y. W. (2011). Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688. Citeseer.
11

---

Wen, Z., Osband, I,, Qin, C., Lu, X., Ibrahimi, M., Dwaracherla, V., Asghari, M., and Van Roy, B. (2022). From predictions to decisions: The importance of joint predictive distributions.
Wilson, A. G. and Izmailov, P. (2020). Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint wrXiv28022.0899
Wilson, A. G., Iamailov, P., Hoffman, M. D., Gal, Y., Li, Y., Pradier, M. F., Vikram, S., Foong, A., Lotfi, S., and Farquhar, S. (2021). Evaluating approximate inference in Bayesian deep learning.
Woodbury, M. A. (1950). Inverting modified matrices. Statistical Research Group.
12


Reviews:
[{"review_id": "bjGKXG2L2nK", "reviewer": "Reviewer_4Cxf", "summary": "This paper proposes a benchmark task to evaluate Bayesian deep learning agents on the calibration of their joint predictions rather than just marginal predictions. This benchmark generates generative processes via randomly initialized neural networks and evaluates agents based on their KL-divergence from the true generative distribution over joint predictions. A major contribution of this work is to show that in marginal predictions, none of the Bayesian methods perform significantly better than a standard MLP. However, in joint predictions, several Bayesian deep learning methods underperform. The paper also shows that this performance is correlated with performance on a neural bandit task using Thompson sampling. Finally, the paper shows that the results are fairly robust to the choice of generative model.", "questions": "1. How does this work compare to (Wang et al. 2021)?\n2. Why do the different agents perform differently in the benchmark tasks? Is there some takeaway that the benchmark can give us that demonstrates its scientific value?", "limitations": "The paper talks briefly about limitations, such as the fact that the paper is meant as a sanity check rather than a grand challenge.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Originality:  Many prior works in Bayesian deep learning include benchmarks against other methods, however they do not measure the performance of *joint* predictions. The paper mentions that a prior work (Wang et al. 2021) has a similar motivation of moving beyond measuring the quality of marginal uncertainty estimation, but that their paper shows the potential of measuring joint likelihoods. I would like to see a bit more of a comparison between the methods in that paper and in this one, and how the results compare.\n2. Quality: The paper is motivated well and the experimental framework proposed makes sense. I would like to see some more intuition of *why* certain agents perform better than others in this benchmark in the main-text.\n3. Clarity: The paper is very well written and motivated. The plots are easy to understand.\n4. Significance: This paper gives an important and accessible benchmark for evaluating the quality of Bayesian agents.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NYES4pd9qAF", "reviewer": "Reviewer_Ayr1", "summary": "This paper proposed a new benchmark to evaluate joint predictions (with focus on Bayesian deep learning) by generating random classification problems using a generative process. While previous works largely focused on marginal prediction, this work mainly focused on joint predictions considering its importance in the decision making process. In the paper, 7 different agents (e.g. mlp, ensemble, dropout, etc.) are included. Also, the authors released the code for this paper.", "questions": "The paper keeps switching between deep learning and bayesian deep learning in the text (e.g. 'The Neural Testbed as a simple and clear benchmark for evaluating the quality of joint predictions in deep learning systems. ', 'we compare benchmark approaches to Bayesian deep learning', '..inform the field’s wider efforts in deep learning..', '...less transferable to general deep learning research'). While I can kinda guess why this happened, I am not sure what is the main focus of this paper? Will it be a general framework that can be used in any problem settings regardless of its solutions or it can only be suitable to evaluate bayesian approaches? It seems the latter is the case in this paper.\n\nWhere is the relevant text for Figure 3? it is neither used in the text nor even has a detailed caption. ", "limitations": "The paper briefly discusses the limitations of their method and future direction in the conclusion. Societal impact is not relevant in this paper. ", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The direction of this paper is very interesting and as the authors rightfully said joint predictions as an essential piece in decision making processes. The paper idea is well-motivated and figure 1 clearly shows why this question should be studied. That being said, apart from the first few sections, (unfortunately) the paper lacks a consistent story and doesn't provide details in the main text about the proposed approach, setting, how it works, etc, For instance, one needs to keep checking the appendix to understand main ideas in the paper as lots of important details are relegated to the appendix. In addition, experiments or evaluation scenarios are not clear at all. Take Figure 2 as an example, it is not clear what is the scenario, what is the task?, what is the experiment settings?, why joint prediction is relevant in this example? The same thing is true about experiments in sections 4.2, 4.1, etc. Importantly, their proposed method is not explained well and not clear e.g. how it generates random classification tasks? how data will be generated? etc. The fact that this is a benchmark paper and lacks details, is very concerning. Benchmark paper should be very generous on details and be super clear about experiment setups and procedures, unfortunately neither of them is the case in this paper. The paper needs significant revision and rewriting.\n\nMoreover, it seems this paper is more of a conceptional benchmark rather than a real world one as all experiments/settings are rather synthetic or toy examples. Take robustness study as an example. It only considers a simple MLP with few hidden units, not sure how this result can help with different settings/architectures and what would be the conclusion here? \n\n\nOne of the main and natural applications of this work is to be used in sequential decision making tasks. Although the paper studies a simple and toy bandit case, realistic scenarios could have been utilized in this paper. For example, the paper could have studied the issue of safety in sequential decision making tasks, etc.\n\nIn summary, the proposed method is not clear and the paper doesn't provide a well defined picture about their method, setting, and the experiments. It is not ready in its current format and requires significant revision.  \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "98D7uBVUJg", "reviewer": "Reviewer_Yeyc", "summary": "The paper proposes an open source benchmark for evaluating uncertainty quantification methods. The testbed enables evaluations using marginal and joint predictions by controlling the data generation process using neural network based simulated datasets and proposes to use KL divergence as the evaluation metric. The paper also uses the testbed to provide insights on the performance of Bayesian deep learning approaches. Their analyses show that popular techniques in this category, including Monte Carlo dropout, ensembling and Bayes by Backprop do not outperform point estimates in marginal and joint predictions. They show that the performance of joint estimates is correlated with performance in neural bandits through regret analyses.\n", "questions": "- I recommend adding a discussion on the limitations of the neural testbed. From the manuscript, it is hard to understand whether the assumptions done for analyses are a limitation of the neural testbed.\n- Along the same lines, even though the KL divergence metric is proposed as the core metric, it would be great to include a discussion on other commonly used metrics (like calibration and likelihood based metrics) and how neural testbed can support these.\n- It would be good to explain how the users can define the environment epsilon distribution since it constitutes the main backbone of the data generation.\n- The results reported in Section 4.1: It is interesting to see that probabilistic approaches do not outperform MLP in terms of calibration error, which is counter intuitive given the literature. The fact that the generative model and the model architecture match provides an explanation but it would be great to see some results (at least in an appendix) where this is not the case.\n- It might be good to extend the robustness analysis to include number of layers, which can help provide a more complete picture.\n- The insights on SGMCMC has significant parallels with Izmailov et al. (2021)’s findings and it might be good to include these in the discussion.\n\nMinor comments:\n- There is a typo in Algorithm 1. Add a space between “environment” and “likelihood”.\n- Line 241. I recommend adding a discussion on the gist of how prior functions enable the benefit of ensemble+ to ensemble in the manuscript.\n- Line 287. Missing a verb between “we will also” and “these agents”\n\nIzmailov, P., Vikram, S., Hoffman, M.D. and Wilson, A.G.G., 2021, July. What are Bayesian neural network posteriors really like?. In International conference on machine learning (pp. 4629-4640). PMLR.\n", "limitations": "The main limitation of the paper is the toy nature of the datasets and models considered in this study. I would recommend adding a discussion on how neural testbed scales to more realistic settings. Along similar lines, the manuscript can benefit from a discussion on the general limitations of neural testbed. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The methods are not new but the paper introduces a new open source framework to provide a unified approach to evaluate the existing methods. Furthermore, the extensions of such evaluations to joint predictions is quite novel to my knowledge.\n\nQuality: The paper provides a framework for evaluation and provides extensive empirical analyses with this framework. The proposed framework is technically sound.\n\nClarity: The paper is well written and organized.\n\nSignificance: This is an important open source contribution that can help standardize the uncertainty quantification approaches, which is a bit scattered at this point in my opinion. Therefore, I believe this is an important contribution to the field. However, the experiments are conducted in rather small data environments with toy MLP models, which is limiting its evaluation on how it can help advance the state of the art.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8XoNAuW5XKl", "reviewer": "Reviewer_CzHp", "summary": "This paper introduces an open-source benchmark called \"Neural Testbed\" to evaluate the quality of neural networks' predictive distribution. Instead of using a finite collection of datasets, the Neural Testbed adopts a synthetic data generation process to avoid overfitting. Besides, this paper advocates evaluating the quality of prediction on joint prediction rather than merely on marginal distribution. The authors find that previous Bayesian inference methods that give an excellent marginal prediction may fail to provide an accurate joint prediction, and the quality of the joint prediction is closely related to the performance on sequential decision tasks.", "questions": "4.1 Performance in marginal prediction\n\n- Why do you choose aggregated KL as the objective? Does it imply there is a trade-off between joint prediction and marginal distribution? Does it mean a model that is good at $\\mathbf{d}_{\\mathrm{KL}}^{10}$ prediction may not perform well on marginal distribution or joint distribution with 100 examples?\n- Have you verified the two hypotheses for the discrepancy? What results do you get if you tune for ECE? What happens if the architecture does not match?\n- Why does the ensemble model achieve worse accuracy than the simple MLP model?\n- Why do agents with better $\\mathbf{d}_{\\mathrm{KL}}^{10}$ have worse ECE performance?\n\n4.2 Performance beyond marginals\n- Why do some agents have better joint prediction ability, such as hypermodel, ensemble+, sgmcmc? And why do ensemble, dropout, bbb fail to do so? Figure 12 shows that diversity may be a key factor. Does this hold for the other methods? What are other hypotheses?\n\n5.2 Sequential Decision\n- It is interesting to see that the quality of joint distribution and the performance of a sequence decision task is highly correlated. But why does the sequential decision problem rely on joint prediction quality? It seems that Thompson sampling does not explicitly exploit the joint prediction. What's the explanation for this observation?\n\nMisc\n- Line 287: \"we will also these agents over alternative environments,\" missing a verb.\n- In tutorial.ipynb, it would be easier for the reader if some of the key concepts like \"ENN\" were explained. This abbreviation does not appear anywhere in the main text or appendix.", "limitations": "This paper discusses its limitation and points out some future research directions.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- This paper provides a simple benchmark to evaluate the quality of distribution, which can benefit the community.\n- This paper points out the importance of the quality of joint distribution. It observes a positive correlation between the joint distribution quality and the performance of sequential decision-making problems.\n- This paper is well-written and easy to follow. All implementation details can be found in the provided codebase.\n\nWeakness\n- The synthetic data generation processes and model architecture are simple. It's unclear whether the results still hold on large-scale and more challenging datasets or more complex deep models.\n- Lack of explanations for some key observations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept