Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces an open-source benchmark called "Neural Testbed" to evaluate the quality of neural networks' predictive distribution. Instead of using a finite collection of datasets, the Neural Testbed adopts a synthetic data generation process to avoid overfitting. Besides, this paper advocates evaluating the quality of prediction on joint prediction rather than merely on marginal distribution. The authors find that previous Bayesian inference methods that give an excellent marginal prediction may fail to provide an accurate joint prediction, and the quality of the joint prediction is closely related to the performance on sequential decision tasks.
strengths_and_weaknesses: Strength
- This paper provides a simple benchmark to evaluate the quality of distribution, which can benefit the community.
- This paper points out the importance of the quality of joint distribution. It observes a positive correlation between the joint distribution quality and the performance of sequential decision-making problems.
- This paper is well-written and easy to follow. All implementation details can be found in the provided codebase.

Weakness
- The synthetic data generation processes and model architecture are simple. It's unclear whether the results still hold on large-scale and more challenging datasets or more complex deep models.
- Lack of explanations for some key observations.
questions: 4.1 Performance in marginal prediction

- Why do you choose aggregated KL as the objective? Does it imply there is a trade-off between joint prediction and marginal distribution? Does it mean a model that is good at $\mathbf{d}_{\mathrm{KL}}^{10}$ prediction may not perform well on marginal distribution or joint distribution with 100 examples?
- Have you verified the two hypotheses for the discrepancy? What results do you get if you tune for ECE? What happens if the architecture does not match?
- Why does the ensemble model achieve worse accuracy than the simple MLP model?
- Why do agents with better $\mathbf{d}_{\mathrm{KL}}^{10}$ have worse ECE performance?

4.2 Performance beyond marginals
- Why do some agents have better joint prediction ability, such as hypermodel, ensemble+, sgmcmc? And why do ensemble, dropout, bbb fail to do so? Figure 12 shows that diversity may be a key factor. Does this hold for the other methods? What are other hypotheses?

5.2 Sequential Decision
- It is interesting to see that the quality of joint distribution and the performance of a sequence decision task is highly correlated. But why does the sequential decision problem rely on joint prediction quality? It seems that Thompson sampling does not explicitly exploit the joint prediction. What's the explanation for this observation?

Misc
- Line 287: "we will also these agents over alternative environments," missing a verb.
- In tutorial.ipynb, it would be easier for the reader if some of the key concepts like "ENN" were explained. This abbreviation does not appear anywhere in the main text or appendix.
limitations: This paper discusses its limitation and points out some future research directions.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This paper proposed a new benchmark to evaluate joint predictions (with focus on Bayesian deep learning) by generating random classification problems using a generative process. While previous works largely focused on marginal prediction, this work mainly focused on joint predictions considering its importance in the decision making process. In the paper, 7 different agents (e.g. mlp, ensemble, dropout, etc.) are included. Also, the authors released the code for this paper.
strengths_and_weaknesses: The direction of this paper is very interesting and as the authors rightfully said joint predictions as an essential piece in decision making processes. The paper idea is well-motivated and figure 1 clearly shows why this question should be studied. That being said, apart from the first few sections, (unfortunately) the paper lacks a consistent story and doesn't provide details in the main text about the proposed approach, setting, how it works, etc, For instance, one needs to keep checking the appendix to understand main ideas in the paper as lots of important details are relegated to the appendix. In addition, experiments or evaluation scenarios are not clear at all. Take Figure 2 as an example, it is not clear what is the scenario, what is the task?, what is the experiment settings?, why joint prediction is relevant in this example? The same thing is true about experiments in sections 4.2, 4.1, etc. Importantly, their proposed method is not explained well and not clear e.g. how it generates random classification tasks? how data will be generated? etc. The fact that this is a benchmark paper and lacks details, is very concerning. Benchmark paper should be very generous on details and be super clear about experiment setups and procedures, unfortunately neither of them is the case in this paper. The paper needs significant revision and rewriting.

Moreover, it seems this paper is more of a conceptional benchmark rather than a real world one as all experiments/settings are rather synthetic or toy examples. Take robustness study as an example. It only considers a simple MLP with few hidden units, not sure how this result can help with different settings/architectures and what would be the conclusion here? 


One of the main and natural applications of this work is to be used in sequential decision making tasks. Although the paper studies a simple and toy bandit case, realistic scenarios could have been utilized in this paper. For example, the paper could have studied the issue of safety in sequential decision making tasks, etc.

In summary, the proposed method is not clear and the paper doesn't provide a well defined picture about their method, setting, and the experiments. It is not ready in its current format and requires significant revision.  



questions: The paper keeps switching between deep learning and bayesian deep learning in the text (e.g. 'The Neural Testbed as a simple and clear benchmark for evaluating the quality of joint predictions in deep learning systems. ', 'we compare benchmark approaches to Bayesian deep learning', '..inform the field’s wider efforts in deep learning..', '...less transferable to general deep learning research'). While I can kinda guess why this happened, I am not sure what is the main focus of this paper? Will it be a general framework that can be used in any problem settings regardless of its solutions or it can only be suitable to evaluate bayesian approaches? It seems the latter is the case in this paper.

Where is the relevant text for Figure 3? it is neither used in the text nor even has a detailed caption. 
limitations: The paper briefly discusses the limitations of their method and future direction in the conclusion. Societal impact is not relevant in this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes an open source benchmark for evaluating uncertainty quantification methods. The testbed enables evaluations using marginal and joint predictions by controlling the data generation process using neural network based simulated datasets and proposes to use KL divergence as the evaluation metric. The paper also uses the testbed to provide insights on the performance of Bayesian deep learning approaches. Their analyses show that popular techniques in this category, including Monte Carlo dropout, ensembling and Bayes by Backprop do not outperform point estimates in marginal and joint predictions. They show that the performance of joint estimates is correlated with performance in neural bandits through regret analyses.

strengths_and_weaknesses: Originality: The methods are not new but the paper introduces a new open source framework to provide a unified approach to evaluate the existing methods. Furthermore, the extensions of such evaluations to joint predictions is quite novel to my knowledge.

Quality: The paper provides a framework for evaluation and provides extensive empirical analyses with this framework. The proposed framework is technically sound.

Clarity: The paper is well written and organized.

Significance: This is an important open source contribution that can help standardize the uncertainty quantification approaches, which is a bit scattered at this point in my opinion. Therefore, I believe this is an important contribution to the field. However, the experiments are conducted in rather small data environments with toy MLP models, which is limiting its evaluation on how it can help advance the state of the art.

questions: - I recommend adding a discussion on the limitations of the neural testbed. From the manuscript, it is hard to understand whether the assumptions done for analyses are a limitation of the neural testbed.
- Along the same lines, even though the KL divergence metric is proposed as the core metric, it would be great to include a discussion on other commonly used metrics (like calibration and likelihood based metrics) and how neural testbed can support these.
- It would be good to explain how the users can define the environment epsilon distribution since it constitutes the main backbone of the data generation.
- The results reported in Section 4.1: It is interesting to see that probabilistic approaches do not outperform MLP in terms of calibration error, which is counter intuitive given the literature. The fact that the generative model and the model architecture match provides an explanation but it would be great to see some results (at least in an appendix) where this is not the case.
- It might be good to extend the robustness analysis to include number of layers, which can help provide a more complete picture.
- The insights on SGMCMC has significant parallels with Izmailov et al. (2021)’s findings and it might be good to include these in the discussion.

Minor comments:
- There is a typo in Algorithm 1. Add a space between “environment” and “likelihood”.
- Line 241. I recommend adding a discussion on the gist of how prior functions enable the benefit of ensemble+ to ensemble in the manuscript.
- Line 287. Missing a verb between “we will also” and “these agents”

Izmailov, P., Vikram, S., Hoffman, M.D. and Wilson, A.G.G., 2021, July. What are Bayesian neural network posteriors really like?. In International conference on machine learning (pp. 4629-4640). PMLR.

limitations: The main limitation of the paper is the toy nature of the datasets and models considered in this study. I would recommend adding a discussion on how neural testbed scales to more realistic settings. Along similar lines, the manuscript can benefit from a discussion on the general limitations of neural testbed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a benchmark task to evaluate Bayesian deep learning agents on the calibration of their joint predictions rather than just marginal predictions. This benchmark generates generative processes via randomly initialized neural networks and evaluates agents based on their KL-divergence from the true generative distribution over joint predictions. A major contribution of this work is to show that in marginal predictions, none of the Bayesian methods perform significantly better than a standard MLP. However, in joint predictions, several Bayesian deep learning methods underperform. The paper also shows that this performance is correlated with performance on a neural bandit task using Thompson sampling. Finally, the paper shows that the results are fairly robust to the choice of generative model.
strengths_and_weaknesses: 1. Originality:  Many prior works in Bayesian deep learning include benchmarks against other methods, however they do not measure the performance of *joint* predictions. The paper mentions that a prior work (Wang et al. 2021) has a similar motivation of moving beyond measuring the quality of marginal uncertainty estimation, but that their paper shows the potential of measuring joint likelihoods. I would like to see a bit more of a comparison between the methods in that paper and in this one, and how the results compare.
2. Quality: The paper is motivated well and the experimental framework proposed makes sense. I would like to see some more intuition of *why* certain agents perform better than others in this benchmark in the main-text.
3. Clarity: The paper is very well written and motivated. The plots are easy to understand.
4. Significance: This paper gives an important and accessible benchmark for evaluating the quality of Bayesian agents.
questions: 1. How does this work compare to (Wang et al. 2021)?
2. Why do the different agents perform differently in the benchmark tasks? Is there some takeaway that the benchmark can give us that demonstrates its scientific value?
limitations: The paper talks briefly about limitations, such as the fact that the paper is meant as a sanity check rather than a grand challenge.
review_title: null

Meta-review: metareview: Getting a reasonable estimation of joint predictions is crucial for many uncertainty estimation tasks. The paper proposed a set of benchmarks for predicting joint probabilities of the outputs over a few input examples. The proposed synthetic tasks are easy to deploy and test on most Bayesian methods, including Bayesian neural networks. 

Uncertainty estimation is one of the fundamental challenges for modern machine learning algorithms. Many downstream application areas in reinforcement learning, active learning, and safety require a model to assess its prediction confidence. Yet, unlike the standard classification tasks, there is a lack of benchmark datasets to evaluate the performance of uncertainty methods. The strength of this paper is:
1) Develop a suite of benchmarks, although synthetic and toyish, to allow a quantitative study of the joint prediction of the existing machine learning methods. The proposed benchmark allows researchers to study uncertainty estimation without invoking any downstream application in RL or active learning. 
2) The work bridge the gap between the benchmarks on marginal predictions, such as Riquelme et al. "Deep Bayesian bandits showdown" and heavy machinery of exploration tasks in RL. 
The weakness of the current submission is a lack of clarity in the current writing, as pointed out by one of the reviewers. Many experimental hyperparameters are omitted from the main text, which would help the readers understand the benchmark details and design choices. Also, there is a glaring limitation of the benchmarks' simplicity and whether the generative model choice could generalize to high-dimensional problems. 

Given the scarcity of other benchmarks in the uncertainty estimation tasks, the strength outweighs the weakness of this paper. 
confidence: Certain
Decision: Accept