Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a modification to the neural ratio estimation (NRE) algorithm in the form of a regularization penalty to avoid overconfident posterior distributions. The authors support their claim by introducing a family of likelihood-to-evidence ratio classifiers which are more conservative than the optimal Bayes classifier, in expectation. By posing an explicit penalty on overconfidence, the authors argue that the simulation-based inference using the NRE algorithm would be more reliable and not leading to false inference. The authors provide empirical results over several datasets, showing how the balanced neural ratio estimation (BNRE) is indeed experimentally conservative and also converges ultimately to the same posterior as the NRE method for large simulation budgets. 
strengths_and_weaknesses: The paper is well written and well presented, with legible figures and thoughtful considerations. I have checked the proofs of Theorems 1 and 2 and they look correct to me. The validation is extensive and well thought of.

The minor weakness that I can see are (a) the applicability to the NRE algorithm only (although other algorithms are mentioned in the discussion) and (b) examples of over-confident simulation-based inference algorithms leading a research direction astray.
For (a), as shown by [1], there does not seem to exist a universally "best" algorithm in SBI, so potentially extending the same logic to more algorithms would be very valuable for the community.
For (b), I appreciate the empirical results pointed out by [2]; my only point is that a practical example of a "failure case" -- or even a "if they used this overconfident SBI algorithm in this real world study it would have been a problem" type-of-argument -- would make this paper much more convincing.

Overall, I believe the paper proposes an interesting idea and it is still worth flagging in the community, although I believe that it is not as compelling as it could be.

[1] Benchmarking Simulation-Based Inference, Lueckmann et al, AISTATS 2021
[2] Averting A Crisis In Simulation-Based Inference, Hermans et al, 2021
questions: I have no main questions for the authors, apart from any feedback on my comments above.
limitations: The one (minor) limitation I would like to flag is the various benchmarks description being relegated to Appendix C.
I believe it would be nice for the reader to appreciate the range of the benchmarks included in the paper, with some of the simulators being not necessarily trivial. Without a description, even a quick one, a non-expert reader might see these benchmarks as all simple toy-examples.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This study builds on a previous state-of-the-art method for simulation-based inference (i.e., NRE), and proposes an improvement that aims at avoiding the usual overconfidence of NRE posterior estimates (an issue that might be common to other simulation-based inference methods). The study provides empirical results on several benchmark problems and shows that the new method (BNRE) clearly improves in reliability compared to NRE as assessed by several metrics.
strengths_and_weaknesses: ### Originality

To the best of my knowledge, this is one of two studies that specifically addresses the issue of overconfidence of simulation-based inference algorithms, the other one being Hermans et al. 2021 (cited by the authors). While Hermans et al. diagnose this problem on several simulation-based inference methods and provide a partial solution to this issue using ensemble approaches, the solution provided by this study is novel.


### Quality

The paper is technically sound, and most claims are backed by empirical evidence. However, there is one claim that I would appreciate the authors to illustrate: "We found  $\lambda$ = 100 to perform well across all benchmarks, which again, is supported by Figure 5." I would recommend the authors to provide a figure (in appendix) illustrating this, even if for a subset of the benchmark problems. 


### Clarity

The manuscript is clearly written, providing enough information to understand the technical contribution and empirical results, while appropriately putting the contributions in the context of previous work. A few small comments:

-page 4, line 119, "does not modify the global optimum". This is a bit imprecise and might lead to confusion, so I would suggest the authors to expand on it;

-page 4, line 122, "[...] result in increasingly conservative [...]". By "increasingly", I assume the authors meant with increasing $\lambda$. As this is not clear, I would suggest the authors to drop the word "increasingly";

-page 4, line 123, the word "Ideally" wrongly suggests that the expression (6) only follows from the previous inequality in the ideal case, which is not what the authors meant. Dropping the word "Ideally" would solve the issue;

-page 7, line 186, for clarity, the notation for nominal parameter ($\vartheta^*$) should be introduced close to the expected squared error over the approximate posterior.


### Significance

The technical development and empirical improvement of BNRE over NRE will be of interest to the ML community, in particular to the simulation-based inference community. Notably, the technical contribution could directly inspire the improvement of other methods for simulation-based inference.
questions: As pointed out above, I believe it would be beneficial to the manuscript if evidence was provided for the statement that $\lambda$ = 100 is a good choice across benchmark problems. An appendix figure would be enough.

limitations: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Simulation-based inference enables Bayesian parameter inference for simulation-based models for which the likelihood cannot be evaluated efficiently, by using simulations from the model as training data. Over the last years there has been developed a suite of neural network-based SBI methods that estimate the posterior distribution directly (neural posterior estimation, NPE), or target the likelihood (neural likelihood estimation, NLE) or the likelihood ratio (neural ratio estimation, NRE) to then obtain posterior samples via MCMC. By design, SBI is usually applied in scenarios where the ground-truth posterior is not known, and with limited training data, e.g., because model simulations are expensive. Thus, the approximated posterior (samples) may be inaccurate for a given problem. Preliminary results in the field indicate that posteriors tend to be overconfident, e.g., they tend to underestimate uncertainties.
This submission introduces a variant of NRE, called Balanced NRE (BNRE), with the goal to make posterior estimates more conservative in the low simulation budget regime. Like NRE, BNRE trains a classifier in order to approximate the likelihood-to-evidence ratio, to then perform MCMC. The authors introduce the concept of balanced classifier and introduce a modified loss function to train BNRE such it is a balanced classifier. Furthermore, they show that the optimal Bayesian classifier is balanced, i.e., that in the limit of unlimited training data BNRE will eventually converge to be NRE and obtain the exact posterior. They perform experiments with tractable benchmarking problems to show that BNRE tends to be more conservative than NRE in the low simulation budgets, i.e., show larger posterior variance. They conclude that BNRE provides a new SBI method suitable to obtain conservative posterior estimates. 
strengths_and_weaknesses: **Originality:** The reliability of posterior estimates obtained with SBI is a timely and very important topic which I am happy to see addressed here. The paper picks one of three main SBI approaches, NRE, and suggests a variant with the goal to make it more conservative in the low simulation budget regimes. The introduction of balanced classifiers is novel and useful for the context of NRE and, as mentioned in the conclusion, it may indeed be applicable to NPE as well.
    
**Quality**: The introduction of balanced classifiers and BNRE as a variant of NRE with an additional term in the loss function is technically sound and a novelty. However, the experimental evaluation of BNRE is limited and less convincing for several reasons:
- Choice of benchmarks: The set of benchmark tasks is limited, it contains rather low-dimensional tasks most of which (all? depending on the variants used, exact definitions of the tasks missing) are tractable via MCMC. It would be more instructive to show one or two tractable examples with reference posteriors (obtained via MCMC or analytically) to study the properties of BNRE in depth, and to then show one or two additional “real-world” examples to demonstrate the use of BNRE in practice and to show how it compares to NRE in practice. I see the current results merely as a first study showing preliminary results.
- Choice of metrics: The results of the paper are mainly based on one single metric measuring the “confidence” of the posterior—the expected posterior coverage. This metric was introduced in a recent preprint by Hermans et al [1] which, again, addresses the very important topic of SBI reliability, but makes relatively strong claims about the general reliability of SBI methods based solely on the expected coverage metric, and was not successfully peer-reviewed by the community yet. The evaluation of BNRE would become more informative if additional metrics were considered. One common metric for investigating the posterior coverage and potential biases is simulation-based calibration (SBC). The authors showed in the appendix that in theory expected coverage is a special case of SBC, however, it would be useful to see the practical similarities in the results. Importantly, SBC is able to detect not only miscalibrated uncertainties like over- or underdispersion, but also positive and negative biases of the posterior estimates. Thus, it is additionally suited for evaluating BNRE given that BNRE tends to introduce a bias in the low simulation budget regimes. In addition to SBC which provides a global marginal coverage test across all x, there is a new method for performing a local coverage test for specific observation (Zhao et al. [https://proceedings.mlr.press/v161/zhao21b.html](https://proceedings.mlr.press/v161/zhao21b.html)), which would be useful to add as a metric as well, e.g., in the “real-world” example. 
- For the tractable tasks with known posteriors it would additionally be useful to calculate actual biases, and dispersion with respect to the reference posteriors. 
- For the real-world examples additional checks, e.g., prior and posterior predictive checks would be instructive to show the practical effect of conservative posteriors and difference of BNRE to NRE.  
- Finally, a comparison to other established SBI methods like NPE and NLE would be illustrative. They are readily available in open-source software packages so that adding them to the benchmark would not result in large algorithmic or implementation overhead.

Overall I think, given the importance of this topic, it is essential to evaluate a new method on the available and established metrics (especially when additionally introducing a new metric).
    
**Clarity**: The paper is very well and clearly written, it was straight forward to follow the line of arguments and the presented results. Regarding the appearance of the figures I have the following comments:
- A visual explanation of the approach as Figure 0 would be nice to have
- For most of the subplots in Figure 1 it is hard to see any differences between the methods, a different visualization, e.g., on the log-scale would be more illustrative
- For Figure 2 it would be good to show standard error of the mean as error bars, given that 5 repetitions were performed
- For Figure 3 it would better to show standard error of the mean instead of the standard deviation (minor)
    
**Significance**: As mentioned above, I find the general topic and the approach of BNRE important for SBI research. However, the experiments and results presented here appear to me as preliminary with no direct insights or consequences for other researchers or practitioners (yet). In theory, the BNRE approach makes sense, however, in practice (as indicated by the results presented here) I do not see an advantage over NRE. The results **do** show that on the benchmark tasks presented here, BNRE leads to broader posteriors than NRE. However, the additional results in Figure 3 and 4 also indicate that it comes with a larger bias (which makes sense due to the additional term in the loss). While the bias reduces with increasing simulation budget, but even for the same budget, it seems that BNRE performs worse than NRE. Furthermore, BNRE comes with the additional hyperparameter choice for $\lambda$. The authors make a suggestion for a default value of lambda based on extensive parameter search on the benchmark tasks, however, it is not clear how this choice would extrapolate to real-world scenarios, e.g., for models with more parameters. Thus, from the practitioners point of view, it would not makes sense to use BNRE as of yet.
questions: As outlined in detail above, I think this study would benefit from additional tasks and from additional metrics to study the effect of BNRE in more detail. For the benchmarking tasks I suggest to use one that is fully tractable so that it can be studied in detail, and one with real-world character that is challenging, e.g., high-dimensional, so that one can see the practical relevance of BNRE. For metrics I suggest to additionally use SBC, local coverage tests and analytical over and underdispersion and bias (for the tractable tasks). 


limitations: The authors transparently addressed limitations of their approach, given the scope of the results presented in the paper (see comments above). 

Update after rebuttal: The authors addressed all of my questions and the revised version of the submission now adequately discusses limitations of the approach to the low-dimensional parameter regime. I updated my score accordingly.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces and discusses a regularization approach for neural ratio estimation that is based on enforcing a balancing condition of the classifier (via an additional regularisation term). It is shown theoretically that on average (expectation) this results in a more conservative classifier.
In several experiments it is shown that the approach indeed results in a more conservative posterior. 
strengths_and_weaknesses: __Strengths__
- Ensuring that posteriors derived by deep learning algorithms are valid is very important.
- The use of the balancing condition as an additional measure is an interesting idea.  
- The method is easy to implement.
- The experiments show that the posterior tends to be more conservative when the regularizer is applied.
- The induced bias and variance diminish with increasing sample size
- Little to no exaggeration. The authors are very honest about the advantages and disadvantages of their method which makes the contribution all the more valuable to the field. 

__Weaknesses__
- The penalty term does result in a worse calibration in most examples. While the posterior is more conservative the calibration is often worse, in particular when the number of samples is low. Especially for models where simulation is costly this might be an issue. 
- While I agree that conservative posteriors are preferable, calibration is not a binary issue. Overly conservative posteriors might be unable to identify a parameter to a reasonable degree. In the Weinberg, SIR and Lotka Volterra models, a relatively accurate posterior is traded in for a mis-calibrated conservative posterior. I think the trade-off is important.  
- There is a broad literature in statistics considering model (over-)confidence [A - C] which I think should be mentioned. In particular, power posteriors are quite popular to adjust the posterior in order to account for discrepancies between a model and real data. Since NRE does not estimate the likelihood directly, it might be necessary to use the presented approach instead of power posteriors, but a discussion should be added.

> Recently, Dellaporta et al. [26] further improved GBI by combining 
- The cited work has advantages and disadvantaged but it's not an "improved" version of GBI, it's a variant. In particular, it best applies to IID data. When using time-series data as many of the models in this paper [D] would be a more fitting GBI procedure. 

> M/G/1, originally introduced by Papamakarios et al. [4],
- Papamakarios et al. however reference [E] for the M/G/1 model. This should be amended. 


__References__

[A] - Peter Grünwald. Safe learning: bridging the gap between Bayes, MDL and statistical learning
theory via empirical convexity. In Proceedings of the 24th Annual Conference on Learning
Theory, pages 397–420, 2011.

[B] - Peter Grünwald. The safe Bayesian. In International Conference on Algorithmic Learning
Theory, pages 169–183. Springer, 2012.

[C] - Chris Holmes and Stephen Walker. Assigning a value to a power likelihood in a general
bayesian model. Biometrika, 104(2):497–503, 2017

[D] - Dyer, Joel, Patrick Cannon, and Sebastian M. Schmon. "Approximate Bayesian Computation with Path Signatures." arXiv preprint arXiv:2106.12555 (2021).

[E] -  A. Y. Shestopaloff and R. M. Neal. On Bayesian inference for the M/G/1 queue with efficient MCMC
sampling. arXiv:1401.5548, 2014.
questions: - Could you please write down the definition of a classifier? I assume you mean something like $d: X \rightarrow [0, 1]$
- The discussion around equation (6) is confusing. You say that $\hat{p}(\theta|x) < {p}(\theta|x)$ whenever the $\hat{d}(\theta,x) < {d}(\theta,x)$, but clearly $\hat{p}(\theta|x) < {p}(\theta|x)$ can't hold for all $\theta$ since we are considering densities. Hence, the crucial aspects of whether the density is more conservative is _for which_ $\theta$ one has $\hat{p}(\theta|x) < {p}(\theta|x)$ and that seems to be missing. 
- Generally the model details are a bit short. For example, you reference Lotka and Volterra indicating that the underlying model is a deterministic ODE following the Lotka-Volterra equations. However, commonly people in SBI work with the stochastic version, describing a Markov jump process. I would be nice to have more information here. 
- I am not so sure about the expected coverage as a special case of SBC. The internal consistency of the joint distribution that you're relying on is something that is remarked even in the SBC paper to be known before. The devil is in the detail and from my point of view SBC is about an actionable algorithm, which is more than just the expectation that you allude to in Appendix A. In particular, the actual computations (checking whether rank statistics are uniformly distributed vs comparing coverage) are quite different in both cases. 
limitations: The paper contains a good discussion of limitations. A slight improvement would be to discuss the trade-off between accurate calibration and conservative posteriors as mentioned above.
review_title: null

Meta-review: metareview: The paper proposes a modification to the neural ratio estimation algorithm in the context of SBI (simulation-based inference) that tends to avoid overconfident posteriors. This is important for applications (for example in scientific discovery) where excluding plausible inferences can be more detrimental than including implausible ones.

The reviewers found the paper to be well written, technically solid, and a useful contribution to the SBI literature. Most concerns were addressed during the discussion period, with the paper strengthening its discussion of limitations as a result. In the end, the reviewers unanimously awarded the paper a score of 6 (weak accept). Therefore, I'm happy to recommend this paper for acceptance.
confidence: Certain
Decision: Accept