Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation
Authors: Arnaud Delaunoy
Joeri Hermans
François Rozet
Antoine Wehenkel
Gilles Louppe
Venue: NeurIPS 2022 Accept (2022)
Keywords: simulation-based inference
likelihood-free inference
approximate Bayesian inference
neural ratio estimation

Abstract: Modern approaches for simulation-based inference build upon deep learning surrogates to enable approximate Bayesian inference with computer simulators. In practice, the estimated posteriors' computational faithfulness is, however, rarely guaranteed. For example, Hermans et al., 2021 have shown that current simulation-based inference algorithms can produce posteriors that are overconfident, hence risking false inferences. In this work, we introduce Balanced Neural Ratio Estimation (BNRE), a variation of the NRE algorithm designed to produce posterior approximations that tend to be more conservative, hence improving their reliability, while sharing the same Bayes optimal solution. We achieve this by enforcing a balancing condition that increases the quantified uncertainty in low simulation budget regimes while still converging to the exact posterior as the budget increases. We provide theoretical arguments showing that BNRE tends to produce posterior surrogates that are more conservative than NRE's. We evaluate BNRE on a wide variety of tasks and show that it produces conservative posterior surrogates on all tested benchmarks and simulation budgets. Finally, we emphasize that BNRE is straightforward to implement over NRE and does not introduce any computational overhead.

Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation
Arnaud Delaunoy" University of Liège delaunoy@uliege
Joeri Hermans" Unaffiliated joeri@peinser.co com
François Rozet
Antoine Wehenkel
Gilles Louppe
University of Liège
University of Liège
University of Liège
rrancois..rzeettullll be
antoine wereeeneeelleeeee be
g. w.llpppeuuleege.ee
Abstract
Modern approaches for simulation-based inference rely upon deep learning sur- rogates enable approximate inference with computer simulators. In practice, the estimated posteriors' computational faithfulness is, however, rarely guaranteed. For example, Hermans et al. [1 show that current simulation- based inference algo- rithms can produce posteriors that are overconfident, hence risking false inferences. this work, we introduce Balanced Neural Ratio Estimation (BNRF), a variation of the NRE algorithm 2
designed to produce posterior approximations that tend to be more conservative, hence improving their reliability, while sharing the same Bayes optimal solution. We achieve this by enforcing balancing condition that increases the quantified uncertainty in small simulation budget regimes while still converging to the exact posterior as the budget increases. We provide theoretical arguments showing that BNRE tends to produce posterior surrogates that are more conservative than NRE's. We evaluate BNRE wide variety of tasks and show that it produces conservative posterior surrogates on all tested benchmarks and simulation budgets. Finally, emphasize that BNRE is straightforward to implement over NRE and does not introduce any computational overhead.
1 Introduction
Many areas of science and engineering parametric computer simulations to describe complex stochastic generative processes. this setting, Bayesian inference provides a principled framework to identify parameters matching empirical observations. Computer simulations, however, define the necessary likelihood function only implicitly, which prevents its evaluation and the use of classical inference algorithms. To overcome this obstacle, recent simulation-hased inference (SBI) algorithms 3 build upon deep learning surrogates to approximate parts the Bayes rule and enable approximate inference. For example, 4 build surrogate of the likelihood function while ratio. The posterior can also be targeted directly with variational inference, proposed by [10 [1 5]. These algorithms are either amortized or run sequentially to drive the training towards target observation and improve the simulation efficiency of the procedure [10 12 I g [. However, sequential methods have the drawback of being computationally expensive to diagnose as the surrogates are only valid for the target observation M Truncated marginal neural ratio estimation 99] alleviates this issue by introducing sequential algorithm that builds surrogate valid in local region around the target.
[]
6, g approximate the
"Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Since modern simulation canule inference algorithms rely on deep learning surrogates, concerns naturally arise regarding their computational faithfulness and whether they are sufficiently adequate for the inference task of interest. In Bayesian inference, these concerns can be at least partially addressed with diagnostics designed to probe the correct behaviour of the inference method, such as f diagnostics for MCMC [3], or to assess the quality of posterior approximations directly. The latter include diagnostics such as simulation-be baasad calibration (SBC) [4 or coverage-based diagnostics As discussed by Hermans et al. M. posterior approximations must be conservative to not faithful. For example, in the physical sciences, where the goal is often to constrain parameters of interest, wrongly excluding plausible values could drive the scientific inquiry in the wrong direction, whereas failing to exclude implausible values because of (too) conservative estimations is much less detrimental. Unfortunately, the same authors also demonstrate that current simulation-base inference algorithms can lead to overconfident surrogates and therefore false inferences.
[1 M.
guarantee reliable inferences, even when approximations
Scientific use cases requiring conservative inference include for example the study of dark matter models in particle physics and astrophysics []6], which could be cold, warm, or hot dark matter. In general, thermal dark matter models are described by single parameter. the dark matter thermal relic mass, which can be intuitively thought of as the energy the dark matter particle had in the Early Universe. Small values correspond warm or hot dark matter, while high values are descriptive of cold dark matter. Applying an inference algorithm without diagnosing the learned estimator could lead to constraints that tighter than they should be. For example, whenever an overconfident estimator produces posterior estimates that favor cold dark matter models, it could simultaneously reject alternative models, such as the extensively studied Sterile Neutrino, potential candidate for the Warm Dark Matter particle. Making a scientific statement in this direction therefore requires the uttermost care to not wrongly exclude values of the thermal relic mass that actually plausible.
In this work, we develop a novel algorithm that not only converges to exact inference as the simulation budget increases, but which is also more likely to produce conservative surrogates in small simulation budget regimes. Towards this objective, we propose a variant of the NRE algorithm called Balanced Neural Ratio Estimation (BNRE), which enforces a balancing condition the binary neural classifier to increase the reliability of its posterior approximations.
The structure of the manuscript is outlined as follows. Section 2 describes the formalism and the necessary background. Section 3 describs BNRE and provides theoretical arguments towards its conservativeness and reliability. Section[4] illustrates our main results and provides insights regarding the behaviour of the method. Finally, Section discusses related work while Section summarizes our contributions and hints at future work.
2 Background
2.1 Statistical formalism
This work is concerned with simulation-basee inference algorithms that produce posterior approx- imations p(0 x) under the following semantics. Target parameters O denote the parameters of the model and we make the reasonable assumption that the prior p(0) is tractable. The model is generically expressed as computer program, a simulator, that describes the forward dynamics of interest based on the input parameters V. The simulator implicitly defines the likelihood function p(æ| O). While we cannot directly evaluate the likelihood p(æ> a: 1), we can execute the computer program to generate synthetic observables x ~ p(æ: I). Every observable æ, is tied to ground truth parameters O" whose forward evaluation within the simulator produced x
Of special importance to Bayesians is the notion of credible region, which is a domain θ within the target parameter space that satisfies Se p(0 x xx) d = 1 a for some observable x and confidence level - a. Because many such regions exist, we target the credible region with the smallest volume, also known as the highest posterior density region [8]
2.2 Neural ratio estimation
Neural Ratio Estimation (NRF) an established approach the simulation-basee based inference literature both from frequentist [6] and Bayesian 7 2 g 9 perspectives. In essence, all protocols rely on

---

the rensity-ratio trick [19, 20, 6 to construct surrogate of the likelihood ratio. In this work, we consider an amortized estimator f(æ ()) of the intractable
ratio r(æ v) p(0, / p((=)) = p( 0 0)((()) that can be learned by training a binary classifier d: X x θ + [o, to distinguish between samples of the joint p(0, x) with class label and samples of the product of marginals p( p())p(z) with class label O, with equal label marginal probability. For the binary cross-entropy loss, the Bayes optimal classifier is
p(0, 2: 0( a:) p(0, x) p
p(0.z) p( p()))()))
log
where a(-) is the sigmoid function. Given target parameters and an observable supported by p(0) and p(x) respectively, the learned classifier d provides an approximation for the log likelihood-to- evidence ratio log r(x 2) because r( 2)
logit (d( (. x)) ~ logit( d 0, x)) log fag [el 0). The log posterior density function is approximated log p( x) log pl ( log f(x (
Balanced binary classification for neural ratio estimation
Following Hermans et al. (1 let us first define the expected coverage probability of the 1 highest posterior density regions derived from the posterior estimator p(1) x) as
E m) [1 (0 E @pto [z) - (1-a))]-
(2)
where the function @pr0) (1 a) yields the 1 - highest posterior density region of p(0) [x).
This diagnostic probes the conservativeness of the posterior estimator (or the lack thereof) and can be interpreted the expected frequentist coverage Ep( | [1 Opc (1)1=)))
In this work, a posterior estimator has coverage at the confidence level coverage probability is larger equal to the nominal coverage probability, - a. We say that posterior estimator conservative when has coverage for all confidence levels. The expected coverage probability can be plotted for various levels a, which allows to visually identify conservative posterior estimators. The expected coverage can also be shown to a special case of the SBC diagnostic [14
whenever the expected
(see Appendix A), further motivating the usage of expected coverage.
Our main objective is to restrict the hypothesis space of the approximate classifiers d to those leading conservative posterior estimators, hence solving the reliability concerns of NRE. Towards this goal, we construct a hypothesis space of balanced classifiers and show both theoretically and empirically that they lead to posterior estimators that tend be more conservative.
3.1 Balanced binary classification
Definition 1. A classifier d is balanced if Ep(o (a(0) .r) = Ep(()p(a)
- 1-d(0, -((0xx) 2x) or
Ep(0,a) x:) Bp(())(a))))
[d(0, x:)
1.
(3)
d(0,a))
Theorem 1. Any balanced classifier satisfies B)(
2 1.
d(0,x)
Proof.
The integral form of the balancing condition
Jf (p(0, x) =()
(4)
implies that (p(x, 0) + p( ())(()) d(O x)) is valid density, both integrating to 1 and positive everywhere. Therefore, its Kullback-Leible (KL) divergence with p(0, x) is positive. Through
3

---

Jensen's inequality, we obtain
0 x KI. (p(0, (1) p(0)p(==)))))) x:)
p(0.x)
o)().x)
log (p(0,z) x) + (())p((()))ll))
d(0,2) x)
Bp(0.z)
log di ( x)
exp log d(0)z) x) d(0,x))
d(0, x) d(1), x)
S B (00,z)
= E (p(
1 d(0, x))
Theorem 2. Any bulanced classifier d satisfies Bp(()p(a)
2 1.
d(0, x)
Proof. Similar Theorem see Appendix B.
Theorem shows that, in expectation over the joint distribution p(0, æ), balanced classifier tends make predictions whose probability values d(0, æ)) are smaller than the exact probability values d(0,, æ)). In other words, a balanced classifier d tends to be less confident than the Bayes optimal classifier d. Similarly, Theorem shows that, in expectation over the product of the marginals p( p(0))((:) balanced classifier tends to make predictions whose probability values 1 - d(0, æ)) are smaller than the exact probability values 1 d(1), æ)), hence showing that balanced classifier d tends to also be less confident than the Bayes optimal classifier d. We note however that these two theorems hold only in expectation, which implies that neither d((O, x) S d( x)) for all 0, x nor
1 - -0) a) < 1 d(0,, a)) for all 0, can generally be guaranteed.
Theorem 3. The Bayes optimal classifier d(0, x) is balanced.
Proof. Replacing the Bayes optimal classifier
p(0,x) x)
d(0,x)
(5)
p(0, + p( +p()ppp(ppp) (p))))
in the integral form of the balancing condition, we have
|/ +
d0 dæ (((((=)))>(000) p(0,x)
/I/ p (ar)
do dæ
= If p(0, x)0 dx
Theorem 3 states that the Bayes optimal classifier is balanced. Therefore, minimizing the cross- entropy loss while restricting the model hypothesis space to balanced classifiers results in the same Bayes optimal classifier of Eqn. I]
3.2 Balanced neural ratio estimation
We now extend the NRE algorithm to enforce the balancing condition. The previous results show that enforcing the condition should result in more conservative classifiers d and therefore dispersed posterior approximations. Let us first note that Theorem [ Eptc) [ =) [d(1, 00, æ)] 2 1, which can (ideally) be achieved when the inner expecta- tion is larger than for all In this case, the classifier d will be such that d(O x) d(0, x) in
can be expressed as

---

regions of high posterior density. Then.
d(0,z)
d(0,x)
-- 1- x))
which is equivalent f(xz [ S r( 9).
(6)
1- 1-d(0.0)
and p(0 S p æ) since p( σ) f())f(=) 0). Similarly, Theorem 2 iimlies d(0) x) 2 1- d( x)) in regions of high prior density, which results in p(0 [) S p x)) Between those two opposite effects, the constraint on p( x) that will dominate depends on whether p(0 x)) > p(0)
[x)
or p( x) < p(0). If p(0 x) p()), then (( Φ) I p @ æ). whereas if p x)) p(O) then p(0 x) I p x). Overall, imposing the balancing condition will therefore result in approximate
posteriors that lie between the prior and the exact posterior, without being more confident than they should.
Practically, the balancing condition can be targeted through regularization penalty. For the binary eross-entropy [[d] - (.) [log d((), - - (p))))) [log(1 1 d(0), x))] and given that the balanc- ing condition only depends on samples from p(a) ) (1) and p(æ, 2). the full loss functional including the balancing condition can be expressed as
Lb
L d + (ppt))p(a)
[x) + Bp(o.e) (d(0, ax) -
where À is a (scalar) hyper-panne parameter controlling the strength of the balancing condition's 's contribution. The training procedure summarized in Algorithm[]] Since classifier balanced the balancing condition cancels out, À could, in principle, be set arbitrarily large. However, as the balancing condition is estimated via Monte Carlo sampling, setting À to large value could impair the classifier's learning ability. We found that À 100 works well across many problem domains with varying simulation budgets.
Algorithm 1 Training algorithm for Balanced Neural Ratio Estimation (BNRE).
Inputs:
Implicit generative model p(æ æ () (simulator) and prior p(0)
Outputs:
Approximate classifier dy. (0, .x) parameterized by th
hyper-parumeters:
Balancing condition strength À (default 100) and batch-size n
repeat
Sample data from the joint (o,, x; ~ p(0, x), y1 == 1/
Sample data from the marginals (ú,, 2; ~ (())p(z), y: = 0/
C[d,]] = 1 Σ log du (0,, x,) + (1 vi) log(1 d (0,,
Bidg] = 2 Σn! de (0,, + E (0..
s = minimizer_steppparreee until convergence
loss= + X(Bida] 1)2)
return x0)
4 Experiments
We start by providing an extensive validation BNRE on broad range of benchmarks demon- strating that the proposed method alleviates the problem. Section 4.2 follows up with illus- trative demonstration on the behaviour of BNRE and hyper-parameters. Code is available at https:/ htte::///iiti wwtp:///wnnoeoooooraaaaaaaaaaaaa
4.1 Extensive validation
Setup We evaluate the expected coverage of posterior estimators produced by both NRE and BNRE on various problems. Those benchmarks cover a diverse set of problems from particle physics (Weinberg), epidemiology (Spatial SIR), queueing theory (M/G/1), population dynamics (Lotka Volterra, and astronomy (Gravitational Waves). They are representative of real scientific applications of simulation- inference. A more detailed description of the benchmarks can be found in Appendix C. The architectures and ceper-parameters used for each problem are defined in Appendix
5

---

D. Our evaluation considers simulation budgets of increasing size, ranging from 2 = 1024 to 2 131, 072 samples, and credibility levels from 0.05 0.95. For every simulation budget, we train 5 posterior estimators for 500 epochs and determine the credible region by evaluating the approximated posterior density function in discretized and empirically normalized grid of the parameter space with sufficient resolution. The subsequent credible region is the set of parameters whose estimated (and normalized) posterior density is higher or equal to an inclusion threshold fitted to obtain the desired credibility level I Details on this procedure are described in Appendix E. The expected coverage probability is estimated on 10000 unseen samples from the joint p( ( Φ), for each considered credibility level.
Expected coverage The expected coverage curves and their interpretation are detailed in Figure We observe that NRE often produces posterior estimators that are overconfident, especially for small simulation budgets. However, NRE's reliability increases with the availability of training data. By contrast, BNRE produces posterior estimators that are conservative on all benchmarks for all simulation budgets. E the same phenomena through a quantity which we call the coverage AUC, highlighting the effect of the simulation budget. Coverage AUC corresponds to the integrated signed area between the expected coverage curve and the diagonal of a particular simulation. From this quantity it is evident there is clear distinction between NRE and BNRE with respect to the available simulation budget. Both methods have the tendency to converge towards O, indicating both methods moving closer to the Bayes optimal classifier. However, the difference between these methods lies with how this solution is approached. While NRE can approach this limit from both sides, BNRE consistently produces coverage AUC's above O, corresponding conservative posterior approximations, and therefore exhibits the desired behaviour (in expectation).
SLCP
Weinberg
Spotial SIR
M/G/1
Lotia Volterra
Graviaaiional Waves
1.0
NRE
0.5
0.00
1.0
Espeste soerage
BNRE
0.5
Coolliality level
1024
4096
1634
B3636
Calikeated istimater
2018
8192
32768
131072
Figure I: Expected coverage for increasing simulation budgets. A perfectly calibrated posterior has an expected coverage probability equal to the nominal coverage probability and hence produces diagonal line. A conservative estimator has an expected coverage curve at or above the diagonal line, while an overconfident estimator produces curves below the diagonal line. The diagnostic therefore provides immediate visual interpretation. We observe that NRE can produce overconfident estimators, while BNRE always produces coverage curves above the diagonal line and therefore the desired behaviour: conservative posterior approximations. The means over 5 runs are reported.
Statistical performance In addition to the reliability of the posteriors, we evaluate and compare the statistical performance of the posterior approximations produced by NRE and BNRE. We estimate the expected approximate log posterior density (x(o,.)) log p(1 æz] over a large number of pairs
&, x It captures how well the posterior surrogates p(O x)) approximate the true posteriors p( x)
since Ep((,.)) [log p(0 x)] -pp(a) KL [p(0 æ) p( [)] (p()) m) [log p(0 xx] 21
Figure 3 shows our results. We observe that enforcing the balancing condition for À 100 is associated with loss in statistical performance. However, the loss in statistical performance is eventually recovered by increasing the simulation budget. In fact, practitioners might be inclined to favor reliability over statistical performance D.. although it is always trade-off that depends on the use case. Nevertheless, it is possible improve the statistical performance by tuning the surrogate, or by increasing the available simulation budget have demonstrated.

---

NRE
BNRE
1.0
0.8
0.30
Expecte, covergo
AUC
Cheereop ADC
0.6
0.00
0.4
AUC.
0.00
-0.10
0.15
0.00
0.25
11.00
0.70
1.00
-0.00
10
10*
10
10
10'
Credibility leve
Sundlabion budges
SLCP
Weinbrg
Spatial SIR
Caliiiated estimatio
M/G/1
Tatla Valuers
Grartadiona Wines
Figure 2: Coverage AUC measures the integrated signed area between the expected coverage curve and the diagonal. A perfectly calibrated posterior has expected coverage probability equal to the nominal coverage probability, producing a diagonal line and has a coverage AUC of 0, as shown on the left subplot. A conservative estimator on the other hand has a coverage AUC larger than O and an overconfident estimator smaller than We observe that while NRE can produce coverage AUC both below or above O, BNRE always produces coverage AUC larger than O, implying that its posterior approximations are conservative on average. The means over 5 runs are reported. A complete overview, including standard deviations, are provided in Appendix F.
SLCP
Weinberg
Spatial SIR
M/G/1
Lotla Volterza
Gravitational Waves
Egcal ((g
0
10"
10
10
10
10
10
10
Simulation budger
NRE
BNRE
Prior
Figure 3: Expected value Ep(o..) log p( x)] of the approximate log posterior density of the nominal parameters with respect to the simulation budget. We observe that BNRE produces log posterior densities lower than NRE. This shows that enforcing the balancing condition to have more reliable posterior approximates comes at the price of small loss in information gain. However, BNRE improves over the prior and eventually converges towards NRE as the simulation budget increases. Solid lines represent the mean over runs and shaded areas represent the standard deviation.
4.2 In-depth analysis
In this section, consider the Weinberg benchmark described in Appendix C. The quality of the posterior approximations produced by BNRE initially discussed with respect to the simulation budget. Afterwards, the effects of the hyper-parameter À are studied.
Quality assessment Because the expected coverage does not capture the quality of an approxima-
tion in terms of information gain, we complement our assessment with a bias and variance analysis of the posterior approximations. Let us consider the expected squared error over the approximate poste-
rior Eppo 2)) (0 0²)2) where 0* is the ground truth parameter value. With ((x) Ep (e) (0),
decompose Epco
(0 )0 as
(00 (0-0(x))) + (0(x)) 0¹) Ep(0 m)) [(0 M(x))] + +ppo (x) Ex(0|a))
(0(x) 07)
=0
= =Bpoo (00 (0- (0(x)) -²))

---

NRE
BNRE
Blas
Varianae
1.00
0.08
0.08
Exposeed coverog
0.75
0.50
0.04
0.00
0.02
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
10"
10P
10a
Credibility level
Crecibility level
Sumnlatio budget
Sumulation budget
63336
2016
16384
131072
NRE
BNRE
4006
32708
Calibrated estiraelor
Figure 4: Comparison between NRE and BNRE in terms of expected coverage, bias and variance on the Weinberg benchmark. On the left side, the coverage is shown with respect to the simulation budget represented by the colormap. The bias and variance plot. BNRE is run with À = 100. Consistent with our previous observations in Figure we observe that the gap in both bias and variance reduces the simulation budget increases. Futhermore, in contrast with NRE, the posterior approximations of BNRE are tending towards being increasingly calibrated while at the same time being conservative. Solid lines represent the mean over 5 runs and shaded areas represent the standard deviation.
represented on the right side of the
The expectation over the joint distribution p(0",x)) of the expected squared error can hence be decomposed in bias term defined as
bias (ias( x)) - E Ep(or.))
( 0-)
(8)
which can be interpreted as the expected discrepancy between the nominal value o and the expected posterior value 0. The variance term
variance(p(1) x)) Ep(Ø
[ m) [(0 0(z))
and measures the dispersion of the posterior approximations. Note that these terms differ from the typical statistical bias and variance of point estimators since we are considering full posterior estimators, In particular, the bias of the Bayes optimal model does not necessarily reduce to 0.
Figure shows the evolution expected coverage, bias and variance with respect to the available simulation budget. By taking all plots into consideration with respect to the simulation budget, we can validate that as suggested by theorems[[and]
the increase expected coverage tied to an increase in variance. However, this increase comes the price of slight increase in bias. Consistent with our previous observations in Figure 3 we observe that the gap in both bias and variance reduces as the simulation budget increases. The bias gets close to 0 for high simulation budgets, showing that the bias induced by BNRE vanishes as the simulation- budget increases. A bias and variance analysis for all remaining benchmarks is discussed in Appendix G.
Effeets of A Finally, Figure showw the effect the tions, their expected coverage and the balancing condition. BNRE is run 5 times for À ranging from 1 to 2 and for fixed simulation budget of 1024. Initially, the effect on the posterior approximations is limited for small values of X. However, once À increases, the balancing condition forces the posterior approximations to become increasingly dispersed and conservative. Eventually, at least for this specific simulation budget, the posterior approximation reduces to the prior as the balancing condition becomes dominant over the cross-entropy term. Although the global optimum remains unchanged as stated by Theorem 3 large À values are likely impair the training procedure. In particular, a large À can inflate the statistical noise of the Monte Carlo estimation of the balancing condition and make the classifier d degenerate to classifier that is trivially balanced such as the random classifier d( .r) 0.5 for all v. In this case, f(x ( for all 1, a and the approxi- mate posterior degenerates to the prior. This effect directly evident from Figure starting from A ~ 1000. In practice, À should be sufficiently large such that the approximate classifier is balanced,
À on the posterior approxima-
8

---

Effect on posterions
Effect on exqueded envergge
Effect coverrugr AUC
Effict on. 55
Tainestel preterie dendit
1.26
1.00
1.00
Expreted covernge
0.8
Conorae AUCC
0.00
0.76
0.6
0.2
0.50
0.4
0.002
0.25
0.2
0,0
0.00
0.0
0.00
0.5
6.5 Coulitiitt
Hoperepmenter
NRE (130072 samilatoon)
35
91
241
001
1731
4639
12088
Figure 5: Effect of the left to right shows the evolution of the approximate posterior for a given observation at fixed Ø", indicated by the red vertical line. This approximate posterior is compared large simulation budget, shown in yellow and serving as a proxy for the true posterior. The second plot illustrates the empirical expected coverage. The third plot provides summarized view of the second plot using the coverage AUC as summary statistic. The fourth plot shows that classifiers are becoming increasingly more balanced as X increases. In addition, the plots show that À directly tied to the statistical performance and reliability of the posterior approximations. Classifiers trained with small \ are associated with (relatively) tight posteriors and overconfident approximations, while classifiers trained with larger values of À are increasingly more dispersed and conservative until the posterior approximations reduce to the prior due to inflated statistical noise of the Monte Carlo estimation of the balancing condition. Furthermore, the expected coverage plot shows the estimator is almost perfectly calibrated and implicitly balanced. Immediately visible from the various posterior approximations in the leftmost subplot, is the fact that BNRE produces overconfident and biased approximations in the presence of a small simulation budget and a small X, indicated by their dark blue color. However, the balancing condition can be applied to the underlying estimator to improve its reliability by increasing A. Ideally, À should be as small as possible to maximize predictive performance, while at the same time remain sufficiently large guarantee coverage. From the 3th subplot from the left, in this particular problem setting, that happens at the point where the coverage AUC transitions from being negative to positive (A 25.0).
À for a fixed simulation budget of 1024. The first plot from
NRE trained on a
while maximizing the statistical performance of the posterior estimator. Therefore, we recommend to start with a small value for À and to gradually increase À until the posterior estimator becomes conservative. We empirically found À = 100 to be a reasonably good default value leading to good performance across all considered benchmarks with various model architectures.
Related work
In the Bayesian setting, BNRE improves the reliability of NRE by constraining the classifier hypothesis space balanced classifiers, which results in more conservative posteriors. Towards the same objec- tive of conservative and reliable approximate posteriors, Hermans et al. ( have shown empirically that ensembling posterior estimators increases their expected coverage. Since the two solutions are complementary, we suggest that ensembling BNRE is a safe practice to follow. To the best of our knowledge, no other related work exists to make Bayesian simulation- based inference algorithms more conservative and reliable.
In the frequentist setting, Cranmer al. 6 make use of neural ratio estimation to learn likelihood ratio test statistics. They show that the classifier d does not need be exact for the statistic remain the most powerful, provided that the approximate likelihood ratio is monotonic with exact likelihood ratio. When this is not the case, robust inference remains possible by calibrating the classifier, the price of a loss in statistical power. Similarly, for frequentist likelihood-fii [22] use classifiers to estimate likelihood ratio statisties and propose a procedure for guaranteeing valid hypothesis tests and confidence sets. Finally, Dalmasso et al. 23 propose practical procedure for the Neyman construction of confidence sets with inite-sample guarantees of nominal coverage as well as diagnostics that estimate conditional coverage over the entire parameter space.
inference, Dalmasso et al.

---

In this work, we make the assumption that the simulator is well-specified, in the sense that it accurately models the real data generation process. However, this assumption is often violated. To overcome this issue, Generalized Bayesian inference (GBI) extends Bayesian inference by replacing the likelihood term by with arbitrary loss function (4). Those loss functions can be designed to mitigate specific types of misspecifications and enable robust inference, even with intractable likelihoods [25 Power likelihood losses have also been shown to increase robustness to model misspecification [28] It consists in raising the likelihood to a power to control the impact it has over the prior. The lower the power of likelihood, the lower the importance given to the data and the higher the uncertainty of the posterior. It can either be set based practitioner knowledge or derived from observed data 229 Following the same objective, Miller and Dunson [30] introduce coarsened posteriors that condition neighborhood of the empirical data distribution rather than on the data itself. This neighborhood is derived from a distance function that, when set to the relative entropy, allows the approximation of coarsened posteriors by power posterior. Recently, Dellaporta et al. [31] applied Bayesian non-parametric learning to SBI, making inference with misspecified simulator models both robust and computationally efficient.
Conclusions and future work
In this work, we introduced Balanced Neural Ratio Estimation (BNRE), a variation of neural ratio estimation designed to produce more conservative posterior estimators, even when the likelihood-to- evidence ratio estimator is not computationally faithful. We provide theoretical arguments suggesting that enforcing the balancing condition should lead to more conservative posteriors without sacrificing exactness in the large simulation budget regime. Our theoretical results are experimentally validated on benchmarks of varying complexity.
Nevertheless, our inference algorithm comes with limitations that practitioners should keep in mind. First, we emphasize that theorems [ and 2hold only in expectation, which means that we cannot provide any guarantee at the level of single inferences. Second, the balancing condition is enforced through a regularization penalty that is not estimated exactly. This implies that the classifier d is rarely strictly balanced, although close to be, in which case theorems [[ and doo not hold. Third, the benefits of BNRE remain to be assessed in high nigh-dimensional parameter spaces. In particular, the posterior density must be evaluated on discretized grid over the parameter space to compute credibility regions, which currently prohibits the accurate computation of expected coverage in the high-dimensional setting. In conclusion, BNRE should not be viewed estimators with 100% reliability, but rather as a way to increase the reliability of the posterior estimators with minimal effort and no computational overhead.
way to obtain conservative posterior
Looking forward, the balaneing condition could potentially be applied to other simulation- inference algorithms. Future works could include generalization to neural posterior estimation (NPE). In fact, the likelihoodto--evidenee ratio can be extracted from an approximate posterior by removing its dependence the prior, log f(x +) log P(O x) log p(0), which in turn can be expressed as classifier d(10 x) = ollog f(x ()) on which the balancing condition can be evaluated and enforced. Although our work focuses on amortized approximate inference, the balancing condition could also be applied to sequential inference algorithms to increase their reliability.
Finally, although our initial motivation is framed within the field of simulation- theoretical results directly applicable any binary classification task by replacing the joint and marginal distributions in the balancing condition with the distributions of the two considered classes. Therefore, provides an casy-to-implement modification for high-risk classification problems.
inference, our
Acknowledgments and Disclosure of Funding
Arnaud Delaunoy, Joeri Hermans and Antoine Wehenkel would like to thank the National Fund for Scientific Research (F.R.S.-FNRS) for their scholarships. Gilles Louppe is recipient of the ULiège- NRB Chair on Big Data and thankful for the support of NRB. Computational resources have been provided by the Consortium des Équipements de Calcul Intensif (CÉCI), funded by the National Fund for Scientific Research (F.R.S.-FRRS under Grant No. 2.5020.11 and by the Walloon Region.
10

---

References
[1] Joeri Hermans, Arnaud Delaunoy, François Rozet, Antoine Wehenkel, and Gilles Louppe. Averting A Crisis In Intullttio-Basse Inference. arXiv e-prinis, art. arXiv:21 10.00681, October 2021.
[2] Joeri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free MCMC with amortized approximate ratio estimators. Hal Daumé III and Aarti Singh, editors. Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4239 4248. PMLR, 13-18 Jul 2020.
[3] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference. Proceedings of the National Academy of Sciences, 2020.
[4] George Papamakarios, David Sterratt, and lain Murray. Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 837-848. PMLR, 2019.
[5] Manuel Glöckler, Michael Deistler, and Jakob Macke. Variational methods for simulation- based inference. In International Conference on Learning Representations, 2021.
[6] Kyle Cranmer, Juan Pavez, and Gilles Louppe. Approximating likelihood ratios with calibrated discriminative classifiers. arXiv preprint arXiv: 1506. 1699, 2015.
[7] Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, Michael U Gutmann, et al. Likelihood-free inference by ratio estimation. Bayesian Analysis, 2016.
[8] Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood- free inference. In International Conference on Machine Learning, pages 2771-2781, PMLR, 2020.
[9] Benjamin Miller, Alex Cole, Patrick Forré, Gilles Louppe, and Christoph Weniger. Truncated marginal neural ratio estimation. Advances in Neural Information Processing Systems, 34: 149-143, 2021.
[10] George Papamakarios and lain Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. In Advances in neural information processing systems, pages 1028-1036, 2016.
[11] David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-free inference. International Conference on Machine Learning, pages 2404- 2414. PMLR, 2019.
[12] Jan-Matthis Lueckmann, Pedro J Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnen- macher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. Advances in Neural Information Processing Systems, 30, 2017.
[13] Andrew Gelman and Donald B Rubin. Inference from iterative simulation using multiple sequences. Statistical science, 7(4):457-472, 1992.
[14] Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Val- idating bayesian inference algorithms with simulation-based calibration. arXiv preprint 17Xiv:1804.06788, 2018
[15] David Zhao, Niccolò Dalmasso, Rafael Izbicki, and Ann B Lee. Diagnostics for conditional density models and bayesian inference algorithms. Uncertainty in Artificial Intelligence, pages 1830-18400 PMLR, 2021.
[16] Jocri Hermans. Nilanjan Banik, Christoph Weniger, Gianfranco Bertone, and Gilles Louppe. Towards constraining warm dark matter with stellar streams through neural simulation based inference. Monthly Notices of the Royal Astronomical Society,
2021.
[17] Rob J Hyndman. Computing and graphing highest density regions. The American Stafistician, 10(2):120-122) 1996.
11

---

[18]
George EP Box and George C Tiao. Bayesian inference in statistical analysis, volume 40. John Wiley & Sons, 1973.
[19] Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the bregman divergence: unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics, 1405)11009-1044, 2012.
[20] Ian Goodfellow, Jean
Mehdi Mirza, Bing Xu, David Warde-Farley Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[21] Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation-baset inference. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 343-351. PMLR, 13-15 Apr 2021.
[22] Niccolò Dalmasso, Rafael Izbicki, and Ann Lee. Confidence sets and hypothesis testing in a likelihood-free inference setting. In International Conference on Machine Learning, pages 2334.. PMLR, 2020.
[23] Niccolo Dalmasso, David Zhao, Rafael Izbicki, and Ann B Lee. Likelihood-free frequentist inference: Bridging classical statistics and machine learning in simulation and uncertainty quantification. arXiv preprint arXiv:2107.03920 2021.
[24] Pier Giovanni Bissiri, Chris C Holmes, and Stephen G Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(5):1 103-1130, 2016.
[25] Sebastian M Schmon, Patrick W Cannon, and Jeremias Knoblauch. Generalized posteriors in approximate bayesian computation. arXiv preprint arXiv:201 1.001.0.888 1.08644, 2020.
[26] Takuo Matsubara, Jeremias Knoblauch, François François-Xavier generalised bayesian inference for intractable likelihoods. arXiv preprint arXiv:2104.07359, 2021.
Xavier Briol, Chris Oates, et al. Robust
[27] Lorenzo Pacchiardi and Ritabrata Dutta. Generalized bayesian likelihood-free inference using scoring rules estimators. arXiv preprint urXiv:2/04.03889, 2021.
[28] Peter Grünwald and Thijs Van Ommen. Inconsistency of bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12(4):1069- 103, 2017.
[29] Chris C Holmes and Stephen G Walker. Assigning value to power likelihood in general bayesian model. Biometrika, 104(2):497-503, 2017.
[30] Jeffrey W Miller and David B Dunson. Robust bayesian inference via coarsening. Journal of the American Statistical Association, 2018
[31] Charita Dellaporta, Jeremias Knoblauch, Theodoros Damoulas, and François-Xan Xavier Briol. Robust bayesian inference for himulator-based models via the mmd posterior bootstrap. In International Conference on Artificial Intelligence and Statistics, pages 943-970. PMLR, 2022.
[32] Kyle Cranmer, Lukas Heinrich, Tim Head, and Gilles Louppe. "Active Sciencing" with Reusable Workflows. https://github... http:/tthuccoooeeeeee http:///neeraatieeeeeeeeeeeeeeee
2017.
[33] Alexander Y Shestopaloff and Radford M Neal. On bayesian inference for the m/g/1 queue with efficient meme sampling. arXiv preprint 4rX15:1401.5548, 2014.
[34] Alfred Lotka. Analytical note on certain rhythmic relations in organic systems. Proceedings of the National Academy of Sciences, 6(7):410-415, 1920.
[35] Vito Volterra. Fluctuations the abundance of species considered mathematically. Nature,
1 118(2922):58-566) 1926.
12

---

[36] LIGO Scientific Collaboration. LIGO Algorithm Library LALSuite. free software (GPL), 2018.
[37] C. M. Biwer, Collin D. Capano, Soumi De, Miriam Cabero, Duncan A. Brown, Alexander H. Nitz, and V. Raymond. PyCBC Inference: A Python-based parameter estimation toolkit for compact binary coalescence signals. Publ. Astron. Soc. Pac., 131(996):024503, 2019. doi: 01888//5588887////8
13


Reviews:
[{"review_id": "d71VQgqr_yi", "reviewer": "Reviewer_oxwn", "summary": "The authors propose a modification to the neural ratio estimation (NRE) algorithm in the form of a regularization penalty to avoid overconfident posterior distributions. The authors support their claim by introducing a family of likelihood-to-evidence ratio classifiers which are more conservative than the optimal Bayes classifier, in expectation. By posing an explicit penalty on overconfidence, the authors argue that the simulation-based inference using the NRE algorithm would be more reliable and not leading to false inference. The authors provide empirical results over several datasets, showing how the balanced neural ratio estimation (BNRE) is indeed experimentally conservative and also converges ultimately to the same posterior as the NRE method for large simulation budgets. ", "questions": "I have no main questions for the authors, apart from any feedback on my comments above.", "limitations": "The one (minor) limitation I would like to flag is the various benchmarks description being relegated to Appendix C.\nI believe it would be nice for the reader to appreciate the range of the benchmarks included in the paper, with some of the simulators being not necessarily trivial. Without a description, even a quick one, a non-expert reader might see these benchmarks as all simple toy-examples.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and well presented, with legible figures and thoughtful considerations. I have checked the proofs of Theorems 1 and 2 and they look correct to me. The validation is extensive and well thought of.\n\nThe minor weakness that I can see are (a) the applicability to the NRE algorithm only (although other algorithms are mentioned in the discussion) and (b) examples of over-confident simulation-based inference algorithms leading a research direction astray.\nFor (a), as shown by [1], there does not seem to exist a universally \"best\" algorithm in SBI, so potentially extending the same logic to more algorithms would be very valuable for the community.\nFor (b), I appreciate the empirical results pointed out by [2]; my only point is that a practical example of a \"failure case\" -- or even a \"if they used this overconfident SBI algorithm in this real world study it would have been a problem\" type-of-argument -- would make this paper much more convincing.\n\nOverall, I believe the paper proposes an interesting idea and it is still worth flagging in the community, although I believe that it is not as compelling as it could be.\n\n[1] Benchmarking Simulation-Based Inference, Lueckmann et al, AISTATS 2021\n[2] Averting A Crisis In Simulation-Based Inference, Hermans et al, 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SydkzNUhyZ7", "reviewer": "Reviewer_NmkK", "summary": "This study builds on a previous state-of-the-art method for simulation-based inference (i.e., NRE), and proposes an improvement that aims at avoiding the usual overconfidence of NRE posterior estimates (an issue that might be common to other simulation-based inference methods). The study provides empirical results on several benchmark problems and shows that the new method (BNRE) clearly improves in reliability compared to NRE as assessed by several metrics.", "questions": "As pointed out above, I believe it would be beneficial to the manuscript if evidence was provided for the statement that $\\lambda$ = 100 is a good choice across benchmark problems. An appendix figure would be enough.\n", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\n\nTo the best of my knowledge, this is one of two studies that specifically addresses the issue of overconfidence of simulation-based inference algorithms, the other one being Hermans et al. 2021 (cited by the authors). While Hermans et al. diagnose this problem on several simulation-based inference methods and provide a partial solution to this issue using ensemble approaches, the solution provided by this study is novel.\n\n\n### Quality\n\nThe paper is technically sound, and most claims are backed by empirical evidence. However, there is one claim that I would appreciate the authors to illustrate: \"We found  $\\lambda$ = 100 to perform well across all benchmarks, which again, is supported by Figure 5.\" I would recommend the authors to provide a figure (in appendix) illustrating this, even if for a subset of the benchmark problems. \n\n\n### Clarity\n\nThe manuscript is clearly written, providing enough information to understand the technical contribution and empirical results, while appropriately putting the contributions in the context of previous work. A few small comments:\n\n-page 4, line 119, \"does not modify the global optimum\". This is a bit imprecise and might lead to confusion, so I would suggest the authors to expand on it;\n\n-page 4, line 122, \"[...] result in increasingly conservative [...]\". By \"increasingly\", I assume the authors meant with increasing $\\lambda$. As this is not clear, I would suggest the authors to drop the word \"increasingly\";\n\n-page 4, line 123, the word \"Ideally\" wrongly suggests that the expression (6) only follows from the previous inequality in the ideal case, which is not what the authors meant. Dropping the word \"Ideally\" would solve the issue;\n\n-page 7, line 186, for clarity, the notation for nominal parameter ($\\vartheta^*$) should be introduced close to the expected squared error over the approximate posterior.\n\n\n### Significance\n\nThe technical development and empirical improvement of BNRE over NRE will be of interest to the ML community, in particular to the simulation-based inference community. Notably, the technical contribution could directly inspire the improvement of other methods for simulation-based inference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PFshcbP6ec9", "reviewer": "Reviewer_Vwgm", "summary": "Simulation-based inference enables Bayesian parameter inference for simulation-based models for which the likelihood cannot be evaluated efficiently, by using simulations from the model as training data. Over the last years there has been developed a suite of neural network-based SBI methods that estimate the posterior distribution directly (neural posterior estimation, NPE), or target the likelihood (neural likelihood estimation, NLE) or the likelihood ratio (neural ratio estimation, NRE) to then obtain posterior samples via MCMC. By design, SBI is usually applied in scenarios where the ground-truth posterior is not known, and with limited training data, e.g., because model simulations are expensive. Thus, the approximated posterior (samples) may be inaccurate for a given problem. Preliminary results in the field indicate that posteriors tend to be overconfident, e.g., they tend to underestimate uncertainties.\nThis submission introduces a variant of NRE, called Balanced NRE (BNRE), with the goal to make posterior estimates more conservative in the low simulation budget regime. Like NRE, BNRE trains a classifier in order to approximate the likelihood-to-evidence ratio, to then perform MCMC. The authors introduce the concept of balanced classifier and introduce a modified loss function to train BNRE such it is a balanced classifier. Furthermore, they show that the optimal Bayesian classifier is balanced, i.e., that in the limit of unlimited training data BNRE will eventually converge to be NRE and obtain the exact posterior. They perform experiments with tractable benchmarking problems to show that BNRE tends to be more conservative than NRE in the low simulation budgets, i.e., show larger posterior variance. They conclude that BNRE provides a new SBI method suitable to obtain conservative posterior estimates. ", "questions": "As outlined in detail above, I think this study would benefit from additional tasks and from additional metrics to study the effect of BNRE in more detail. For the benchmarking tasks I suggest to use one that is fully tractable so that it can be studied in detail, and one with real-world character that is challenging, e.g., high-dimensional, so that one can see the practical relevance of BNRE. For metrics I suggest to additionally use SBC, local coverage tests and analytical over and underdispersion and bias (for the tractable tasks). \n\n", "limitations": "The authors transparently addressed limitations of their approach, given the scope of the results presented in the paper (see comments above). \n\nUpdate after rebuttal: The authors addressed all of my questions and the revised version of the submission now adequately discusses limitations of the approach to the low-dimensional parameter regime. I updated my score accordingly. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** The reliability of posterior estimates obtained with SBI is a timely and very important topic which I am happy to see addressed here. The paper picks one of three main SBI approaches, NRE, and suggests a variant with the goal to make it more conservative in the low simulation budget regimes. The introduction of balanced classifiers is novel and useful for the context of NRE and, as mentioned in the conclusion, it may indeed be applicable to NPE as well.\n    \n**Quality**: The introduction of balanced classifiers and BNRE as a variant of NRE with an additional term in the loss function is technically sound and a novelty. However, the experimental evaluation of BNRE is limited and less convincing for several reasons:\n- Choice of benchmarks: The set of benchmark tasks is limited, it contains rather low-dimensional tasks most of which (all? depending on the variants used, exact definitions of the tasks missing) are tractable via MCMC. It would be more instructive to show one or two tractable examples with reference posteriors (obtained via MCMC or analytically) to study the properties of BNRE in depth, and to then show one or two additional “real-world” examples to demonstrate the use of BNRE in practice and to show how it compares to NRE in practice. I see the current results merely as a first study showing preliminary results.\n- Choice of metrics: The results of the paper are mainly based on one single metric measuring the “confidence” of the posterior—the expected posterior coverage. This metric was introduced in a recent preprint by Hermans et al [1] which, again, addresses the very important topic of SBI reliability, but makes relatively strong claims about the general reliability of SBI methods based solely on the expected coverage metric, and was not successfully peer-reviewed by the community yet. The evaluation of BNRE would become more informative if additional metrics were considered. One common metric for investigating the posterior coverage and potential biases is simulation-based calibration (SBC). The authors showed in the appendix that in theory expected coverage is a special case of SBC, however, it would be useful to see the practical similarities in the results. Importantly, SBC is able to detect not only miscalibrated uncertainties like over- or underdispersion, but also positive and negative biases of the posterior estimates. Thus, it is additionally suited for evaluating BNRE given that BNRE tends to introduce a bias in the low simulation budget regimes. In addition to SBC which provides a global marginal coverage test across all x, there is a new method for performing a local coverage test for specific observation (Zhao et al. [https://proceedings.mlr.press/v161/zhao21b.html](https://proceedings.mlr.press/v161/zhao21b.html)), which would be useful to add as a metric as well, e.g., in the “real-world” example. \n- For the tractable tasks with known posteriors it would additionally be useful to calculate actual biases, and dispersion with respect to the reference posteriors. \n- For the real-world examples additional checks, e.g., prior and posterior predictive checks would be instructive to show the practical effect of conservative posteriors and difference of BNRE to NRE.  \n- Finally, a comparison to other established SBI methods like NPE and NLE would be illustrative. They are readily available in open-source software packages so that adding them to the benchmark would not result in large algorithmic or implementation overhead.\n\nOverall I think, given the importance of this topic, it is essential to evaluate a new method on the available and established metrics (especially when additionally introducing a new metric).\n    \n**Clarity**: The paper is very well and clearly written, it was straight forward to follow the line of arguments and the presented results. Regarding the appearance of the figures I have the following comments:\n- A visual explanation of the approach as Figure 0 would be nice to have\n- For most of the subplots in Figure 1 it is hard to see any differences between the methods, a different visualization, e.g., on the log-scale would be more illustrative\n- For Figure 2 it would be good to show standard error of the mean as error bars, given that 5 repetitions were performed\n- For Figure 3 it would better to show standard error of the mean instead of the standard deviation (minor)\n    \n**Significance**: As mentioned above, I find the general topic and the approach of BNRE important for SBI research. However, the experiments and results presented here appear to me as preliminary with no direct insights or consequences for other researchers or practitioners (yet). In theory, the BNRE approach makes sense, however, in practice (as indicated by the results presented here) I do not see an advantage over NRE. The results **do** show that on the benchmark tasks presented here, BNRE leads to broader posteriors than NRE. However, the additional results in Figure 3 and 4 also indicate that it comes with a larger bias (which makes sense due to the additional term in the loss). While the bias reduces with increasing simulation budget, but even for the same budget, it seems that BNRE performs worse than NRE. Furthermore, BNRE comes with the additional hyperparameter choice for $\\lambda$. The authors make a suggestion for a default value of lambda based on extensive parameter search on the benchmark tasks, however, it is not clear how this choice would extrapolate to real-world scenarios, e.g., for models with more parameters. Thus, from the practitioners point of view, it would not makes sense to use BNRE as of yet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2oJEZOpRDYy", "reviewer": "Reviewer_X2bu", "summary": "The paper introduces and discusses a regularization approach for neural ratio estimation that is based on enforcing a balancing condition of the classifier (via an additional regularisation term). It is shown theoretically that on average (expectation) this results in a more conservative classifier.\nIn several experiments it is shown that the approach indeed results in a more conservative posterior. ", "questions": "- Could you please write down the definition of a classifier? I assume you mean something like $d: X \\rightarrow [0, 1]$\n- The discussion around equation (6) is confusing. You say that $\\hat{p}(\\theta|x) < {p}(\\theta|x)$ whenever the $\\hat{d}(\\theta,x) < {d}(\\theta,x)$, but clearly $\\hat{p}(\\theta|x) < {p}(\\theta|x)$ can't hold for all $\\theta$ since we are considering densities. Hence, the crucial aspects of whether the density is more conservative is _for which_ $\\theta$ one has $\\hat{p}(\\theta|x) < {p}(\\theta|x)$ and that seems to be missing. \n- Generally the model details are a bit short. For example, you reference Lotka and Volterra indicating that the underlying model is a deterministic ODE following the Lotka-Volterra equations. However, commonly people in SBI work with the stochastic version, describing a Markov jump process. I would be nice to have more information here. \n- I am not so sure about the expected coverage as a special case of SBC. The internal consistency of the joint distribution that you're relying on is something that is remarked even in the SBC paper to be known before. The devil is in the detail and from my point of view SBC is about an actionable algorithm, which is more than just the expectation that you allude to in Appendix A. In particular, the actual computations (checking whether rank statistics are uniformly distributed vs comparing coverage) are quite different in both cases. ", "limitations": "The paper contains a good discussion of limitations. A slight improvement would be to discuss the trade-off between accurate calibration and conservative posteriors as mentioned above.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n- Ensuring that posteriors derived by deep learning algorithms are valid is very important.\n- The use of the balancing condition as an additional measure is an interesting idea.  \n- The method is easy to implement.\n- The experiments show that the posterior tends to be more conservative when the regularizer is applied.\n- The induced bias and variance diminish with increasing sample size\n- Little to no exaggeration. The authors are very honest about the advantages and disadvantages of their method which makes the contribution all the more valuable to the field. \n\n__Weaknesses__\n- The penalty term does result in a worse calibration in most examples. While the posterior is more conservative the calibration is often worse, in particular when the number of samples is low. Especially for models where simulation is costly this might be an issue. \n- While I agree that conservative posteriors are preferable, calibration is not a binary issue. Overly conservative posteriors might be unable to identify a parameter to a reasonable degree. In the Weinberg, SIR and Lotka Volterra models, a relatively accurate posterior is traded in for a mis-calibrated conservative posterior. I think the trade-off is important.  \n- There is a broad literature in statistics considering model (over-)confidence [A - C] which I think should be mentioned. In particular, power posteriors are quite popular to adjust the posterior in order to account for discrepancies between a model and real data. Since NRE does not estimate the likelihood directly, it might be necessary to use the presented approach instead of power posteriors, but a discussion should be added.\n\n> Recently, Dellaporta et al. [26] further improved GBI by combining \n- The cited work has advantages and disadvantaged but it's not an \"improved\" version of GBI, it's a variant. In particular, it best applies to IID data. When using time-series data as many of the models in this paper [D] would be a more fitting GBI procedure. \n\n> M/G/1, originally introduced by Papamakarios et al. [4],\n- Papamakarios et al. however reference [E] for the M/G/1 model. This should be amended. \n\n\n__References__\n\n[A] - Peter Grünwald. Safe learning: bridging the gap between Bayes, MDL and statistical learning\ntheory via empirical convexity. In Proceedings of the 24th Annual Conference on Learning\nTheory, pages 397–420, 2011.\n\n[B] - Peter Grünwald. The safe Bayesian. In International Conference on Algorithmic Learning\nTheory, pages 169–183. Springer, 2012.\n\n[C] - Chris Holmes and Stephen Walker. Assigning a value to a power likelihood in a general\nbayesian model. Biometrika, 104(2):497–503, 2017\n\n[D] - Dyer, Joel, Patrick Cannon, and Sebastian M. Schmon. \"Approximate Bayesian Computation with Path Signatures.\" arXiv preprint arXiv:2106.12555 (2021).\n\n[E] -  A. Y. Shestopaloff and R. M. Neal. On Bayesian inference for the M/G/1 queue with efficient MCMC\nsampling. arXiv:1401.5548, 2014.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept