Generalizing Coverage Plots for Simulation-based Inference
Authors: Maximilian Lipp
Benjamin Kurt Miller
Lyubov Amitonova
Patrick Forré
Venue: Accepted by TMLR ()

Abstract: Simulation-based inference (SBI) aims to find the probabilistic inverse of a non-linear function
by fitting the posterior with a generative model on samples. Applications demand accurate
uncertainty quantification, which can be difficult to achieve and verify. Since the ground
truth model is implicitly defined in SBI, we cannot compute likelihood values nor draw
samples from the posterior. This renders two-sample testing against the posterior impossible
for any practical use and calls for proxy verification methods such as expected coverage
testing. We introduce a differentiable objective that encourages coverage in the generative
model by parameterizing the dual form of the total variation norm with neural networks.
However, we find that coverage tests can easily report a good fit when the approximant
deviates significantly from the target distribution and give strong empirical evidence and
theoretical arguments why the expected coverage plot is, in general, not a reliable indicator
of posterior fit. To address this matter, we introduce a new ratio coverage plot as a better
alternative to coverage, which is not susceptible to the same blind spots. It comes at the
price of estimating a ratio between our model and the ground truth posterior, which can be
done using standard algorithms. We provide experimental results that back up this claim,
and provide multiple algorithms for estimating ratio coverage.

Published in Transactions on Machine Learning Research (03/2026)
Generalizing Coverage Plots for Simulation-based Inference
Maximilian Lipp
m.lipp@arent
Science Pork 106, 1098 XG Amsterdam, The Netherlands Advanced Research Center for Nanolithography (ARCNL) Vrije Universitest Amsterdom
Benjamin Kurt Miller
Science Park 904, Amsterdam, The Netherlands Informatics Institute, Institute of Physics University of Amsterdam
Lyubov V. Amitonova
Crmnttonoea@arcn...
Science Pork 106, 1098 XG Amsteriam, The Netherlands Advanced Research Center for Nanolithography (ARCNL) Deportment of Physies and Astronomy Vrije Universiteit Amsterdom
Patrick Forré
f.d.forreeuva..l
Science Park 904, Amsterdam, The Netherlands Informatics Institute University of Amsterdam
Reviewed on OpenReview: https: // openreview. net/ forum? id= eo
Abstract
Simulation-based inference (SBI) aims to find the probabilistic inverse of non-linear function by fitting the posterior with generative model samples. Applications demand accurate uncertainty quantification, which can be difficult to achieve and verify. Since the ground Iruth model is implicitly defined in SBI, we cannot compute likelihood values nor draw samples from the posterior. This renders two-sample testing against the posterior impossible for any practical use and calls for proxy verification mothods such as expected coverage lesting. We introduce a differentiable objective that encourages coverage in the generative model by parameterizing the dual form of the total variation norm with neural networks. However, we find that coverage tests can easily report good fit when the approximant deviates significantly from the target distribution and give strong empirical evidence and theoretical arguments why the expected coverage plot is, in general, not a reliable indicator of posterior fit. To address this matter, we introduce a new ratio coverage plot as a better alternative to coverage, which is not susceptible to the same blind spots. It comes at the price estimating a ratio between our model and the ground truth posterior, which can be done using standard algorithms. We provide experimental results that back up this claim, and provide multiple algorithms for estimating ratio coverage.
1 Introduction
We are concerned with a perennial question Simulation-Baue Inference (SBI): How do we determine if learned approcimation to the posterior accurately represents the ground truth?

---

Published in Transactions on Machine Learning Research (03/2026)
Recall that the necessary ingredients for deep baaring--bssed SBI include: a non-linear, stochastic function, known as simulator, that takes in parameters A and returns simulated observation x; a prior distribution over all possible parameters p(0); and a method that infers the distribution over parameters that would plausibly generate 2o if they were passed to the simulator. This target distribution is called the posterior p(0 z) == p( p(0) evaluated at observation Zo. namely p(0 zo). In SBI, we typically cannot evaluate the likelihood plar 0), so we need to estimate the posterior using a generative model.
p(a)
Classical Coverage
3]
4)
0.8
Coverage
0.6
0.4
b)
0.8
Coverage
0.6
0.4
100
Coverage
0.6
0.4
0.2
Credible level
Figure 1: A one-dimensional example highlighting the problem of classical coverage, which cannot separate the very different model and true distribution. The classical coverage is calculated using 1D normal distributed data p(=) and different models g(2). Subplot a) shows the histogram of the data well as the learned model distribution q(2). The true data distribution p(z) is not shown since it is not available in practice. In subplot b), some of the original 1D data points = ~ p(z) are shown, where orange points are selected by the threshold g(=) 2 and blue data are not. To calculate the classical coverage, first a thrreshol t is chosen on the model distribution q()), which leads to the super-level set (t) illustrated in subplot c). Then for subfigure d), the distribution p(z) is integrated in the region of the super-level set (t), (using the histogram data) and plotted against the integrated distribution (t), for many different super-level sets with corresponding thresholds l. Since the model g(z) closely matches the histogram p(=), the coverage correctly shows diagonal line. In the top row, the model and data distribution agree. In the middle row, the model is shifted to the side and in the bottom row the model has two peaks, which also creates perfect coverage.
The inference method produces an approximation to the posterior q(0 x.). Since the true posterior p(0 x)) is unknown for any practical use case, it is not straightforward to determine whether our approximation accurately represents the ground truth using two-sample testing (Gretton et al. (2006); Lopez-Paz Oquab (2017); Friedman (2004)). An ideal algorithm would compare q(O xx) to p(0 xx)) without evaluating or sampling the posterior. Since such comparison is impossible in practice, recent papers instead demonstrate
2

---

Published in Transactions on Machine Learning Research (03/2026)
proposed inference method using two-sample tests on toy problems with known solutions Lueckmann et al (2021). This does not help practitioners who need to know whether their specific posterior approximation can be trusted, not the performance of the inference algorithm on toy problem.
In lieu of this ideal, fictional algorithm, practitioners typically test the expected conditional coverage of their approximation (Miller al. (2021); Hermans et al. (2022); Talts et al. (2020); Zhao et al. (2021)). Intuitively, expected conditional coverage tests whether the approximation's credible region cover the ground truth's credible region, at all credible levels, averaged over p(z). Although this sounds promising as a criterion to determine whether two distributions agree, there are issues with expected conditional coverage. One known limitation is that the prior p(0) has exact expected conditional coverage at all credibility levels Delaunoy et al. (2023); Lemos et al. (2023), i.e. this method cannot distinguish between p(0) and p(0 | æ)) Despite this limitation, many works have proposed algorithms aimed at improving the expected conditional coverage of learned posteriors through regularization (Delaunoy et al. (2022; 2023); Dey et al. (2022); Falkiewicz et al. (2023)). We will propose another one.
We find an additional, significant blind spot in expected conditional coverage that has gone hithertofore unmentioned in literature: Coverage does not always penalize the approzimant for putting mass in regions where the ground truth has none! Note that this limitation is quite deep in the notion of coverage itself, extending to the so-called unconditional coverage, i.e. coverage testing between arbitrary distributions p(z) and q(2) that is not averaged over a conditional variable. We visualize an example of such a failure mode in Figure 1. In addition to this pathological example, we create a differentiable objective that encourages unconditional coverage on q() x) for every z, and thereby expected conditional coverage, and find that we can produce distributions with good expected conditional coverage, even though the approximation and the posterior significantly disagree.
To address this issue, we propose generalized version of coverage that (a) discriminates between distributions and p. i.c it only returns the typical diagonal line when = p, (b) does not require samples from the ground truth posterior, and (c) does not suffer from either blind spot mentioned above. However, it comes at the cost of estimating the ratio ((0))))(()) ((0/2)
p(0.z) p(02)
Contribution (1) We formulate rigorous definition for both unconditional and expected conditional coverage. We relate coverage tests to divergences between probability measures and show that the typical learning objective for Neural Posterior Estimation (NPE) penalizes a lack of coverage. (2) Propose a differentiable and adversarial regularization objective based on the total variation distance that encourages coverage and show that unconditional coverage is not discriminative, i.c., when q has unconditional coverage w.r.l. p that does nol imply p = q. We thereby address a commonly held misconception that issues with expected conditional coverage only stem from the necessity to average over p(z). (3) Present an alternative form of coverage called the ratio coverage which is discriminative. We formulate it both unconditionallyy over the joint distribution of 0 and x, and as expected conditional ratio coverage. (4) Provide experimental evidence for (2) and (3).
Related Work We addressed the relevant two-sample tests and coverage testing for SBI above; however, we specifically mention the work E-C2ST Linhart et al. (2023). for sequential procosses SSNL. Dirmeier et al. (2024). E-C2ST, which more precise and computationally efficient than its concurrent method local-HPD Zhao et al. (2021), is similar to ratio coverage because we also estimate ratio between q( x)) and p(0 x)) using the likelihood ratio trick (Hermans et al. (2020); Durkan et al. (2020); Miller et al. (2022); Gutmann & Hyvärinen (2010); Oord et al. (2018); Dalmasso et al. (2020)) or other methods (Miller et al. (2023); Federici et al. (2023); Nguyen et al. (2010); Yao & Domke (2023)). e-C2ST and the HPD met hodology summarizes the whole information concerning 0 into a single scalar or a d-vector respectively, while this work uses an estimator to improve the statistical power of the well established coverage plot. So instcad of proposing a new method for inspecting local posterior consistency, we show that the widely used coverage plot can be modified to achieve full discriminative power. An estimator with ratio coverage would pass 4-C2ST for all T..
3

---

Published in Transactions on Machine Learning Research (03/2026)
2 Preliminaries
Since this paper deals with the fundamental propertics of coverage a mcasurement of similarity between probability distributions, we treat the problem from a fundamental perspective. In particular, we will treat the properties unconditional coverage using on a general probability space 2, which we take to be the joint space of parameters and data X' in the SBI case. Furthermore, critical to many of our practical results for SBI is the asymmetry between ground truth posterior p and approximation q. We typically assume we neither evaluate nor sample from p(0 x), but we do both with q(0 x). This the normal sit uation for expected coverage testing.
Throughout, we deal with comparing distributions and we seek to formalize this using the following divergences and measures.
Definition 2.1 (Kullback-Leibler divergence). Let B2) be a measumble space and P and Q two probability measures on (2, B2) with densities p and q. The Kulbbeck-Leibler divergence then:
KL(PIIQ) Ep log
(1)
The Kullback- Leibler divergence is a common statistical distance, often used as an NPE training objective and also forms the loundation for our regularizer (see Section 5).
Definition 2.2 (Total variation distance/noni
Let (2,Bz) be a measurable space and P and Q two probability measures on (2,Bz). We then define their total variation distance/nows norm by the formala:
TV(PIIQ) = sup IP(A) Q(A)),
(2)
AEBs
which is always number in the interval [0, 1].
Theorem 2.3 (For proof see Theorem A.6). We have the following identilies for the total varialion norm:
TV(PI[Q) = 1p(2) q(=) v(d2)
sup 112-1-111 measurable
Eplf Eolfil.
(3)
where p and q are densilies for P and Q, resp., w.r.l. any fixed joint dominaling measure v, e.g. (P)))) Furthermore, the supremam on the rhs is attained for the map f* given as follows:
{() p(z), if q(2) = =(2),
((2) p(z)
[*(=) = sgn log
(4)
if q(=) p(=).
The Total variation distance is also statistical distance between probability distributions, which we can relate to an optimality condition on the classical coverage and subsequently results in the adversarial total variation norm regularizer (sce Section 5).
3 Generalized Coverage Plots
We define the fundamental terms analogous to the choice from Hermans et al. (2022), but split the derivation into two steps: first introcigngg the mathematical simpler case unconditional coverage and then extending to the more widely used definition of expected conditional coverage. This generalized framework allows investigate different kinds of coverage plots, e.g. the classical coverage plot in Example 3.3.
3.1 Generalizing Unconditional Coverage Plots
We start with a generalized definition of coverage for unconditional probabilities, then explain the specific choice leading to the classical coverage and finally facuses on estimating these coverage plots.
4

---

Published in Transactions on Machine Learning Research (03/2026)
Definition 3.1 (Unconditional coverage plots and coverage gap). Let (2, B2) be meosurable space, P and Q two probability mensures on that space, and, let g: 2 + RR == R Ù (+ be measurable map. The (unconditional) coverage plot of P and Q w.r.t. discriminating function is defined to be (an estimate) of the following graph:
(P = ((Q())) P(C,(0)) LE R] S [0, x [0, 1].
(5)
where (t) is the super-level set of g for threshold R:
Cy(t) = f2 E 2 (l()) 2 t = g ool).
(6)
Purthermore, we define the (unconditional) coverage gap belween P and Q w.r.l. g as follows:
(PII)) = sup ((C,(())) [0.
(7)
EER
For further information, see Proposition A.4.
Remark 3.2 (Estimating the coverage plot with samples). Assume that we have i.i.d. sample {21,.... 2N from P and on i.i.d. sample ( from Q. Then we can approzimate the coverage plot T g Q) as follows. Pirst consider the set of all relevant thresholds:
= = (2(en)) [N]] Ü (o(2()) m [M]}
(8)
Then for every & E 'T we compule:
p(t) == # {n [N]] 2 t /N,
((t) # {m E [M]] (((()))))) 2 t /M,
(9)
and plot the corresponding points for all t T:
(10)
See Algoritlo and Figure for more information.
The coverage gap (P))), more intuitively, measures the maximal distance between the coverage plot I (P, and the diagonal (o. ((r,r) [0, of the square [0, 1] [0. 1].
Example 3.3 (Classical coverage plot). We regain the classical coverage plot for the function g(2) := q()), the probability densily of the model distribution Q. Then, the super-level sels correspond to the highest probobility confidence regions of q(2). This choice for g(2) comes however with severe blind spot, because it canmot generally distinguish between P and Q, i.e., it does not detect any probability mass in region where q(z) has small probability. Figure 1 shows that the classical coverage gap vanishes: (PI))) = 0, although P # Q. A more rigorous proof can be found in Framplle A.9.

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 1 Generalized unconditional coverage plot
Require: simulator p(z(0), prior p(0), surrogate model ((0)z), discriminating function g(8, x), sample size numbers N, M;
Ensure: list of confidence level pairs {(7n, 7,)) I E [N]];
for m = to M do
8m ~ p(0)
Im p mm)
fm ~ {( pm
end for
for to N do
On (00)
In p(x(0n)
Nn + ΣM [g((mm Fm) 2 g(On. zn)l
end for
for all Nh in sort((m) In E [A]]) do
(Mk, 1/)) + (7k, )
end for
3.2 Generalizing Expected Conditional Coverage Plots
Next, we follow again three steps for the expected conditional coverage: giving generalized definition, discuss the expected condilional classical coverage widely used in literature and finally talk about practical implications for optimization.
Definition 3.4 (Expected conditional coverage plots and coverage gap). Let (0,Ba) and (x) be measurable spaces, let P = P(e,X) X) and Q = Q(e, X) be two probability measures on 2 = O x X with the same maryinal P(X) = 2 and lel g x X IR = RR (+x)) be a mensurable map. Firsl, consider the conditional survival function of g w.r.t Q, which is on E R, x E and 0 O given by:
S(t|z) Q(g x) 2 +|X x),
(11)
Its partial inverse, the threshold function is given for 7 E [0. 1] and X as follows:
=up =up {1 E S(1x) 2 7) ER.
(12)
With this we define the set:
C,(7) := (00,x) E 0 x (10002)) (x))
(13)
The expected conditional coverage plot of P and Q w.r.l. g is defoned to be (an estimale) of the following graph:
(P.Q)) ((OC))))((((C,))))))))C)))))))) [0, 1]) 00, [0,1]
(14)
Purthermore, define the expected conditional coverage gap between P and Q w.r.t. 9 follows:
(PIIQ)
sup IP(C,())) Q(C,( e [0. >E[0,1
(15)
Before estimating the erpected conditional coverage plot, note that for every x E X' and [0, we always have:
2 {
= x)) 2%
Q(C, 2%
(16)
In SBI, the simulated data P(X) input data model Q(X) both generated in the same way: P(0) and
then P(X 0), hence P(X) Q(X)

---

Published in Transactions on Machine Learning Research (03/2026)
with equalities under suitable continuity/positiiity conditions on the conditionals Q(OIX z).. In those cases we are thus just plotting P(Cg(1)) against É 10.
Additionally note that we have the following equivalence for & E R and ? E [O, 1] under suitable continu- ityppositivity conditions:
S(1x) S
1 1 (2).
(17)
Remark 3.5 (Estimating the expected conditional coverage plot with samples). For n E [NV] sample 0n P()) and Fn P(X 0n) and put t, g(0n, Fn)) For m [M] further sample &n,m Q(e Ia). Then pul:
M
=
lg Fnd 2 tul Qlo(O, 2t. 2t = z.)) = S(tn)=a))
(18)
M
m=1
Further define for E [];
1 N F()) = N
N
(18)
Σ
1 nn $ 7]
~
[[S(tn (xn) s
(19)
N
==1
N
1 [4n 2 (3mn)
(20)
N
R=1
Σ Ig(0n, Un) Ly (xn))
(21)
Pl((0,X)24(0
(22)
With these approrimations and the argument in (16) can thus estimate the expected conditional coverage
plot as:
(PIIQ) (1: (() /
e 10.11] S [0.1] x 10.1].
(23)
See Algorithm and Figure for more information.
Example 3.6 (Expected conditional classical coverage plot). Analogue to Example 3.3, we regain the classical coverage for the choice g(0,2) = log q(flz). In contrast to the previous Ezample, this plot first determines the coverage condilioned on over 0, before then averaging the resull over all Hence, the expected conditional coverage plot compares p(0).) to x)) while the unconditional coverage plot compares p(())))(()) to q(0 p)))
Remark 3.7.
1. We always have the trivial inequality:
A₂(PIQQ < sup
-
(24)
CEBaxx
where the supremum ranges over all measurable subsets C. of the product space O X.
Because we assumed that the marginals agree: P(X) Q(X), we also have the following tighter inequalities (full proof see Remark A.2):
(p(1)) s Ep(x) KS(g P(10)
(25)
< Ep(x
[TV P(e) ((0))
(26)
sup
-
(27)
-
measurable

---

Published in Transactions on Machine Learning Research (03/2026)
with the push-forward probability measures X Q(([X) and 9 P(O]X) (Tao (2021)), KS the Kobmogoroo-Swne Smirnoo divergence (Definition A.1) and / = [[] X), X) where for x E X we abbreviate the partially evaluated map:
g" - R.
g²( :- -(0):=g((xx))
(28)
Algorithm 2 Generalized expected conditional coverage plot
Require: simulator p(z]0), prior p(0), surrogate model ((0)ar), discriminating function g(0,z), sample size mumbers N, M;
Ensure: list of confidence level pairs {(7a, 7h) I E [N]];
for n = 1 to N do
0n pp)
In (n)
for to M do
On.m x(0
end for
Tn + M Em= [g(On, vn) 2 g(On, xn)]
end for
for all 7ik in sort([7n In [N]]) do
(Th, 112) + (7k, )
end for
4 The Ratio Coverage
As we pointed out that classical coverage tests cannot in general discriminate different distributions, we propose the rafão coverage, which does not have this deficiency, but still shares valuable properties with the classical coverage, e-g. does not require samples from the ground truth posterior. The ratio coverage motivated by the supremum found in Theorem 2.3 which (*) sgn g(z) with the choice g(0, x) = log (=)/(=))p(z) (see Remark A.3 and Figure 2).
Theorem 4.1 (The ratio coverage plots discriminating). Let p= P(O,X) x) and Q Q(O, X) be two probability measures on Z = θ X with the same marginal Q(X) P(X) and with densities p and q. Consider the (log) rulão function g given as follows:
g(0,z)) = =log ((0,z))
=log g(0)x))
(29)
p(0,xz) p(0) x))
Then both, the unconditional coverage plot (PIIQ) and the expected conditional coverage plot F, (PIIQ)
each can discriminate between and Q:
(PI = - 0 P=Q
(P(
P Q.
(30)
Furthermore, we have the following finullualssies for the coverage gaps and TV norms:
(PIlQ) < TV Q) TV(PIQQ) Â (PI(Q) I Ex(x) TV (( TV(P))).
(31)
(32)
Proof. See Theorem A.7, Corollary A.8 and Figure 2.
However, in a practical setting the ratio coverage cannot be calculated directly, because it requires the model distribution q(2) as well as the true posterior distribution p(z). Therefore, we propose train a classifier approximate the ratio (z2)/p(2). This approach must be distinguished from Neural Ratio Estimation (NRE), which instead learns the Hkelihood-to--viddee ratio. Additionally, single value metrics also be
8

---

Published in Transactions on Machine Learning Research (03/2026)
obtained by calculating the total variation norm between both distributions directly using the learned ratio as similarly proposed in Gutmann et al. (2018).
This ratio training is also a limitation of the ratio coverage, because of potentially large parameter space spanned by θ x X. The training requires additional computational cost, and the ratio coverage directly depends on an accurate estimation. However, the additional effort is justified by the full discriminating power, specifically in comparison to the classical coverage, which also requires intensive computing resources.
In summary, proofed that the ratio coverage preserves full discriminating power for both plots, the unconditional coverage plot and the expected conditional coverage plot. The unconditional coverage plot calculates the coverage symmetrically over the joined space θ x X. In contrast, the expected conditional coverage plol first determines the coverage conditioned on I over 0, before then averaging the result over all I. Hence, the expected conditional coverage plot compares p(0 x) to q(O x)) while the unconditional coverage plot compares p(0 x))(() to q(0) x)p(x)) The unconditional coverage plot much more simpler in regards of derivation well as in terms of computational cost, since it requires far fewer samples from the model. Although the expected conditional coverage has been mainly used throughout literature (Hermans et al. (2022)), the unconditional coverage has clear advantages in terms of simplicity and computing cost.
p(0|x)
p[0] x8 0)
coverage
0.8
Expected Coverage
truth
model
0.6
class, coverage
ratio coverage
0.2
-2g
8 -5
5
0.5
Credible level
Figure 2: An example illustrating the problem of classical coverage for conditional probabilities. The true and model dist. given by p(0]xr) and ((0)x)) are shown in the first image and the cross section for x x= 0 is presented in the second plot (black dotted line). Similar to the unconditional case, these distributions exhibit one and two Gaussian peaks, whose mean value is shifted by x. The third plot show the approximated classical and ratio coverage basod on the Algorithm for a finite data sets drawn from the distributions.
Looking at ratio coverage from the point of view of hypothosis testing, we consider a sample 2 drawn from P or Q, and our test should determine from which distribution most likely originates. For some (summary) test. statistic y and threshold I, we decide for P g(=) < and decide for Q g(z) > L. In other words, the null hypothesis corresponds to Ho P while the alternative hypotheses is HA Q. Our unconditional ratio coverage plot directly plots the following test quantities against cach other: P(g > t) typeIerror and Q(g 1) = - typel classical Neyman-Pearson lemma the unique uniformly most powerful test is the ratio test. So, with our approach, we learn the most powerful summary statistic.
power. So our coverage plot has direct statistical interpretation and by the
5 Classical Coverage Regularization Can Be Deceptive
Because the classical coverage is widely viewed as key performance metric, usually a next step would be to develop regularizer to emphasize well calibrated models. In this section focus finding such a regularization objective to exemplify how this goes wrong because of the blind spot of the classical coverage. As shown in Figure 1, the classical coverage can be optimal, although the model did not learn the true distribution. We develop the Adversarial Total Variation (TV) Regularization as a differentiable upper bound to the coverage gap, which will lead to improved classical coverage in Section 6, but simultancously create great disagreement between the learned and the true posterior distributions. This problem then exacerbated by the fact that the classical coverage is unable to detect this specific kind of misalignment,
9

---

Published in Transactions on Machine Learning Research (03/2026)
because of its blind spot. The same phenomenon can also be observed when regularizing with techniques from the literature (Section 6.3).
By using that the expected conditional coverage gap for general choice of g is bounded by the total variation norm, we derive an adversarial regularizer for minimizing the coverage gap.
Equation (27) provides us with an adversary regularization bound for minimizing the coverage gap:
1 sup ( -
[Bp[]] - -Fq/Il
(33)
f: f measurable
with / /
Under certain regularity assumptions on the densities p and g we can restrict the model class F for the discriminator f: R X + [-1,1] to sufficiently flexible neural network classes with tanh-outputs such that the corresponding Universal Approxination Theorem holds, see Kurt Hornik (1991). We thus arrive al the regularizing adversarial objective for such neural net network classes f FE
Rg f) Eplf ol Eolf o g]l.
(34)
We can now combine this regularizer with the usual NPE training objective, the Kullback- Leibler divergence (Papamakarios et al. (2017)), resulting in the adversarial loss function with Lagrange multiplier 2 0:
(g KI.(PII) + A Rg,
(35)
where we maximize w.r.t. the parameters of and minimize w.r.t. the parameters of Q.
For the expected conditional classical coverage, i.c. 2 O X and g(0, x) log ((0]2)), we get the adversarial loss (written with densities and arguments):
C(g. = Dog [(0[=)] + A- [B (fog (0])z), x)] E
(36)
where we, again, maximize w.r.t. the parameters of f and minimize w.r.t. the parameters of ((0[z).
Analogously to the adversary TV regularizer for the classical coverage, we could also investigate an adversarial ratio regularizer. But for the NPE objective Equation (35) it important to note that for the log ratio g log we have the chain of inequalities (Pinsker's inequality) for the coverage gap:
1 A,(11) < TV(g -VVPP) S KL(PIIQ)
TV(P1Q))
(37)
So, the usual NPE objective (the KI. divergence) already closely upper bounds the adversarial ratio regularizer (Remark A,5), which can thus be dropped from the NPE training objective in this case. Similar arguments hold for the expected conditional version.
6 Experiments
In this section we present the empirical evidence for the problem with the TV regularization, the superior discrimination power of the natio coverage and the comparison between expected conditional and unconditional coverage plots. In our experiments, we follow the procedure from Hermans et al. (2022) for evaluating SBI models on the SLCP. Specifics about the implementation and computational setup can found in Section A.2 and additional results for the two moons benchmark task can be found Section A.3.
Code is available: https: //tp://///nnmosssee/mm 4open. accience/rrrratiocorrrrgseese
6.1 Results
The first aspect of the evaluation of the models the expected conditional classical coverage in Figure for simulation budgets 20 and 216 with and without TV regularization. There is clear correlation between
10

---

Published in Transactions on Machine Learning Research (03/2026)
210
1024
216
65536
21
1024
2
-65536
withont regularization
Expected coverage
Expected coverage
with regularizatio Expected coverage
Expected coverage
0.5
0.5
0.5
0.5
0
0.5
0.5
0
0.5
0.5
Credible level
Credible level
Credible level
Credible level
Figure 3: These plots indicate improvements due to the TV regularization for the simulation budgets 2 2 but this originates from the lack of discriminating power of the expected conditional classical coverage. The coverage plots are determined using Algorithm 2 (M = N = 1024).
better coverage and larger simulation budget. In the regularized case, this trend is not clearly visible since the initial coverage is already close to the optimal diagonal and therefore does not leave much space for improvement. Comparing the approach with and without regularization, the coverage appears clearly improved by regularization, especially for the smaller training budgets. This signals a great success for the regularized approach, if one relies only on the classical coverage.
To investigate the performance of these models more deeply, we utilize the reference posterior, which is provided by the sbibm package for this benchmark task. For fixed measurement. &, 10000 samples are drawn from estimated posterior ((0]2) and the true posterior p(0]2), respectively. Based on those samples, Figure 4 shows the parameter regions with the highest probability of the true (orange) and estinated (blue) probability density corresponding to the same models shown in the previous figure.
The models without regularization show the expected trend of better agrocment between the true and approx- imated distribution for larger sinulation budgets. However, the regularized models show no improvement for larger training sizes and, in general, have less conformance with the true posterior. This is especially visible for the diagonal plots in Figure 4. The models without regularization perform better than the models with regularization, which confirms that the regularizer harms the optimization (see Section 4). This stands in opposition to the result of the classicul coverage performed before.
This apparent contradiction corresponds to the general blind spot of the classical coverage, which allows for two different distributions to achieve perfect coverage (compare Example 3.3). The model distribution learned by the regularized model, resembles the prior very closely, which has been identified in the literature, e.g- Delaunoy al. (2023), and can also be shown in the simple 1D case (Figure 8). With this experiment, we show that this mathematical case occurs in practice, specifically when using training method that optimizes for perfect classical coverage. Hence, we show empirically that distributions close to the prior also have perfect classical coverage. In this example, we are only able to detect this by using true posterior samples, which is however not possible in practical applications.
6.2 The Ratio Coverage
Therefore, we propose the rafão coverage, which can discriminate these models correctly without using samples from the true posterior. In Figure the ratio coverage plots based on the trained ratio estimation are compared with the classical coverage plots, for the same models. The models themselves are identically and have not been retrained, hence the expected conditional classical coverage is identical to Figure 3.
The expected conditional ratio coverage plots also show an improvement when comparing the step from low to high simulation budget. But in contrast to the classical coverage, the ratio coverage favors the unregularized over the regularized model for all simulation budgets. The same conclusion follows from the values of the total variation norm, calculated from the same trained ration estimation. For comparison, also the Kullback-Leibier divergence calculated using the ratio estination and supports previous findings. Specifically, using the TV regularizer hinders the performance of the models.
11

---

Published in Transactions on Machine Learning Research (03/2026)
210 1024
2 65536
99.7 %
99.7 %
80
95.5 %
Bo
95.5 %
68.3
model
model
truth
truth
without rogularization
8
8p
01
02
Os
06
01
02
0.
04
99.7 %
99.7 %
95.5 %
B
95.5 %
68.3 %
68.3 %
model
truth
truth
8
with rogularization
84
8g 0,
8, 8s
8
8, 82
Figure 4: These plots reveal the devastating effect of the TV regularization despite its apparent positive effect on the classical coverage. They show the true and model posterior distribution for the SLCP benchmark task with simulation budgets 210 226 with and without TV regularization contour plot of the three specified probability levels. The distributions shown for the fixed observation therefore only the dimensional & dependence is depicted.
12

---

Published in Transactions on Machine Learning Research (03/2026)
2. 1024
2 65536
2
1024
n
65536
without regulacization Expected coverage
Expected coverage
cless.
with reguler regularization ization Expected coverage
Expected coverage
class.
0.5
0.5
0
0.5
0.5
Credible level
Credible level
Credible level
Credible level
kl.
Figure 5: In contrast to the classical coverage, the ratio coverage can detect negative effect of the TV regularization and favors the unregularized models. These plots show the direct comparison between the classical and ratio coverage based on the same models shown previously for simulation budgets 2 216 with and without TV regularization. The coverage's are determined using Algorithm 2 (M = = 1024). The TV and KL norm is calculated over the same data.
Hence, we have shown that also in practice the ratio coverage has improved discriminating capabilities compared to the classical coverage and the TV norm can be evaluated as single value performance metric. Additionally, this performance metric offers the same benefits as the classical coverage, easy interpret, casy to use without access to the true posterior samples. But there is the additional computational cost of learning the ratio from set of samples.
6.3 Classical Coverage Regularization
unregularized
CalNPE
BNPE
TV reg.
nato
Expected coverage
Expected coverage
Expected coverage
Expected coverage
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5 Credible level
Credible level
Credible level
Credible level
Figure 6: The classical (red line) and ratio coverage (green line) are shown for the SLCP task with a simulation budget of 20 for standard NPE (KI. divergence), CaINPE (Falkiewicz et al. (2023)), BNPE (Delaunoy et al. (2023)) and TV (Section 5) regularized models using Algorithm 2 (M = = 1024).
This section extends the previously shown blind spot problem of the classical coverage using other regularization techniques from the literature. In accordance with Figures 4 and 5, Figure 6 additionally shows models trained using the CaINPE (Falkiewicz et al. (2023)) and BNPE (Delaunoy et al. (2023)) method. All presented regularized models show the blind spot behavior where the classical coverage plot gives ideal results while the posterior plot clearly deviates from the ground truth. In contrast, the ralio coverage favors the unregularized model, which in agreement with the observation of the distribution plots.
This underlines the problem of the lacking discriminating power of the classical coverage in scenarios where the true posterior distribution is not available and direct comparison between the learned and true posterior is
13

---

Published in Transactions on Machine Learning Research (03/2026)
therefore impossible. Additionally, it shows that it is not problem of the specific regularization technique, but is related to the general blind spot problem of the classical coverage, which disqualifies it as a regularization objective.
6.4 Unconditional Coverage
Lastly, we compare the unconditional with the expected conditional coverage plot from the previous section. Figure 7 visualizes the same models as Figure 5 using the classical and ratio coverage, but calculates the plots using the unconditional coverage plots. Although this method is mathematically and computationally simpler, the plots are very similar and show overall the same features. Although these results look very promising, a limitation of this example is small number of parameter. The joint parameter space of SLCP only consist. of 13 parameter and going to higher dimensional problems might not work so well. In Section A.4 in the Appendix, we further compare both methods and also investigate their dependents on the number of samples, i.c. parameters N and M).
2 1024
2
65536
2
1024
2
65536
without regulactzation Expected coverage
Expected coverage
with regulaziza tonn Expected coverage
Expected coverage
class. joint
ratio joint
rotio joint
0.5
Credible level
Credible level
Credible level
Credible level
Using the unconditional coverage plot instead of the expected conditional coverage plot reduces the number of required samples, is mathematically simpler and reproduces identical results for classical and ratio coverage. Reproducing Figure 5 for simulation budgets 200 2 with and without TV regularization, but using the unconditional coverage plot (Algorithm 1 with N M 1024).
Figure 7:
7 Conclusion
The motivation for this work originates from the blind spots of the classical coverage and its limited explanatory power. Specifically training objectives tuned to improve the coverage can very easily work in the opposite way and hinder the successfully training. Additionally in these situations, the expected conditional classical coverage plot is not able to detect the failure but instead signals a perfect performance.
We develop an adversary total variation objective function, derived from base principles, to optimize for the classical coverage. Although the expected coverage be substantially improved, we provide empirical evidence and theoretical arguments that the quality of the surrogate model has actually been decreased. Since this deterioration remains undetected by the classical coverage, we propose the ratio coverage with increased discriminatory capabilities. At the same time, the ratio coverage is easy to use and interpret and does not require access to the true posterior. Lastly, we propose the unconditional coverage as an alternative to the expeoled conditional coverage, which reduces the computational cost the calculation while also being mathematical more intuitive. To summarize, we showed that the KL divergence optimizes for the coverage without additional objective, instcad encourage to improve diagnostics by using the ratio coverage.
Broader Impact Statement
The implications of SBI are similar to other SBI methods and are primarily scientific, but one must be careful to use an accurate generative model and to carefully test results. We emphasize the importance the diagnostics in our paper, bocause untested inference can lead to incorrect conclusions, which can be missed by practitioners doing inference in any field. Special care applies results that may influence human behavior factors responsible for elimate change.
14

---

Published in Transactions on Machine Learning Research (03/2026)
References
Jean Bretagnolle and Catherine Huber. Estimation des densités risque minimax. Séminaire de probabilités de Strraboougg, 12:342-360 363, 1978. URL attp:://eudml..org/boo////
Niccolò Dalmasso, Rafael Izbicki, and Ann B. Lee. Confidence sets and hypothesis testing in likelihood-free inference setting. In Proceedings of the 37th International Conference Machine Learning, 1CML'20. JMLR.org, 2020.
Arnaud Delaunoy, Joeri Hermans, François Rozet, Antoine Wehenkel, and Gilles Louppe. Towards reliable simulation--aase inference with balanced neural ratio estimation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openrevieen cest/forummideee 62mMj4XK. AX
Arnaud Delaunoy, Benjamin Kurt Miller, Patrick Forré, Christoph Weniger, and Gilles Louppe. Balancing simulation-based inference for conservative posteriors. Fifth Symposium on Advances in Approzimate Bayesian Inference, 2023. URL https://openree http://pppaeeeeeeeeeeeeer/rrraaa
het/fforummidxkpooZhhr
Biprateep Dey, David Zhao, Brett H. Andrews, Jeffrey A. Newman, Rafacl Izbicki, and Ann B. Lee. Towards Instance Wise Calibration: Local Amortized Diagnostics and Reshaping of Conditional Densities (LADaR).. arXiv e-prints, art. M.Xiv:2205.14666 May 2022. doi: 10.48550/arXiv.2200 2205000......22..55555..
Simon Dirmeier, Carlo Albert, and Fernando Perez-Cruz. Simulation-based inference using surjective sequential neural likelihood estimation, 2024. URL https://aaaat http:///aawww.waa////////e
Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood-free inference. In Proceedings of the 37th International Conference on Machine Learning. ICML'20. JMLR.org, 2020.
Maciej Falkiewicz, Naoya Takeishi, Imahn Shekhzadeh, Antoine Wehenkel, Arnaud Delaunoy, Gilles Louppe, and Alexandros Kalousis. Calibrating neural simulation-based inference with differentiable coverage probability, In Thirty-seventh Conference on Newral Information Processing Systems, 2023. URL https: // //openrevier http//orrmmmddwwiiM/aaaa
Marco Federici, David Ruhe, and Patrick Forré. On the effectiveness of hybrid mutual information estimation, 2023. URL
Friedman. On multivariate goodness-of-fit and two-sample testing. Technical report, SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States), 01 2004. URL https: wwwp:o.stiggoo gov/ biblio/826696.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. kernel method for the two-sample-problem. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Newral Information Processing Systems, volume 19. MIT Press, 2006. URL https: htttps://proceedingg .df..
wwwurpps.c/ppaeeee files/
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304 JMLR Workshop and Conference Proceedings, 2010.
Michael U. Gutmann, Ritabrata Dutta, Samuel Kaski, and Jukka Corander. Likelihood-free inference via classification. Statistics and Computing, 28(2):411-425,, Mar 2018. ISSN 1573-1375 doi: 10.1007/
011222-017-9738-6 URL https://doo org/10. /tt///100000111222-00-0799333330
Joeri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free MCMC with amortized approximate ratio estimators. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4239-4245 PMLR, 13-18 Jul 2020. URL ctttps://proceedinge mlr. wwl..prees//ee
html.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Jocri Hermans, Arnaud Delaunoy, François Rozet, Antoine Wehenkel, Volodimir Begy, and Gilles Louppe. A crisis in simulation-based inference? beware, your posterior approximations can be unfaithful. Transactions
on Machine Learning Research, 2022. ISSN 2835-8856. LHAbHkktAq.
URL httpp://openrevieennet///ee
Tosif Pinelis. Total variation distance. MathOverflow, 2023. URL http::///aaaaorrroooooeee//////a
(version: 2023-05-12).
Mark Kelbert, Survey of distances between the most popular distributions. Analytics, ((1):225-245) 2023. ISSN 2813-2203. doi: 10.3390/aman /mayyyyc2yyyss20001222.. URL https://www.cr mdpi ccmm/2813-220322/1111
Pablo Lemos, Adam Coogan, Yashar Hezaveh, and Laurence Perreault-Levasser Sampling- accuracy testing of posterior estimators for general inference. In Andreas Krause, Emna Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. PMLR, 23 29 Jul 2023. URL https: http://pcoceeee.ee//////// mlr, prwpppeeeeeeeee http//222/1/11/ http//22222eeemeeeeeee emos23a. html.
Julia Linhart, Alexandre Gramfort, and Pedro L. C. Rodrigues. L-c2ST: Local diagnostics for posterior approximations in simulation-based inference. In Thirby-seuenth Conference on Neural Information Processing Systems, 2023. URL httpp////ppaeeeeeeeeee///oomooop wttp///rrmmiddkk/kkeeee.e..
David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017. URL https://openreviiet met/forumnid-=SJKKEE
Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation--assd inference. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Slatistics, volume 130 of Proceedings of Machine Learning Research, pp. 343 351. PMLR, 13 15 Apr 2021.
Benjamin K Miller, Alex Cole, Patrick Forré, Gilles Louppe, and Christoph Weniger. Truncated marginal neural ratio estimation. Advances in Neural Information Processing Systems, 34:129-143, 143, 2021.
Benjamin Kurt Miller, Christoph Weniger, and Patrick Forré. Contrastive neural ratio estimation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2022. URL https://openrevier
Benjamin Kurt Miller, Marco Federici, Christoph Weniger, and Patrick Forré. Simmlation-based inference with the generalized kulbeeck-leibler divergence. In 1st Workshop on the Synergy of Scientific and Machine Learning Modeling ICML2023, 2023.
XuanLong Nguyen, Martin J Wainwright, and Michael I Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, (6611))5847-5801 2010.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arxiv preprint Xrxi::1801 03740,748 2018
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density es- timation. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceeet neurips..rr..p... hccp/aperfilleee/eeeeeeeeeeeeeee fwtles/paper22117/iiL..
pdf.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep lcarning library. In Advances Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers. http:://papers.nurppp.... courrps..c/papeee. 9015-pytorch- wwwpppytorcc---immrattteee www.aaapppetttteeeeeeeeeeeeeeeee
pdf.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2): 251-25: 1991. ISSN 0893-6080. doi: fttper///bbocrrr////////////0000 // ottp///cccccaattaaa////000000000
URL https:
François Rozet, Arnaud Delaunoy, Benjamin Miller, et al. LAMPE: Likelihood-free amortized posterior estimation, 2021. URL https://pypi. pttppoooopp//ttaappppa
Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating bayesian inference algorithms with simulation-base calibration, 2020. URL https://arxiv. wtwpa//wwww...raaba///..........
Terence Tao. An introduction to measure theory, volume 126. American Mathematical Society, Los Angeles, CA, 2021. ISBN 978-1-4704-6640-4.
Alvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Marthis Lueekmann, Conor Durkan, Pedro J. Gonçalves, David S. Greenberg, and Jakob H. Macke. sbi: A toolkit for simulation-based inference. Journal of Open Source Software, 5(52):2505, 2020. doi: 10.21105/joss.025500 02500.. URL attps://dol 0tgg/0..2100 02505.
joss. 02505.
Yuling Yao and Justin Domke. Discriminative calibration: Check bayesian computation from simulations and flexible classifier. In Thirth- Conference on Neural Information Processing Systems, 2023. URL https://openreet
David Zhao, Niccolò Dalmasso, Rafael Izbicki, and Ann B. Lee. Diagnostics for conditional density models and bayesian inference algorithms. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Thirty-Seventh Conference on Uncertainly in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 1830-1840, 1840. PMLR, 27-30 Jul 2021. URL attps://proceediin mlr. press/v161/2hnr 20t/61////obbbbbb html.
A Appendix
A.1 Proofs and further Results
Definition A.1 (Kolmogorov-Smmeg Smirnov divergence). We also highlight the Kolmogorov- Smniioooroo-mmiioo divergence between probability measures P and Q on R R Ü (oo)), which is described by.
KS(PIIQ) = sup P([t. oo)) - ([(t,oo)))
(38)
te
"Note that, the definition the Kalmogorov-Smimna divergence, took intervals [t. oc] instead of (the equivalent. choice of) [-no, I] or corresponding (half-)open intervals, which can also be found the literature.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Remark A.2. Because assumed that the marginals agree: P(X) Q(X), also have the following
lighter inequalilies:
(P(I)
sup IP(C((T)) Q(Cg(T))
(39)
>E(0.1]
< sup
E p(() [| E (x))x) e (x)))
(40)
7E/0.1]
S E(x)
sup
P (+)) - e EC (>))1))
(41)
[E|0,1]
Ex(x)
sup 2 - 2 (x)
((o(0,))
(42)
pE(0.1]
S E(x)
suplP(g(0, 2 (X) Q(0(0,
sup
(43)
EER
EP(x
[KS P(O1X)
(44)
S Ex))
(45)
TV (Lo P(e]X)) 8
@ P(X))
(46)
sup
Ep [r] -
(47)
1:ExX-1-111 meosuruble
where for X abbreviate the partially evaluated map:
g 1 R
g²) g(0,
(48)
Remark A.3.
1. Note that the supremum in Equation (27) is by Theorem 2.3 ottained for:
(" = sgn log (y)xx) p(y)x))
(49)
where q(ylz)) and p(ula:) denote the probability densities of the push-forward probability measures get Q z) and g P(O]X= = z), resp., on R.
If we choose g either to be:
g(0, log g(flz))
or g(0.z)) sgn log ((flz))
(50)
p(0 x))
p(02z)
then by Theorem 4.1 the bound Equation (27) equals:
Ep(x) P(O Ep(x) [TV(P(@)X]]Q(((1))))
(51)
-TV(PII)
(52)
1 sup
EP Eg[/]|,
(53)
2 exx
measurable
and the supremaon is by Theorem 2.3 allained with f* given by:
(0) = sgn log q(6.z) p00,x)) sgn log ( x))
(54)
p(0
Proposition A.4. Let the notation be like in Definition 3. 1.
18

---

Published in Transactions on Machine Learning Research (03/2026)
1. We can rewrite the occurring quantities with help of the push-forward measures 9 P and 9 Q from Equation 7 us follows::
(L), = (0# (lt,
Q(C,(()) (9# ()((
(55)
2. With this we see that the coveruge gap between P and Q w.r.l. can be identified with the Kolmogorov- Smirnov divergence between their corresponding push-forward probability measures:
Ag (PIIQ) = sup (9 P)(t, ool) (9# (It, *ER
Plgs Q).
(56)
3. We then directly get the following upper bound with the total variation norm (see Remark A.5):
KS(g*Pllg Q) sup
(f# P)()) (B)) (
(57)
BEB
sup
[ Rols g]و,
(58)
/--11-111
where the first supremam ranges over all measurable subsets B of IR, and, the second supremum over all measurable maps f from I to [-1,1].
Remark A.5. We have the following well-known inequalities:
1. KS(PIIQ) & TV(P)IQ) Kelbert (2023).
2. Binker-Bretognole Huber inequality (see Bretagnolle & Huber (1978)):
TV(P11Q) & min (s -
(59)
Theorem A.6 (See Iosif Pinelis (2023)). We have the following identilies:
2 TV(PI)) = / l0 - 9l du=
Epif] - Eglfll.
(60)
[-2-1-1.1]
and the supremum on the rhs is ottained for:
P f= 12, -1z, sgn log
(61)
9
where
log p(2)
2, = f 2 plp(2) > ((z)) = 2 2
> 0
(62)
(())
2 = 2(xz) <())) = E Z log x(z)
0
(63)
((3)
where P and q are densities for P and Q, resp., u.r.l. any fixed joint dominating measure U, e.g. P Q.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Proof. For measurable subset A Z we get:
/ p(z) v(d2) - -fx ((a))
- =
(64)
A
/ (o(=) - <(()) u(dz)
(65)
/ A
/ (9(z) -p(=)) m(zz)
(p(z) - g(x)) v(dz)
(66)
Ac
20
20
(Ls (p(=) - {(2)) -(()) u(dz), / (u(z) - -v(z)) u(dz)
< 1max
(67)
( (o(2) a(z)) udd))
[ (9(2) -p(2)) v(dz)
S max
(68)
2.z
Note that the maximum on the rhs can be achieved by either putting A Zz A Ze, depending on which set maximizes the rhs. Also note that for A = Z we have P(Z) = Q(Z), so the Ihs vanishes. The above equality then shows:
J 2, [((te)) (()))) udz) = / (g())-p(z)) v(dz).
(69)
So both values in the maximum above are the same and the maximium can thus also be written as the convex combination:
TV(PIIQ) = max (1 (p(2)
(p(2)) - g(z)) -((e))-())))
/ (9(2) - p(=)) (c()) v(dz)
(70)
2
22
(1 (p(2) ((=))
/
/ (9(z) - p(z)) vldz)
(71)
(f f(s)) p(z) / f(z) (()))( u(dz))
(72)
2
(Er[s] -B(l1),
(73)
where:
f:= 1z, 1z sgn log ('
(74)
Note that we can also write:
TV(PIIQ) max (L (p(z)- v(dz),
(q(z)- -00-pppo p(2))
(75)
2<
2 2 (p(z) ((z)) v(dz) + / (9(2) - p(z)) v(dz)
(76)
ze
(/2
/
z, Im(=) ((=)) (ddz)
Iz Ip(z) v(d))
(77)
(p(=) (ddz)
(78)
20

---

Published in Transactions on Machine Learning Research (03/2026)
Now if : 2 - [-1,1] is any measurable map, we get:
Ep - Bg[b]] = f 11(2) p(z) v(dz) -f f (1(2) ((2) udzz)
(79)
/ h(2)) (p(z) {())) mizz)
(80)
- / (+(a) (p(=) -
(81)
SI
- / Im(z) q(>)l
(82)
2 TV(PIIQ).
(83)
So we finally get:
1 sup
TV(PIQ) 2 1-2-1-1.11 measurable
Ep[h]
(84)
So all claims are shown.
7
Theorem A.7. We now consider the functions given by the log-ratio between the corresponding probability densities and its sign function:
gir log ((2)
gx(2) sgn log 9(2) p(z)
(85)
p(2))
Both correspording coverage plots r are able to distinguish P and Q. Furthermore, we have for those two g is the identity:
= Plo
(86)
Proof. We first show the last claim. It is clear that:
TVgg*Plo S TV(PIIQ).
(87)
For the reverse, note that by Theorem 2.3 we have with g(z) sgn log g(=).
p(=)*
1
TV(PIQ)=
[Bz-[l((]] (Zz2oll(2]
(88)
2
1
< =
sup
1E 1h o g(Z) - Ez 1h g(Z)]
(89)
2
[-1,1-1,11]
1
sup
By -
(90)
2
1-111-1,11
TV Pll Q).
(91)
This shows the claim. Similarly, for g(z) = log {(z)
p(z)*
Now assume that:
0 (P =
(92)
then gy P= as KS proper divergence. In particular, we get:
0 Q))
(93)
which implies P QQ TV is a proper divergence.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Theorem 4.1 together with Equation (25) now show the following:
Corollary A.8. Assume that the marginals are equal: P(X) Q(),, and, that we use the log ratio
g(0, x) = log [00 p( Then we get the following inequalities:
(P(II) & BP(X) (XX))
(94)
S Ep(x) [()))
(95)
=Ep(x) TT(((01X)]QQ(6)1))
(96)
- =TV(PIIQ).
(97)
Purthermore, if (Pll() = 0 then we have: P = Q.
Proof. The chain of inequalities follows from Theorem 4.1 together with Equation (25).
If now (PII)) 0, then also Ep(x) [K P(O)x)ll 0. Since KS is proper divergence, have that for -(mm-almost-all X' the equality:
4 P(O =
(98)
which implies:
P(O)) =) =x))] =0.
(99)
By the above identitics we thus get:
0 Ex() ((O))) TV(P)
(100)
Since TV is a proper divergence, this implies P= Q.
Example A.9 (Superlevel sets cannot discriminate probability measures). Let q be a probability density supported on [-2, -1] and p be the probability density given by:
1 1 p(z)== = 90() 2 ,9(2= 3).
(101)
which is supported on [-2, -1,,, Let Q and P be the corresponding probability measures of R w.r.t. and p. resp. Now consider the supertevel set for for thresholds > 0:
(t == (a R 2
(102)
(= E [-2, -1 11) 2 Ú (= e (1,2) 2 #
(103)
2 [-2, -1] 1 50(3) 2t) u uf
((())2/]
2 g0(= 3) 2
1
(104)
q ((22, 1
U (q (12t, 20)) +3)
(105)
11 12,
(106)
22

---

Published in Transactions on Machine Learning Research (03/2026)
with 1 q-' " 00)) - (1)) P(h) P(I2)
and 12 = + 3 _ [1. With this get:
(107)
J, p(=) +
S p(2) dz
(108)
11+3
/,, (5olee)
2 g(( 3)
de +
2
1
g(z)
+50lz -3) dz
1
(109)
12
=0 on I
=0 on la
1 (J, d(2) /
11- alz al=-3) dz
(110)
2
1 (J, d(z)dz+ + S, a(3)) dz'
(111)
2
g(z)dz de
(112)
h Q(1))
(113)
+(1)++
+
(114)
Q(),
(115)
This shows that P and Q agree on all superlevel sets of P, but clearly P + Q. Note that for t = 0 we have C,(1)= = Th and (I), (t), as well.
p(0lx)
x=0
0.8
Expected Coverage
truth
coverage
model
0.6
class. coverage
ratio coverage
x
0.4
0.5
-1
0.2
-2g
5
5
0.5
Credible level
Figure 8: Analog to Figure 2, this example extends the same scenario but uses 10 peaks in the model distribution instead of two. As such, the model distribution becomes quite flat and resembles what we see in the contour plots in the experiment. The true and model dist. given by p(0)x: and ((0]zr) are shown in the first image and the cross section for I = 0 is presented in the second plot (black dotted line). The model distribution does not depend on & in contrast to the previous figure and the 10 peaks are equally distributed between -4 and The third plot show the approximated classical and ratio coverage based on the Algorithm for finite data sets drawn from the distributions.
A.2 Setup
In our experiments, we follow the procedure from Hermans et al. (2022) for evaluating SBI models and also use the implementations from the sbi 0. 21.10 0 package (Tejero-Cantero et al. (2020)), in order to be comparable to litcrature. All caleulations are performed on a Windows Server 2019 with a AMD Ryzen Threadripper 3970X processor with 32 cores (3693 MHz, 64 logical processors) and NVIDIA GeForce RTX 2070 SUPER. The total available physical memory 128 GB (3200 MHz).
23

---

Published in Transactions on Machine Learning Research (03/2026)
We conducted two kinds of experiments. First, using the the NPE class from sbi package, use normalizing flow networks together with the standard Kullback-Leibler training objective to create a base line results close to the literature reference. We chose different simulation budgets 210 ,216 and train each model for 50 epochs. After each epoch, the validation set of 2 elements is evaluated and the model with the lowest validation loss is restored after the training. We observe similar results for different batches sizes, the learning rate is scaled accordingly, and therefore chose a batch size of 512 for maximal efficioncy. The data is simulated using the standard benchmark SLCP implemented in the sbibm 1 1.1.0 package (Luecknann al. (2021)). It simulates fictive problem with 5 parameters and 8 observables sampled from multivariate Gaussian whose mean and covariance matrix are parameterized.
Second, the TV regularized case is trained with the additional adversary optimization performed on the same data as the training of the surrogate model. To ensure good comparability with the reference models, the starting conditions are the same with and without regularization. Custom implementations were only necessary for the adversary training part of the TV regularizer using the PyTorch 2.2.0 (Paszke al. (2019)) and LAMPE 0.8.2 (Rozet et al. (2021)) package. The adversary network consists of dense neuronal network with ReLU activation functions and a tanh final layer. It receives 10 update steps for each update to the surrogate model, to ensure good convergence. The regularization coefficient (Equation (35)) was set to x = 10², which roughly aligns both loss contribution in strength at the beginning of the training.
For the evaluation of the ratio coverage, another classitier is trained. It uses an AdamW optimizer with a batch size of 512 and a dense network with x 64 hidden features and ELU activation trained for 32 epochs. An architecture search was performed to find these values. However, for our experiments, the training of the NPE model far surpassed the training of the classifier in terms of computational cost and amount of required training data. Hence, we selected classifier architecture that is sufficiently flexible, but did not optimize it extensively. It is trained using binary cross entropy loss on total of 1000 data points consisting of (f,z) and (0, 2) with 0 being generated from the simulator 0 ~ p(0|z) and Ô from the surrogate model 0 ~ g(0]z)) Then the optimal classifier is the probability ratio as discussed in Theorem 4.1.
The main challenge when using the TV objective function is the adversary training, which is unstable. Using the
X, it can be controlled at the cost of reducing the regularization effect.
All code to reproduce the results presented in this paper is available: https://anonymou/ 4opp:n science/
A.3
Two Moons
In the main manuscript we present results based on the SLCP benchmark Lest, that show improved discrim- inating power of the ratio coverage. In this section, we present analog findings based on the two moons benchmark test, which is lower dimensional compared to SLCP, but its shape is more difficult to learn.
Figure 9 and 10 confirm the findings of the main paper for the two moons task. First, Figure 9 clearly shows improved performance for the regularized based on the classical coverage, but Figure 10 shows that the performance should actually be worse. Hence, the same blind spot problem arises here as discussed for SLCP, but the ratio coverage was again not effected.
A.4 Variance of Coverage plots
In this scction we investigate the stability of the expected conditional and unconditional coverage plot (see Algorithm 2 and 1). The stability and computational cost of these coverage plots depends on the number of samples n and m, where is related to the gramularity of the coverage plot and m is associated with the stability. In terms of computational simulation cost, the expected conditional coverage generates x m simulations while the unconditional coverage only requires n+ m samples. To investigate the stability between both methods, we keep n- = 1024 the same and consider m = 128 and m = 512.
In Figure 11 we show the standard deviation across 10 rocvaluations of the classical coverage and ratio coverage using the expected conditional and unconditional coverage plot. The evaluated model was previously shown in Figure 5 and 7 in the main text. The expected conditional and unconditional coverage plot agree in
24

---

Published in Transactions on Machine Learning Research (03/2026)
27
128
210
1024
27
128
210
1024
withont regularizaton Expected coverage
Expected coverage
with regularization Expected coverage
Expected coverage
0.5
0.5
0.5
0.5
0.5 Credible level
0.5
Credible level
Credible level
Credible level
Figure 9: This figure compare the classical coverage (orange) and the ralio coverage (blue) for TV regularized and unregularized models trained with simulation budgets of 27, 2,000 This plot is analog to Figure 5 in the main paper.
27 128
2 1024
without regularization
with regularization
Figure 10: These plots reveal the devastating effect of the TV regularization despite its apparent positiv effect on the classical coverage. They show the true and model posterior distribution for the two moons benchmark task with simulation budgets 27, 20 with and without TV regularization. This plot is analog Figure 4 in the main paper.
all cases, which shows the unconditional coverage as more cost efficient alternative. The standard deviation of both methods also roughly agrees, while the standard deviation is significantly decreased for the 512 sample case.
A.5 Simple illustration for coverage
In this section we illustrate the interpretability of the classical and the ralio coverage plot expanding on the blind-spot problem shown in Figure/ Fundamentally, the coverage plot is a statistical tool to measure the similarity between true and model distribution without direct access the true distribution. However, for casier visualization we considering simple one-dimensional normal distributions p and q (shown blue and red).
25

---

Published in Transactions on Machine Learning Research (03/2026)
classical coverage SLCP
ratio coverage SLCP
expected conditional
128 samples
Expected cav goverage
unconditional
Expecte
0.2
0.2
0.5
Credible level
Credible level
classical coverage std SLCF
ratio coverage std SLCP
expected conditional
512 samples
Expected coverage
0.8
Expected coverage
0.8
unconditional
0.6
0.6
0.4
0.4
0.2
0.5
0.5
Credible level
Credible level
Figure 11: Plotting the standard deviation of 10 independent coverage plots of the unregularized models depicted in Figure 5 and Figure 7 with budget of 1024 estimated over 128 and 512 test samples. The surrogate model has not been retrained. The blue line corresponds to the expected conditional coverage plot using Algorithm and the red line corresponds to the conditional coverage plot using Algorithm 1.
Figure 12 consists of 5 sub-figures that each investigate a different true distribution, but have the same model distribution. The top, center figure considers the case where both distributions perfectly agree and we observe the expected perfect coverage from the ralio and the classical coverage plot. The second row considers shift between both distributions and both coverage plots show deformation below the optimal diagonal, while the ratio coverage exhibits higher sensitivity, i.e., more distance to the diagonal. The bottom row investigates change in the width of the distribution and for the wider model distribution the classical and the ratio coverage show mirrored behavior. However, both coverages detect the disagreement between the distributions, because the distance from the diagonal is the similar, but the direction is different. Although, in these simple cases both coverages are able to detect the difference between the distributions, this does not hold general, because of the blind spot of ratio coverage illustrated in Figure 1.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Expected Covergee
71554
deza conorass
Credible level
p(0]x=D)
Seusoe papodda
prooode Sheapws
0.3
Credible level
Creditle leve
()E(x=0)
u(Bix-0)
6.5
2.5
level
Figure 12: Plotting multiple simple cases to illustrate how the coverage behaves in comparison to the classical coverage plot. One can observe that for the left and right shifted case, the ratio coverage behaves similar to the classical coverage but is more sensitives. For the sharper and wider distribution example one can observe that the classical and ratio coverage agree almost perfect, except for the wider model where the ratio coverage gives the rotated result, which corresponds to the choice of the ration & versus 4
27


Reviews:
[{"review_id": "aSKgvg2b5n", "reviewer": "Reviewer_ojnF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:** This paper addresses the challenge of evaluating posterior approximations in SBI, as the ground-truth posterior is not available. In this context, classical metrics based on two-sample testing (MMD, C2ST, etc.) are not applicable. In the SBI literature, one solution is typically to perform expected coverage tests (ECT, a.k.a. simulation-based calibration - SBC), but it is common knowledge that these tests do not present a necessary condition to guarantee the correctness of the posterior approximation. \nThe authors mention the well known \"prior\" blind-spot (the prior will always pass the ECT) and additionally claim to have found a second blind spot, namely that the ECT can even pass for approximations that put mass in unwanted regions-where the ground-truth posterior has none. Still, many works have focused on regularizing SBI methods to enforce good expected coverage. In light of this trend, the authors propose an alternative regularization method with the same purpose. \nHowever, the main contribution of this paper is a new way to assess coverage that addresses the blind-spots of the classical ECT. The authors introduce the notion of \"ratio coverage\" and claim that it is \"discriminative\", i.e. a sufficient condition to guarantee the correctness of the posterior approximation. However, it comes at the cost of estimating the ratio between the approximated and true posteriors (e.g. by training a binary classifier), which is the main limitation of the proposal.\n\n**Key strengths:**\n- The authors address an important topic in SBI\n- The problem statement is well formulated in the introduction\n- The authors provide rigorous mathematical definitions of unconditional and expected conditional coverage\n- The main claims are supported with theoretical and empirical evidence through theorems and numerical experiments\n\n**Key weaknesses:**\n- The paper is not easy to read (especially the theoretical sections)\n- Some major clarifications are needed to understand the main contribution - ratio coverage: how it is indeed discriminative and how it works in practice \n- The experiment section lacks comparisons to existing work\n- Some key references are missing (local coverage tests)", "claims_and_evidence_yn": "No", "claims_explanation": "The claims are supported by interesting empirical evidence: the experiments show the \"discriminative\" nature of the proposed ratio-based coverage method, as opposed to the classical coverage, which fails to detect bad posterior approximations. They also show how their regularization improves the expected coverage, but can hurt the accuracy of the learned posterior. However some clarifications about the experimental setup and more exhaustive experiments about the sensitivity of the method would make it more convincing. \n\nThe claims are also supported by accurate theoretical evidence. However, the lack of clarity makes them less convincing. \n\nI have some important questions that i'd like to be addressed!", "audience_yn": "Yes", "audience_explanation": "I think people working in the field of SBI would find this work interesting, due to its new perspective on the very popular coverage plots and their interesting and enlightening empirical findings about how regularization can hurt accuracy.", "requested_changes": "### 1. Major clarifications about the proposal\n- When introducing KL, KS and TV, it is not clear why and where this is going to show up / to be relevant  in the rest of the article. I suggested making that more explicit. To be more specific: it would be important to clarify / explicit the relationship between the ratio coverage and  the other divergences. The authors mention it at several places, but never in a concise and clear way. Also, the KS only shows up in the appendix. I would have liked to have a clear paragraph that gives the reader a reference point to hold on to and that guides him through the ideas of the paper.\n- I like the example 3.2, it is crucial to understand what's happening. Here, the authors mention the blind-spot as a consequence to choosing $g=q$. Could the authors explain further why that happens? \n- In section 5, the authors mention that in practice, one has to train a classifier to estimate the ratio. They also state that estimating the unconditional ratio coverage plot is simpler (in terms of derivations and computational cost) than the expected conditional one. I understand what they mean, but it would be good for readers to have more explanations on how they do that and what in what setting unconditional coverage is useful / interesting to look at. If I'm not mistaken, the unconditional version is not used in the experiments. Why is that?\n- Again in Section 5, the authors make a comment about the NPE objective being an upper bound to the adversarial ratio regularizer introduced in Section 4, which can thus be \"dropped from the NPE training objective\". Does this mean that the regularizer has no effect in the NPE case? Or even that it harms optimization in terms of improving the accuracy of the posterior approximation? I believe this is shown in the experiments in Section 6.1. If that's the case it would be good to refer to those. As is, it is unclear why they make that comment. \n\n### 2. Minor clarifications about the proposal\n- In remark 3.6 the authors relate the coverage gap to the total variance. First, I understand that this is important for section 4., but could also be put in the appendix. Second, aren't equations (19) and (22) the same thing? I thought this was a direct consequence of Theorem 2.4 in Section 2. Or am I missing something?\n- In section 4, the authors mention that $f$ can be restricted to *sufficiently flexible neural network with tanh outputs*. What does *sufficiently flexible* mean, and why is that important for us? Why tanh outputs? I don't know how relevant these informations are for the main paper, but if the authors are going to give the reader this information, they should also say why.\n- In Definition 3.1, the authors mention that $g$ is a *discriminating function*, without specifying what that is supposed to mean or how relevant that is. I suggest clarifying that. Same goes for the mention of the super-level set, even if some intuition is given in the example below. \n- Definition 3.4 assumes $P$ and $Q$ to have the same marginals. I know it is a classical assumption in SBI, but in might be worth mentioning why that is (in a footnote or so).\n\n### 3. More intuition and accessibility for theoretical results\n- In Section 3, is the main reference for definitions (with choices of notations and names for used quantities) always Hermans et al.? Or are the authors taking liberties and / or notations from other references? Given the title of the section \"*generalizing* coverage plots\", I would expect they are taking liberties, but it would be good to know what their additions are w.r.t. what already exists in the literature. \n- I find Section 3.1 very math heavy and not easy too understand for the general SBI community.  I believe in the main paper, things should be explained in a less heavy, more intuitive and more practical way. The math can be put in the appendix for people who are looking for a deeper understanding.\nFor instance, I like the remark 3.3. about how the plots are computed with samples, but it is still not very easy to digest and not obvious how things are done in practice. The authors refer to Algorithm 2 for more information, which I find very helpful, especially to better understand what is plotted (and against what) in the end. Why not put the algorithm here? Or at least explain how the plotting in algorithm compares to what is explained in remark 3.3 (instead of computing the super-level sets twice, for $P$ and $Q$, algorithm computes it only once, for $Q$, as samples from $P$ are used to define the thresholds, meaning that the number of samples from $P$ above those thresholds will be in the range $[1,N]$). \n- Again, in definition 3.4, I find the notations not trivial and difficult to understand. I believe it could be formulated much easier or authors could give more intuition (e.g. $S$ is related to the c.d.f. !). The authors clarify things in Example 3.5 by saying that expected conditional coverage, as the name indicates, is just taking the expectation of the conditional coverage over $x$. Why not reflect that in the math by using (conditional) expectations? (i believe this what Zhao et al. did in their definition of Local Coverage Plots for example)\n- Theorem 5.1 - in particular Equations (19) and (20) - is the main result and main contribution of this paper from my point of view. It would be valuable to at least have a sketch of the proof here in the main part of the paper, or some intuition on why that is true. \n- In the inequalities in Theorem 5.1. the authors use a new notation ($g$ _# $P$) without introducing it. This notation is only introduced in the Appendix, but should be introduced when they first use it in the main paper. Also, it is not clear to me what it means. This made understanding the proof very difficult. On that note, I second my comment from before about wanting an another version of the proof that is easier to understand / more intuitive.\n\n### 4. Experiments\n- Have the authors compared their method (TV-regularization) to other regularization techniques, like the one from Hermans et al. (or the other techniques mentioned in the introduction)?\n- The problem pointed out in 6.1 about regularized models not improving the posterior approximation for larger training sets is very interesting! I really like how visible this is in the diagonal subplots from Fig 4. where the regularization clearly results in a \"mode collapse\" (for 1,2 and 3 modes corresponding to diagonal subplots 2,3 and 4). I suggesting highlighting that by referring to the figure! \n- Another question about this blind-spot detected in 6.1: it concerns the \"prior\" blind-spot that is very well known in SBI. However the authors claimed to have found another blind-spot, which they illustrate in Figure 1. Could that also happen with their regularization technique? Do the authors have any idea what example one could use to show that empirically? \n- In 6.2, how many samples are used to train the classifier? How sensitive is the method to the sample size? Some additional experiments, similar to what has been done in Linhart et al. for L-C2ST (or just referring to their work), would be important to convince the readers about the effectiveness of the proposed method.\n- It would be interesting to compare the unconditional and the expected conditional ratio coverage plots. Why did the authors choose not to do that? \n\n### 5. Related work\n- The paper seems to be missing an important part of the literature on _local_ coverage tests (see for example Zhao et al., 2021: Diagnostics for Conditional Density Models and Bayesian Inference Algorithms). It would be important for the authors to discuss how their approach relates to these local coverage tests, both conceptually and practically, and to clarify what is genuinely new compared to that line of work. Relevant references seem to be missing.\n- The authors highlight the related work form Linhart et al., L-C2ST, which they explain is the same thing but defined differently (not in terms of coverage). Linhart et al. talk about \"consistency\", which they define as $p=q$  a.e. Isn't that closely related to coverage? From what i understand from this paper, this is what perfect coverage is aiming for and what they claim providing with the ratio coverage.  In fact, couldn't the ratio function  introduced in this work be used for exactly the same purpose as the L-C2ST classifier? And reversely, couldn't the L-C2ST classifier be used as the $g$ function in the ratio plot? In other words, isn't it exactly the same thing, as the discriminator function of the classifier implicitly defines the ratio between p and q? Finally, the authors say that ratio coverage implies passing L-C2ST for all $x_0$. Is that so? Could the authors clarify? Only for the $x_0$s used in the ratio coverage plot, no? It seems obvious to me that the reverse is true aswell. If their statement is true, then that confirms that both methods are equivalent, just introduced in a different context and for a different purpose. \n- In section 5, the authors conclude with a comment on ratio coverage seen from a hypothesis testing point of view, which appears to be very relevant as they claim learning the most powerful test statistic. This paragraph lacks references to strengthen their argument. Also I suggest highlighting / developing this comparison. For example it is unclear to me what the authors mean by \"our test should determine which of them is more likely\". How does this relate to the coverage plot? I can think of an interpretation, but it would be nice to give the reader more intuition.\n\n### 6. Typos and referencing issues\n- In 3.2, the authors refer to a \"Theorem 3.2\". Am I missing something or does this theorem not exist? Also, I believe there is a typo or something missing in the first paragraph when the authors write \"(theorem 3.2), which is used in ...\", this is not a real sentence.\n- In Section 5, the authors mention a Theorem A.4 that doesn't exist. maybe they meant Theorem 2.4 or Remark A.4? Also if they mention it, they should say why / what the observation is that they are referring to.\n-  Figure 2 is never mentioned!\n- I guess the \"s\" is missing in the section 6 title \"Experiment\"?", "broader_impact_concerns": "NA", "additional_comments": "I understand that this review is long and that not all the questions can be answered in detail. My main point is: the proposal is interesting. However, it currently requires significant clarification, a more complete discussion of related work (in particular local coverage plots), and a major effort to improve clarity and accessibility before it can be considered a strong and impactful contribution.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VRxGnfv0ip", "reviewer": "Reviewer_LAwX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the assessment of posteriors obtained by simulation-based inference (SBI) methods. While several approaches have been proposed for this purpose, including simulation-based calibration, there remain unresolved issues. To address these, the authors make three main contributions: (i) they provide a general definition of (un)conditional expected coverage, (ii) they propose differentiable objectives that encourage coverage, and (iii) they present an alternative metric of convergence based on density ratios. Finally, the authors demonstrate the effectiveness of their methods through numerical experiments.", "claims_and_evidence_yn": "No", "claims_explanation": "Strength 1:\nThe discriminative property of an evaluation criterion is critically important; therefore, the problem tackled in this paper is highly relevant and significant.\n\n\nWeakness 1:\nI encourage the authors to add a brief review of expected coverage concepts in the previous literature. In the current manuscript, the explanation in Section 3 feels somewhat abrupt. While Example 3.2 shows that the proposed general definition is a natural extension of the classical one, additional contextualization would improve clarity and accessibility.\n\nWeakness 2:\nI have strong concerns regarding the practical usefulness of the proposed ratio coverage. As the authors note, ratio coverage requires additional training of a density ratio estimator for evaluation, crucially after observing the data. An important aspect of SBI evaluation is the rapid assessment of posterior quality given observed data. The proposed evaluation metric significantly lacks this property, which makes it appear impractical despite its discriminative power.", "audience_yn": "Yes", "audience_explanation": "Please see Strength 1.", "requested_changes": "Major points are described in Weakness 1 and Weakness 2. \n\nMinor points:\n\nSec 3.2: theorem 3.2 => Example 3.2 ?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wMBMQVLWQJ", "reviewer": "Reviewer_vyzo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigate methods of checking the correctness of approximations to the Bayesian posterior distribution based on \"coverage\" (also commonly known as calibration in the literature). Two types of coverage are given a rigorous and general definition. The authors investigate two particular versions: \"classical coverage\" and \"ratio coverage\". They also present an objective function based on an adversarial form of coverage. The adversarial approach is shown to improve classical coverage but nonetheless provide a poor fit to the posterior. Ratio coverage can detect this poor fit, and the authors also prove it has some more desirable theoretical properties: perfect coverage is attained if an only if the approximation equals the posterior.\n\n## Strengths\n\n* A coverage/calibration method avoiding several failure modes.\n\n* Detailed proofs\n\n* Examples clearly demonstrating main points of the paper.\n\n## Weaknesses\n\n* It would be good to further discuss \"Discriminative Calibration: Check Bayesian Computation from Simulations and Flexible Classifier\" by Yao and Domke (NeurIPS 2023). It seems possible to me that the proposed ratio coverage method is a special case of Yao and Domke's work.\n\n* More details are needed on how the ratio coverage classifier is implemented, especially how training data is generated. (These details would have helped me check the previous point.)\n\n* The paper only looks at two simple benchmark models.\n\n* It would be good to give more details of computational cost.\n\n* The paper has some grammar mistakes which occasionally make it hard to follow.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There are detailed and rigorous mathematical definitions and proofs. The examples are excellent, supporting the paper's claims about which methods work well or poorly.", "audience_yn": "Yes", "audience_explanation": "Section 4 proposes a novel adversarial regularisation method for NPE, but goes on to show (in Section 6) this has very poor performance. So this method seems of limited interest to readers. Perhaps the authors can add extra justification of why it's important.\n\nThe proposed ratio coverage method seems useful and of interest to TMLR's audience.", "requested_changes": "## Critical\n\n* Can you give more details on the computational cost of the proposed methods, especially training the ratio estimator.\n\n* Further discuss \"Discriminative Calibration: Check Bayesian Computation from Simulations and Flexible Classifier\" by Yao and Domke (NeurIPS 2023). It seems possible to me that the proposed ratio coverage method is a special case of Yao and Domke's work.\n\n* Give more details on how the ratio coverage classifier is implemented, especially how training data is generated. (These details would have helped me check the previous point.)\n\n## Would strengthen work\n\n* It would be ideal to investigate some more examples. But this is not absolutely necessary in my opinion.\n\n* Discuss whether the method is only applicable to SBI, or also has applications to other approximate Bayesian methods.\n\n## Minor\n\n* Use a spelling and grammar checker to fix mistakes. These are mostly minor. An exception is the line before Definition 3.4, where there's a sentence fragment that I can't understand (referring to theorem 3.2, which doesn't appear in the paper).\n\n* Many citations would be better in parentheses.\n\n* Many equations are unnecessarily split over multiple lines, when they would fit on a single line.\n\n* Comment on meaning of $\\mathcal{B_Z}$. Is this an arbitrary $\\sigma$-algebra? If it's meant to be a Borel $\\sigma$-algebra then I think you need to assume something about the topology of $\\mathcal{Z}$.\n\n* On the line before equation (2), I think $\\pm$ is a typo of $\\pm \\infty$.\n\n* In Definition 3.4, do you need any extra assumptions to ensure $Q(A | X=x)$ is well defined?\n\n* Page 6 says \"we show that the expected conditional coverage gap... is bounded above by the total variation norm\". Wasn't this already shown in Equation (19)?\n\n* The paragraph before equation (25) mentions that regularity conditions are needed for this result. Can you expand on what these are, and give a citation for where a proof can be found.\n\n* Page 6 says \"...ratio coverage is based on the observation in Theorem A.4\". Can you give more details, as the theorem statement is long and makes several observations.\n \n* On page 7, the paragraph starting \"We proofed\" seems to repeat some material from Example 3.5. If this is intended, perhaps you could add a comment that you're recapping this.\n\n* On page 7, is \"expected unconditional coverage\" a typo? Earlier I think you only define \"unconditional coverage\" and \"expected conditional coverage\".\n\n* Appendices A.4 and A.5 should contain some text to explain why you include Figures 10 and 11.", "broader_impact_concerns": "In my opinion, the paper's current Broader Impact Statement is sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)