Title: Discriminative Calibration: Check Bayesian Computation from Simulations and Flexible Classifier
Venue: NeurIPS 2023
Accepted: yes
Keywords: simulation based calibration

simulation based inference

Bayesian computation

diagnostics

classifier two-sample test

likelihood-free
Abstract: To check the accuracy of Bayesian computations, it is common to use rank-based simulation-based calibration (SBC). However, SBC has drawbacks: The test statistic is somewhat ad-hoc, interactions are difficult to examine, multiple testing is a challenge, and the resulting p-value is not a divergence metric. We propose to replace the marginal rank test with a flexible classification approach that learns test statistics from data. This measure typically has a higher statistical power than the SBC test and returns an interpretable divergence measure of miscalibration, computed from classification accuracy. This approach can be used with different data generating processes to address simulation-based inference or traditional inference methods like Markov chain Monte Carlo or variational inference. We illustrate an automated implementation using neural networks and statistically-inspired features, and validate the method with numerical and real data experiments.

Discriminative Calibration: Check Bayesian Computation from Simulations and Flexible Classifier

Abstract

To check the accuracy of Bayesian computations, it is common to use rank-based simulation-based calibration (SBC). However, SBC has drawbacks: The test statistic is somewhat ad-hoc, interactions are difficult to examine, multiple testing is a challenge, and the resulting p-value is not a divergence metric. We propose to replace the marginal rank test with a flexible classification approach that learns test statistics from data. This measure typically has a higher statistical power than the SBC test and returns an interpretable divergence measure of miscalibration, computed from classification accuracy. This approach can be used with different data generating processes to address simulation-based inference or traditional inference methods like Markov chain Monte Carlo or variational inference. We illustrate an automated implementation using neural networks and statistically-inspired features, and validate the method with numerical and real data experiments.

1 Introduction

Simulation based calibration (SBC) is a default approach to diagnose Bayesian computation. SBC was originally designed to validate if computer software accurately draws samples from the exact posterior inference, such as Markov chain Monte Carlo (MCMC, [4, 38, 26]) and variational inference [41]. With recent advances in amortized and simulation-based inferences [6] and growing doubt on the sampling quality [23, 18], there has also been an increasing trend to apply SBC to likelihood-free inference such as approximate Bayesian computation [42] and normalizing-flow-based [31, 1] neural posterior sampling [22, 20], with a wide range of applications in science [15, 7, 34].

Bayesian computation tries to sample from the posterior distribution p ( θ | y ) given data y . We work with the general setting where it may or may not be possible to evaluate the likelihood p ( y | θ ) . Suppose we have an inference algorithm or software q ( θ | y ) that attempts to approximate p ( θ | y ) , and we would like to assess if this q is calibrated, meaning if q ( θ | y ) = p ( θ | y ) for all possible θ, y . Simulation based calibration involves three steps: First, we draw a θ from the prior distribution p ( θ ) . Second, we simulate a synthetic observation y from the data model p ( y | θ ) . Third, given y we draw a sizeM posterior sample ˜ θ 1 , . . . , ˜ θ M from the inference engine q ( θ | y ) that we need to diagnose. SBC traditionally computes the rank statistic of the prior draw θ among the q samples, i.e. r = ∑ M m =1 ✶ ( θ ≤ ˜ θ m ) . If the inference q is calibrated, then given y both θ and ˜ θ m are from the same distribution p ( θ | y ) , hence with repeated simulations of ( θ, y ) , we should expect such rank statistics r to appear uniform, which can be checked by a histogram visualization or a formal uniformity test.

Despite its popularity, rank-based SBC has limitations: (i) We only compute the rank of univariate parameters. In practice, θ and y are high dimensional. We typically run SBC on each component of θ separately, this creates many marginal histograms and does not diagnose the joint distribution or interactions. We may compare ranks of some one-dimensional test statistics, but there is no method to

θ

i

y

i

˜

θ

i

1

˜

θ

i

2

· · ·

˜

θ

iM

simulation table

classification examples

label mapping

Φ

t

1

, ϕ

1

t

2

, ϕ

2

· · ·

t

K

, ϕ

K

.

.

.

.

.

.

.

.

.

.

.

.

i

= 1

, . . . , S

classifier

feature engineering

log

p

, log

q

, rank, etc.

prediction

loss

divergence

D

and its C.I.

permutation

test (

p

value)

find the best summary test statistic. (ii) As long as we test multiple components of θ or test statistics, directly computing the uniformity p -values is invalid (subject to false discovery) unless we make a multiple-testing adjustment, which drops the test power (subject to false negative) in high dimensions. (iii) Often we know inference is approximate, so the final goal of diagnostic is not to reject the null hypothesis of perfect calibration but to measure the degree of miscalibration . The p -value is not such a measure: neither can we conclude an inference with p = . 02 is better than p = . 01 , nor connect the p -value with the posterior inference error. The evidence lower bound, a quantity common in variational inference, also does not directly measure the divergence due to the unknown entropy.

Heuristic: calibration via classification. To address all these drawbacks, while maintaining versatility in various computing tasks, we propose discriminative calibration , a pragmatic and unified framework for Bayesian computing calibration and divergence estimation. The intuitive heuristic behind discriminative calibration is to use a classifier to perform similarity tests-when two or several distributions are similar, we cannot distinguish samples from them so the classification error is large. In Bayesian computation, we compare conditional distributions, the true p ( θ | y ) and inferred q ( θ | y ) , to which we have access via simulations, and in some occasions explicit densities. It is natural to try to classify the samples drawn from p v.s. from q , and the ability to distinguish them would suggest a miscalibration between p and q .

To formulate such heuristics into an algorithm, we design a family of 'label mapping' that prepares the simulations into classification examples that contain the label and feature. In Sec. 2, we first give four concrete label mapping examples, where the rank-based SBC becomes essentially a special case. Sec. 3 states the general theory: if we train a classifier to predict the labels, then the prediction ability yields a computable divergence from p ( θ | y ) to q ( θ | y ) . In Sec. 4, we illustrate the practical implementation to get the divergence estimate, its confidence interval, and a valid hypothesis testing p -value. We explain why the learned classifier helps statistical power. In Sec. 4.1, we discuss the classifier design and incorporate extra known information such as ranks, likelihood, and log densities, whenever available, as features. We also show our method is applicable to MCMC without waste from thinning. We illustrate numerical and cosmology data examples in Sec. 5. We review other related posterior validation approaches and discuss the limitation and future direction in Sec. 6.

2 Generate labels, run a classifier, and obtain a divergence

As with traditional SBC, we generate a simulation table by repeatedly sampling parameters θ and synthetic data y from the target model p , i.e. draws ( θ, y ) ∼ p ( θ, y ) . Then, for each ( θ, y ) , we run the inference routine q to obtain a set of M IID approximate posterior samples ˜ θ 1 , · · · , ˜ θ M ∼ q ( θ | y ) . We wish to assess how close, on average (over different y ), the inference procedure q ( θ | y ) is to the true posterior p ( θ | y ) . Here, we observe that classification example-sets can be created in several ways and these produce different divergences. We generalize the four examples and the claims on divergence in Sec. 3.

Example 1: Binary classification with full features. An intuitive way to estimate this closeness between q ( θ | y ) and p ( θ | y ) is to train a binary classifier.

Input :

one simulation run:

(

y, θ,

˜

θ

1

, . . . ,

˜

θ

M

)

;

Output :

M +1 examples:

label

t

features

ϕ

0

(

θ, y

)

1

( ˜

θ

1

, y

)

1

· · ·

1

( ˜

θ

M

, y

)

Imagine creating a binary classification dataset of ( t, ϕ ) pairs, where t is a binary label, and ϕ are features. For each ( θ, y ) simulated from p , M +1 pairs of examples are created. In the first, t = 0 and ϕ = ( θ, y ) . In the others, t = 1 , and ϕ = ( ˜ θ m , y ) , 1 ≤ m ≤ M . Collecting all data across

1 ≤ i ≤ S , we obtain S ( M + 1) pairs of ( t, ϕ ) classification examples. A binary classifier is then trained to maximize the conditional log probability of t given ϕ . If inference q were exact, no useful classification would be possible. In that case, the expected test log predictive density could be no higher than the negative binary entropy h ( w ) := w log w +(1 -w ) log(1 -w ) of a Bernoulli distribution with parameter w := 1 / ( M +1) .

Now imagine drawing a validation set in the same way, and evaluating the log predictive density of the learned classifier. We will show below (Thm. 1) that the expected log predicted density [10] ELPD = E log Pr( t | ϕ ) of the classifier on validation data is a lower bound to a divergence D 1 between p ( θ | y ) and q ( θ | y ) up to the known constant h ( w ) ,

where w = 1 / ( M +1) and r ( θ | y ) = wp ( θ | y ) + (1 -w ) q ( θ | y ) is a mixture of posterior density. If the classifier c is optimal ( c ( t | ϕ ) = Pr( t | ϕ ) in the distribution) then the bound in the above equation is tight max classifiers ELPD 1 -h ( w ) = D 1 ( p, q ) . Here KL( p ( θ | y ) ∥ q ( θ | y )) denotes a standard conditional Kullback-Leibler divergence 1 . By optimizing the classifier, maxELPD 1 -h ( w ) becomes a commutable divergence, and its approaching zero is a necessary and sufficient condition for perfect calibration since D 1 ( p, q ) = 0 if and only if p ( θ | y ) = q ( θ | y ) almost everywhere.

Example 2: Binary classification without y . Similar to Example 1, from each simulation draw we generate M + 1 pairs of ( t, ϕ ) examples, except that the feature ϕ only contains the parameters θ , not y . A binary classifier is then trained to predict t given ϕ . The ELPD of this classifier on validation data is a lower bound to a generalized divergence D 2 between the prior p ( θ ) and q ( θ ) , up to a known constant h ( w )

t

ϕ

0

θ

1

˜

θ

1

1

·

1

˜

θ

M

(2)

where w = ( M +1) -1 , r ( θ ) = wp ( θ )+(1 -w ) q ( θ ) is the prior mixture, and the bound is tight when the classifier is optimal. A large ELPD reveals the difference between the inference q ( θ | y ) and p ( θ | y ) . But D 2 is only a generalized divergence: D 2 = 0 is necessary not sufficient for q ( θ | y ) = p ( θ | y ) .

Example 3: Binary classification with ranks (where the classical SBC is a special case). Instead of classification using full ( θ, y ) , we construct a feature : the rank statistics. From each simulation draw we generate M + 1 pairs of ( t, ϕ ) examples. The first pair is t = 0 and ϕ = ∑ M m =1 ✶ ( θ < ˜ θ m ) , the rank statistics of the prior draw. In the others, t = 1 and ϕ = ∑ M m ′ =1 ✶ ( ˜ θ m < ˜ θ m ′ ) + ✶ ( ˜ θ m < θ ) , 1 ≤ m ′ ≤ M are the rank statistics of the inferred samples. A binary

Output

:

M

+ 1

examples:

t

ϕ

0

∑

M

m

=1

✶

(

θ

˜

θ

m

)

1

∑

M

m

′

=1

✶

( ˜

θ

1

˜

θ

m

′

)

1

· · ·

1

∑

M

m

′

=1

( ˜

θ

M

˜

θ

m

′

)

✶ classifier is then trained to predict t given ϕ . The ELPD of this classifier is a lower bound to a generalized divergence D 3 between p ( θ | y ) and q ( θ | y ) up to a known constant h ( w )

and again the bound is tight if the classifier is optimal. Here Z ( p, q ) is a random variable defined by Z = Q ( θ | y ) , ( θ, y ) ∼ p ( θ, y ) , where Q is the cumulative distribution function of q ( θ | y ) .

Training a 'generative' classifier on this rank-based label-generating map is similar to testing for uniformity in rank statistics, as done in traditional SBC which estimates the distribution of r | t = 0 by histograms (See Appendix A.1 for precise correspondence between SBC and the naive Bayes classifier). The success of SBC suggests the usefulness of ranks, which motivates us to include ranks or more generally feature engineering in the classifier. However, D 3 is only a generalized divergence: D 3 = 0 is necessary but not sufficient for p ( θ | y ) = q ( θ | y ) . If inference always returns the prior, q ( θ | y ) = p ( θ ) , then D 3 ( p, q ) = 0 , a known counterexample of when rank-based SBC fails [32].

Example 4: Multi-class classification. We go beyond binary labeling. Given a simulation run ( y, θ, ˜ θ 1 , · · · , ˜ θ M ) , we now create an ( M +1)-class classification dataset with M + 1 pairs of ( t, ϕ ) examples. In each one, the features are ϕ = ( y, θ ∗ ) where θ ∗ is a permutation of ( θ, ˜ θ 1 , · · · , ˜ θ M ) that moves θ into a given location and t ∈ 0 , · · · , M indicates the location of θ in the permutation

Output

:

M

+ 1

examples:

t

ϕ

0

(

θ,

˜

θ

1

,

˜

θ

2

,

· · ·

,

˜

θ

M

, y

)

1

( ˜

θ

1

, θ,

˜

θ

2

· · ·

,

˜

θ

M

, y

)

·

· · ·

M

( ˜

θ

1

,

˜

θ

2

· · ·

,

˜

θ

M

, θ, y

)

(See the table on the right). We train a ( M +1)-class classifier to predict t from ϕ . The ELPD on a validation set is a lower bound to the following divergence D 4 between p ( θ | y ) and q ( θ | y ) up to a known constant:

̸

Again the bound is tight if the classifier is optimal. The divergence 0 ≤ D 4 ≤ log( M +1) , and D 4 = 0 if and only if p ( θ | y ) a.e. = q ( θ | y ) , necessary and sufficient for calibration. In Theorem 3 we shows that as M →∞ , D 4 ( p, q ) converges to KL( p ( θ | y ) , q ( θ | y )) at an O (1 /M ) convergence rate.

3 Theory on calibration divergence

To generalize the previous examples, we define a ' label mapping ' Φ : ( y, θ, ˜ θ 1 , . . . , ˜ θ M ) ↦→ { ( t 1 , ϕ 1 ) , . . . , ( t L , ϕ L ) } . that maps one simulation run ( y, θ, ˜ θ 1 , . . . , ˜ θ M ) into a K -class classification example-set containing L pairs of labels t and features ϕ . The label t l ∈ { 0 , 1 , . . . , K -1 } is deterministic and only depends on l . The features ϕ l can depend on l and ( y, θ, ˜ θ 1 , . . . , ˜ θ M ) . We only consider Φ satisfying that, when p ( θ | y ) = q ( θ | y ) , ϕ given y is conditionally independent of t (equivalently, ϕ | ( y, t ) has the same distribution for any t ). Let F be the set of these mappings Φ .

We train a classifier on the classification examples created by Φ collected across repeated sampling. The classifier performance is measured by its expected log predictive density (ELPD), or equivalently the negative cross-entropy loss. Given p , q , and the mapping Φ , let c ( ϕ ) be any probabilistic classifier that uses ϕ to predict the label t , and Pr( k | ϕ, c ) is the predicted k -th class probability. Taking expectations over features and labels with ( θ, y ) ∼ p and ˜ θ m ∼ q ( θ | y ) reaches the ELPD of the classifier,

We then define the prediction ability D ( p, q, Φ , c ) to be the ELPD plus the entropy of a categorical distribution, i.e.

The next theorem is the basic theory of our method: as long as we pass the simulation draws to a label mapping Φ ∈ ❋ , and train a classifier on the classification examples, then D opt ( p, q, Φ) is a generalized divergence between p and q .

Theorem 1 (Prediction ability yields divergence) . Given any p, q , and feature mapping Φ ∈ ❋ , the optimal prediction ability D opt ( p, q, Φ) is a generalized divergence from p to q in the sense that D opt ( p, q, Φ) ≥ 0 , and p ( θ | y ) = q ( θ | y ) almost everywhere implies D opt ( p, q, Φ) = 0 . This generalized divergence is reparametrization invariant and uniformly bounded. For any classifier c ,

That is, any label mapping Φ produces a generalized divergence D opt ( p, q, Φ) , the prediction ability of the optimal classifier. The prediction ability D ( p, q, Φ , c ) of any classifier c estimated on validation data is a lower bound to D opt ( p, q, Φ) . Further, this generalized divergence D opt ( p, q, Φ) is always a proper Jensen-Shannon divergence in the projected feature-label space (Theorem 1 in the Appendix).

̸

When p ( θ | y ) = q ( θ | y ) , to increase the statistical power, we wish that the generalized divergence can be as 'strong' as possible such that we can detect the miscalibration. In the four examples in Section 2, we have used D 1 , D 2 , D 3 , D 4 to denote the (generalized) divergence they yield. The next theorem shows there is a deterministic domination order among these four metrics. Moreover, D 4 is the largest possible classification divergence from any given simulation table.

Theorem 2 (Strongest divergence) . For any given p, q , and any Φ ∈ F , (1) D 4 ≥ D 1 ≥ D 3 ≥ D 2 . (2) D 4 and D 1 are proper divergences. They attain 0 if and only if p ( θ | y ) = q ( θ | y ) almost everywhere. They attain the corresponding upper bound in (8) if and only p ( θ | y ) are q ( θ | y ) are disjoint, i.e., ∫ A p ( θ | y ) q ( θ | y ) dθ = 0 for any measurable set A and almost surely y . (3) For any p, q and Φ ∈ F ( Φ can have an arbitrary example size L and class size K ), D 4 ( p, q ) ≥ D opt ( p, q, Φ) .

The following result shows that the divergence D 4 in Eq. (4) approaches the 'mode-spanning' KL divergence in the limit that the number of posterior draws M →∞ . This is appealing because for many inference routes, increasing the number of posterior draws is cheap. Thus, D 4 provides an accessible approximation to the KL divergence that is otherwise difficult to compute from samples.

Theorem 3 (Big M limit and rate) . For any p, q , generate the simulation { ( y, θ, ˜ θ i , . . . , ˜ θ M ) } and train the multiclass-classier, then D 4 ( p, q ) -KL( p ( θ | y ) || q ( θ | y )) → 0 , as M →∞ .

If further p ( θ | y ) and q ( θ | y ) have the same support, and if E p ( θ | y ) [ p ( θ | y ) q ( θ | y ) ] 2 < ∞ for a.s. y , then

where χ 2 ( ·||· ) is the conditional chi-squared divergence.

4 Practical implementation

Algorithm 1: Proposed method: Discriminative calibration

input : The ability to sample from p ( θ, y ) and q ( θ | y ) , and a label mapping Φ . output : (i) estimate of a divergence between p ( θ | y ) and q ( θ | y ) ; (ii) p -value for testing p ( θ | y ) = q ( θ | y ) . for ( i = 1 : S )

Sample ( θ i , y i ) ∼ p ( θ, y ) , and sample ˜ θ i 1 , ˜ θ i 2 . . . , ˜ θ iM ∼ q ( θ | y i ) ; ▷ simulation table Generate a batch of L examples of ( t, ϕ ) = Φ( y i , θ i , ˜ θ i 1 , . . . , ˜ θ iM ) , 0 ≤ t ≤ K -1 ; ▷ label mapping Randomly split the LS classification examples ( t, ϕ ) into training and validation sets (all L examples for a given i go to either training or validation);

Train a K -class classifier to predict label t on the training examples, incorporating useful features; Compute the validation log predictive density LPD val in (9), obtain an estimate of the divergence (7) and its bootstrap confidence intervals; ▷ divergence for (

b

= 1 :

B

)

Randomly permute the label t in the validation set within each batch;

Compute

LPD

b

val on the permutated validation set;

Compute the calibration p -value p = 1 /B ∑ B b =1 ✶ (LPD val b ≥ LPD val ) .

▷ frequentist test

Workflow for divergence estimate. We repeat the simulation of ( y, θ, ˜ θ 1 , . . . , ˜ θ M ) for S times. Each time we sample ( y, θ ) ∼ p ( θ, y ) and ˜ θ 1: M ∼ q ( θ | y ) and generate a batch of L examples through a label mapping Φ ∈ ❋ . In total, we obtain SL pairs of ( t, ϕ ) classification examples. We recommend using the binary and multiclass labeling schemes (Examples 1 and 4, where L = M +1 and K = 2 or M +1 ) such that we can obtain a proper divergence estimate. We split the classification example-set { ( t, ϕ ) } into the training and validation set (do not split batches) and train a K -class classifier c on the training set to minimize cross-entropy. Denote I val to be the validation index, Pr( t = t j | ϕ j , c ) to be the learned class probability for any validation example ( t j , ϕ j ) , we compute the ELPD (5) by the validation set log predictive density:

For any c , LPD val (Φ , c ) -∑ K -1 k =0 w k log w k becomes a lower bound estimate of the (generalized) divergence D opt ( p, q, Φ) in Thm. 1, and an estimate of D opt ( p, q, Φ) itself when the classier c is good enough. In addition to the point estimate of the divergence, we can compute the confidence interval. It is straightforward to obtain the standard error of the sample mean (9). To take into account the potentially heavy tail of the log predictive densities, we can also use Bayesian bootstrap [37] that reweights the sum in (9) by a uniform Dirichlet weight.

Hypothesis testing. Our approach facilitates rigorous frequentist hypothesis testing. The null hypothesis is that the approximate inference matches the exact posterior, i.e., p ( θ | y ) = q ( θ | y ) almost everywhere. We adopt the permutation test: We train the classifier c once on the training set and keep it fixed, and evaluate the validation set log predictive density LPD val (Φ , c ) in (9). Next, permutate the validation set B times: at time b , keep the features unchanged and randomly permutate the validation labels t within each batch of examples ( Φ generates a batch of L examples each time), and reevaluate the validation set log predictive density (9) on permutated labels, call it LPD val b . Then we compute the one-sided permutation p -value as p = ∑ B b =1 ✶ (LPD val b ≥ LPD val ) /B . Given a significance level, say 0.05, we will reject the null if p < 0 . 05 and conclude a miscalibration.

Theorem 4 (Finite sample frequentist test) . For any finite simulation size S and posterior draw size M , and any classifier c , under the null hypothesis p ( θ | y ) = q ( θ | y ) almost everywhere, the permutation test is valid as the p -value computed above is uniformly distributed on [0 , 1] .

Our test is exact when the simulation size S and M is finite, while the original SBC [4] relied on asymptotic approximation. Further, we learn the test statistic via the classifier, and our test is valid regardless of the dimension of θ and there is no need to worry about post-learning testing [3], while the usual SBC rank test will suffer from low power due to multiple testing.

Our test is always valid even if the classifier c is not optimal. Why does our learning step help? For notional brevity, here we only reason for the binary classification. For any p, q , we apply the binary classification as described in Example 1, t = 0 or 1 and ϕ = ( θ, y ) .

Theorem 5 (Sufficiency) . Let ˆ c ( θ, y ) = Pr( t = 1 | θ, y ) be the probability of label 1 in the optimal classifier as per (7) , and let π p c and π q c be the one-dimensional distributions of this ˆ c ( θ, y ) when ( θ, y ) is sampled from p ( θ, y ) or from p ( y ) q ( θ | y ) respectively, then (i) Conditional on the summary statistic ˆ c , the label t is independent of features ϕ = ( θ, y ) . (ii) Under regularity conditions, there is no loss of information in divergence as the joint divergence is the same as the projected divergence in the one-dimensional ˆ c -space D 1 ( p, q ) = D 1 ( π p c , π q c ) .

That is, the best prediction ˆ c entails the best one-dimensional summary statistics of the high dimensional θ × y space. The enhancement of the test power from using the sufficient statistics is then assured by the Neyman-Pearson lemma [27].

4.1 Feature engineering: use rank, log density and symmetry

Reassuringly, whatever the classifier c , the prediction ability D ( p, q, Φ , c ) is always a lower bound to the corresponding divergence D opt ( p, q, Φ) , and the permutation test is always exact. But we still wish to train a 'good' classifier in terms of its out-of-sample performance, for a tighter bound in divergence, and a higher power in testing. We will typically use a flexible parametric family such as a multilayer perceptron (MLP) network to train the classifier.

The oracle optimal probabilistic classifier is the true label probability, and in the binary and multiclass classification (Example 1 and 4), the oracle has closed-form expressions, although we cannot evaluate:

Statistically-meaningful feature. Depending on the inference task, we have more information than just the sample points and should use them in the classifier. In light of the shape and component of the optimal classifiers (10), the following statistically-meaningful features are useful whenever available: (i) The log target density log p ( θ | y ) . As proxies, the log joint density log p ( θ, y ) and the log likelihood log p ( y | θ ) are often known to traditional Bayesian models. (ii) The log approximate density log q ( θ | y ) , known to volitional and normalizing flows. (iii) When the log approximate density is unknown, a transformation is to integrate the density and obtain the posterior CDF, Q ( θ | y ) = E x ∼ q ( x | y ) ✶ ( x > θ ) , and this CDF can be approximated by the rank statistics among the approximate draws up to a rescaling r ( θ, y ) := ∑ MS m =1 ,i =1 ✶ ( y i = y ) ✶ ( θ < ˜ θ im ) . See Appendix Table 1 for the usage and extension of these features in binary and multiclass classifiers.

Linear features. We call the log likelihood, the log approximate density, or log prior (whenever available) linear features, and denote them to be l ( θ, y ) . For example if both likelihood and q density

is known, then l ( θ, y ) = (log p ( y | θ ) , log q ( θ | y )) . Because they appear in the oracle 10, we will keep linear features in the last layer of the network (followed by a softmax). We recommend parameterizing the binary classification in the following form:

Symmetry in multiclass classifier. The multi-class classifier is harder to train. Luckily, we can use the symmetry of the oracle classifier (10): the probability of class k is proportional to a function of ( θ k , y ) only, hence we recommend parameterizing the multiclass probability as

where l ( θ, y ) is available linear features. We only need to learn a reduced function from θ × y to R , instead of from θ M +1 × y to R , reducing the complexity while still keeping the oracle (10) attainable.

Zero waste calibration for MCMC. If ˜ θ 1 , . . . , ˜ θ M ∼ q ( θ | y ) are produced by MCMC sampler, typically ˜ θ has autocorrelations. Although classical SBC originated from MCMC application, the rank test requires independent samples, so SBC can only handle autocorrelation by thinning: to subsample ˜ θ m and hope the thinned ˜ θ draws are independent. Thinning is a waste of draws, inefficient when the simulations are expensive, and the thinned samples are never truly independent. With MCMCdraws ˜ θ , our method could adopt thinning as our Thm. 1 and 4 are valid even when M = 1 .

Yet we can do better by using all draws. The 'separable' network architecture (12) is ready to use for MCMC samples. For example, we sample ( θ, y ) ∼ p ( θ, y ) , and sample ( ˜ θ 1 , . . . , ˜ θ M ) from a MCMCsampler whose stationary distribution we believe is q ( θ | y ) , and generate examples from the multiclass permutation (Example 4). Then we run a separable classifier(12) to predict t . Intuitively, the separable network design (12) avoids the interaction between ˜ θ m with ˜ θ m ′ , and disallows the network to predict t based on the autocorrelation or clustering of ˜ θ . The next theorem states the validity of our method in MCMC settings without thinning.

Theorem 6 (MCMC) . Suppose we sample ( θ, y ) ∼ p ( θ, y ) , and sample ( ˜ θ 1 , . . . , ˜ θ M ) from a MCMC sampler whose stationary distribution we believe is q ( θ | y ) (i.e., marginally ˜ θ i is from q ( θ | y ) ), and generate examples (( t 1 , ϕ 1 ) , . . . , ( t M +1 , ϕ M +1 )) from the multiclass permutation, such that ϕ = ( θ 0 , θ 1 , . . . , θ M ) . Then we run an exchangeable classifier (12) in which g is any Θ ×Y → R mapping. Denote D MCMC , sep 4 ( p, q ) to be the predictive ability of the optimal classifier among all separable classifiers (12) , then D MCMC , sep 4 ( p, q ) = D 4 ( p, q ) .

Dimension reduction and nuisance parameter. Sometimes we only care about the sampling quality of one or a few key dimensions of the parameter space, then we only need to restrict the classifier to use these targeted dimensions, as a result of Theorem 1. For example, in binary classification, if we reduce the feature ϕ = ( θ, y ) to ϕ = ( h ( θ ) , y ) in the classifier, where h ( θ ) can be a subset of θ dimensions, then the resulting classification divergence becomes projected divergence between h ( θ ) | y, θ ∼ p ( θ | y ) and h ( θ ) | y, θ ∼ q ( θ | y ) , and other nuisance parameters do not impact the diagnostics.

Weighing for imbalanced binary classification. When M is big, the binary classification (Example 1) can suffer from imbalanced labels, and the divergence in (1) degenerates: D 1 ( p, q ) → 0 as M →∞ . One solution is to use the multiclass classification which has a meaningful limit (Thm. 3). Another solution is to reweigh the loss function or log predictive density by the label t . If the weights of class 1 and class 0 examples are M +1 2 M and M +1 2 , used in both training and validation log prediction density, then regardless of M , the weighted classification is equivalent to balanced classification, and the resulting divergence is the symmetric Jensen-Shannon (JS) divergence 1 2 KL[ p ( θ | y ) || r ( θ | y )] + 1 2 KL[ q ( θ | y ) || r ( θ | y )] , where r ( θ | y ) = 1 2 [ p ( θ | y ) + q ( θ | y )] . See Appendix A for proofs.

5 Experiments

Closed-form example. Consider a multivariate normal parameter prior θ ∈ R d ∼ MVN( 0 , Id d ) and a normal data model y | θ ∼ MVN( θ, Σ) , so the exact posterior p ( θ | y ) is explicit. In the experiments,

0

0.5

1

Our p-values: valid under null

0

0.5

1

SBC w/o  multiple test adj.

10

100

1000

-0.5

0.0

0.5

True JSD

bias = 2

True JSD

bias = 1

finite S sample

estimate

(a negative est. will

be truncated to 0)

S (# of simulatons)

estimated divergence

Binary divergence  = JSD

10

100

1000

0

1

2

3

4

True KL div.

bias = 2

True KL div.

bias = 1

M (# of post. draw per sim.)

Multiclass div. converges to  KL

0.8

1.0

1.2

0.0

0.5

1.0

ours

SBC

SBC w/

10x sample

inflation of post. cov

test power

0.01

0.02

0.05

0.1

0.0

0.5

1.0

ours

SBC

bias of post. mean (log scale)

Our test has a higher test power (easier to detect miscalibration)

0

.1

.2

.3

2

20

200

iterations

SLCP

estimated divergence

0

.1

.2

.3

2

20

200

iterations

Gaussian linear

0

.1

.2

.3

2

20

200

iterations

we use neural nets to train binary (11) or symmetric multiclass classifiers (12), which we generally recommend. We assume the inference posterior log q ( θ | y ) is known and added to the classifier.

Legitimacy of testing . First, to validate our hypothesis test under the null, we use true inference q ( θ | y ) = p ( θ | y ) . With d = 16 , we simulate S = 500 draws of ( θ, y ) ∼ p ( θ, y ) , then generate true posterior ˜ θ m ∼ p ( θ | y ) , and run our binary-classifier and obtain a permutation p -value. We repeat this testing procedure 1000 times to obtain the distribution of the p -values under the null, which is uniform as shown in Fig. 2, in agreement with Thm. 4. Because θ has d dimensions, a SBC rank test on each margin separately is invalid without adjustment, and would require Bonferroni corrections.

Power . We consider two sampling corruptions: (i) bias, where we add a scalar noise (0.01 to 0.2) to each dimension of the true posterior mean, (ii) variance, where we inflate the true posterior covariance matrix by multiplying a scalar factor (0.8 to 1.2). We compare our discriminative tests to a Bonferroni-corrected SBC chi-squared rank-test. In both settings, we fix a 5% type-I error and compute the power from 1000 repeated tests. Fig. 3 shows that our method has uniformly higher power than SBC, sometimes as good as SBC with a 10 times bigger simulation sample size.

Divergence estimate . The left panel of Fig. 4 validates Thm. 1: We run a weighted binary classifier on the Gaussian simulations with a scalar bias 1 or 2 added to the posterior mean in q . As the number of simulations S grows, the learned classifier quickly approaches the optimal, and the prediction ability matches the theory truth (dashed line): the Jensen-Shannon divergence between p ( θ | y ) and q ( θ | y ) . The right panel validates Thm. 3: With a fixed simulation size S , we increase M , the number of posterior draws from q , and run a multiclass classification. When M is big, the estimated divergence converges from below to the dashed line, which is the theory limit KL ( p ( θ | y ) , q ( θ | y )) in Thm. 3.

Benchmark examples. Next, we apply our calibration to three models from the SBI benchmark [23]: the simple likelihood complex posterior (SLCP), the Gaussian linear, and the Gaussian mixture model. In each dataset, we run adaptive No-U-Turn sampler (NUTS) and check the quality of the sampled distribution after a fixed number of iterations, varying from 2 to 2000 (we use equal number of iterations for warm-up and for sampling, and the warm-up samples were thrown away). At each given MCMC iterations, we run our classifier calibration, and estimate the JS divergence, as reported

-3

0

3

0.0

0.5

1.0

q

1

classifier prediction

under-confident

-3

0

3

0.0

0.5

1.0

q

1

calibrated

-3

0

3

0.0

0.5

1.0

q

1

over-confident

0.03

0.04

0.05

0.06

0.07

-2

0

2

W

b

(

baryon

densi t y

)

predicted log odds

W

b

: ok

0.5

1.0

1.5

2.0

-2

0

2

concentration para.

conc: under-confident

-25

-20

-15

-10

-5

-2

0

2

q

|y)

log q density:

jointly under-confident

tail

of q

mode

of q

in Fig. 5. In all three panels, we are able to detect the inference flaws at early iterations and observe a gradual convergence to zero.

Visual check. Our diagnostic outputs rigorous numerical estimates, but it also facilities visual checks. We make a scatter plot of the binary classifier prediction Pr( t = 1 | ϕ ) , a proxy of p ( θ | y ) /q ( θ | y ) , against any one-dimensional parameter we need to check: If that parameter is under- or over-confident, this scatter plot will display a U- or inverted-U-shaped trend. Compared with SBC rank histograms, our visualization can further check the magnitude of mismatch (how far away Pr( t = 1 | ϕ ) is from 0.5), tail behavior (small q ( θ | y ) ), and several dimensions jointly. Fig 6 is a visual check in the Gaussian example when we multiply the posterior covariance in inference q by 0.8, 1, or 1.2.

Galaxy clustering. We would like to model galaxy spatial clustering observations in a Lambda

cold dark matter framework [34], where observations y correspond to statistical descriptors of galaxy clustering, and a 14-dimensional parameter θ encodes key cosmological information about the Universe. The forward simulation y | θ involves expensive N -body simulations (each single y simulation takes around 5000 cpu hours). We consider three different physical models p : they contain the same cosmology parameter θ but three types of observations whose dimension d y varies from 38 to 1031, reflecting three power-spectrum designs. We apply simulation based inference to sample from each of the models, where we try various normalizing flow architectures and return either the best architecture or the ensemble of five architectures. We now apply our discriminative calibration to these six approximate inferences using a weighted binary classifier trained from S = 2000 , M = 500 simulations for each inference (that is 1 million examples). We have added the log densities log q ( θ | y ) as extra features in the classifier, as they are known in the normalizing-flows. The table above shows the estimated Jensen-Shannon distances and standard deviation. Compared this table with the estimates from blackbox MLP classifier (Appendix Fig. 8), using statistical features greatly improves the label prediction and tightens the divergence estimate. From the table above, all inferences q have a significantly non-zero but small divergence, among which the 5-ensemble that uses a mixture of 5 flows always has a lower divergence so is more trustworthy. Further visualization of the classification log odds (Fig. 7) reveals that q is under-confident in the parameter that encodes the galaxy concentration rate. To interpret the tail behavior, the predicted log odds are negatively correlated with log joint normalizing-flow density q ( θ | y ) , suggesting q ( θ | y ) > p ( θ | y ) in the tail of q ( θ | y ) , another evidence of q underconfidence.

We share Jax implementation of our binary and multiclass classifier calibration in Github 2 .

6 Discussions

This paper develops a classifier-based diagnostic tool for Bayesian computation, applicable to MCMC, variational and simulation based inference, or even their ensembles. We learn test statistics from data using classification. Through a flexible label mapping, our method yields a (family of) computable divergence metric and an always valid testing p -value, and the statistical power is typically higher than the traditional rank based SBC. Various statically-meaningful features are available depending on the task and we include them in the classifier. Visual exploration is also supported.

Related diagnostics. The early idea of simulation based calibration dates back to [13] who compared the prior and the data-averaged posterior, i.e., to check p ( θ ) = ∫ q ( θ | y ) p ( y ) dy using simulations. [42] further developed techniques for the first and second moment estimate of the data-averaged posterior using the law of total variance. Our method includes such moment comparison as special cases: using the noy binary labeling (Example 2), then comparing the empirical moments of the q sample the p sample can be achieved through a naive Bayes classifier or a linear discriminant analysis.

The rank-based SBC [4, 38] can be recovered by ours using binary labels and taking ranks to be features (Example 3). The rank statistic is central to SBC, which follows the classical frequentist approach-using the tail probability under the posited model quantifies the extremeness of the realized value[11] is only a convenient way to locate the observations in the reference distribution, especially in the past when high dimensional data analysis was not developed-it is the use of modern learning tools that sharpens our diagnostic. Recently, SBC has developed various heuristics for designing (one-dimensional) test statistics ϕ ( θ, y ) . Such rank tests are recovered by our method by including the rank of ϕ in our features (Sec. 4.1). For example, [26] proposed to test the rank of the likelihood ϕ = p ( θ | y ) in MCMC, [20] looked at the rank of proposal density q ( θ | y ) , [22] used the q -probability integral transformation in normalizing flows. In light of our Theorem 5, the optimal test statistic is related to the density ratio and hence problem-specific, which is why our method includes all known useful features in the classifier and learns the test statistic from data.

Classifier two-sample test. Using classification to compare two-sample closeness is not a new idea. The classifier two-sample test (C2ST) has inspired the generative adversarial network (GAN) [16] and conditional GAN [25]. [33] has developed GAN-typed inference tools for SBI. In the same vein, [19] used classifiers to diagnose multiple-run MCMC, and [24] used classifiers to learn the likelihood ratio for frequentist calibration, which is further related to using regression to learn the propensity score [36] in observational studies. The theory correspondence between binary classification loss and distribution divergence has been studied in [21, 28, 35]. This present paper not only applies this classifier-for-two-sample-test idea to amortized Bayesian inference to obtain a rigorous test, but also advances the classifier framework by developing the theory of the 'label mapping', while the traditional GAN-type approach falls in the one-class-per-group category and deploy binary classifiers as in Example 1. Our extension is particularly useful when samples are overlapped (multiple θ per y ), autocorrelated, and imbalanced.

KL divergence estimate from two samples. As a byproduct, our proposed multiclass-classifier provides a consistent KL divergence estimate (Thm. 3) from two samples. Compared with existing two-sample KL estimate tools from the f -divergence representation [29] or the Donsker-Varadhan representation [2], our multiclass-classifier estimate appears versatile for it applies to samples with between-sample dependence or within-sample auto-correlation. It is plausible to apply our multiclassclassifier approach to other two-sample divergence estimation tasks such as the f -GAN [30], which we leave for future investigation.

Limitations and future directions. Like traditional SBC, our method assesses the difference between p ( θ | y ) and q ( θ | y ) , averaged over y . This is a 'global' measure relevant to developing algorithms and statistical software. But sometimes the concern is how close p and q are for some particular observation y = y obs . 'Local' diagnostics have been developed for MCMC [14, 12, 40, 17], variational inference [41, 8] and simulation-based inference [42, 22] that try to assess q ( θ | y obs ) only. It would be valuable to extend our approach to address these cases. Another future direction would be to extend our approach to posterior predictive checks [11, 9, 39] that diagnose how well the statistical model p fits the observed data.

References

Agrawal, A., Sheldon, D. R., and Domke, J. (2020). Advances in black-box VI: Normalizing flows, importance weighting, and optimization. In Advances in Neural Information Processing Systems .

Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, R. D. (2018). MINE: mutual information neural estimation. In International Conference on Machine Learning .

Berk, R., Brown, L., Buja, A., Zhang, K., and Zhao, L. (2013). Valid post-selection inference. Annals of Statistics , 41(2):802-837.

Cook, S. R., Gelman, A., and Rubin, D. B. (2006). Validation of software for Bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics , 15(3):675-692.

Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory . John Wiley & Sons, 2rd edition.

Cranmer, K., Brehmer, J., and Louppe, G. (2020). The frontier of simulation-based inference. Proceedings of the National Academy of Sciences , 117(48):30055-30062.

Dax, M., Green, S. R., Gair, J., Macke, J. H., Buonanno, A., and Schölkopf, B. (2021). Real-time gravitational wave science with neural posterior estimation. Physical Review Letters , 127(24):241103.

Domke, J. (2021). An easy to interpret diagnostic for approximate inference: Symmetric divergence over simulations. arXiv:2103.01030 .

Gabry, J., Simpson, D., Vehtari, A., Betancourt, M., and Gelman, A. (2019). Visualization in bayesian workflow. Journal of Royal Statistical Society: Series A , 182(2):389-402.

Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing , 24(6):997-1016.

Gelman, A., Meng, X.-L., and Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistical Sinica , 6:733-760.

Gelman, A. and Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science , 7(4):457-472.

Geweke, J. (2004). Getting it right: Joint distribution tests of posterior simulators. Journal of American Statistical Association , 99(467):799-804.

Geyer, C. J. (1992). Practical Markov chain Monte Carlo. Statistical Science , 7(4):473-483.

Gonçalves, P. J., Lueckmann, J.-M., Deistler, M., Nonnenmacher, M., Öcal, K., Bassetto, G., Chintaluri, C., Podlaski, W. F., Haddad, S. A., and Vogels, T. P. (2020). Training deep neural density estimators to identify mechanistic models of neural dynamics. Elife , 9:e56261.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial networks. In Advances in Neural Information Processing Systems .

Gorham, J. and Mackey, L. (2017). Measuring sample quality with kernels. In International Conference on Machine Learning .

Hermans, J., Delaunoy, A., Rozet, F., Wehenkel, A., Begy, V., and Louppe, G. (2022). A trust crisis in simulation-based inference? Your posterior approximations can be unfaithful. Transactions on Machine Learning Research .

Lambert, B. and Vehtari, A. (2022). R ∗ : A robust MCMC convergence diagnostic with uncertainty using decision tree classifiers. Bayesian Analysis , 17(2):353-379.

Lemos, P., Coogan, A., Hezaveh, Y., and Perreault-Levasseur, L. (2023). Sampling-based accuracy testing of posterior estimators for general inference. arXiv:2302.03026 .

Liese, F. and Vajda, I. (2006). On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory , 52(10):4394-4412.

Linhart, J., Gramfort, A., and Rodrigues, P. L. (2022). Validation diagnostics for SBI algorithms based on normalizing flows. arXiv:2211.09602 .

Lueckmann, J.-M., Boelts, J., Greenberg, D., Goncalves, P., and Macke, J. (2021). Benchmarking simulation-based inference. In International Conference on Artificial Intelligence and Statistics .

Masserano, L., Dorigo, T., Izbicki, R., Kuusela, M., and Lee, A. B. (2022). Simulation-based inference with WALDO: Perfectly calibrated confidence regions using any prediction or posterior estimation algorithm.

Mirza, M. and Osindero, S. (2014). Conditional generative adversarial nets. arXiv:1411.1784 .

Modrák, M., Moon, A. H., Kim, S., Bürkner, P., Huurre, N., Faltejsková, K., Gelman, A., and Vehtari, A. (2022). Simulation-based calibration checking for Bayesian computation: The choice of test quantities shapes sensitivity. arXiv:2211.02383 .

Neyman, J. and Pearson, E. (1936). Sufficient statistics and uniformly most powerful tests of statistical hypotheses. In Statistical Research Memoirs , pages 1-37. Cambridge University Press.

Nguyen, X., Wainwright, M. J., and Jordan, M. I. (2009). On surrogate loss functions and f -divergences. Annals of Statistics , 37(2):876-904.

Nguyen, X., Wainwright, M. J., and Jordan, M. I. (2010). Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory , 56(11):5847-5861.

Nowozin, S., Cseke, B., and Tomioka, R. (2016). f-GAN: Training generative neural samplers using variational divergence minimization. Advances in Neural Information Processing Systems .

Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., and Lakshminarayanan, B. (2021). Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research , 22(1):2617-2680.

Prangle, D., Blum, M., Popovic, G., and Sisson, S. (2014). Diagnostic tools of approximate Bayesian computation using the coverage property. Australian & New Zealand Journal of Statistics , 56(4):309-329.

Ramesh, P., Lueckmann, J.-M., Boelts, J., Tejero-Cantero, Á., Greenberg, D. S., Gonçalves, P. J., and Macke, J. H. (2022). GATSBI: Generative adversarial training for simulation-based inference. International Conference on Learning Representations .

Régaldo-Saint Blancard, B., Hahn, C., Ho, S., Hou, J., Lemos, P., Massara, E., Modi, C., Moradinezhad Dizgah, A., Parker, L., , Yao, Y., and Eickenberg, M. (2023). SimBIG: Galaxy clustering analysis with the wavelet scattering transform. arXiv:2310.15250 .

Reid, M. and Williamson, R. (2011). Information, divergence and risk for binary experiments. Journal of Machine Learning Research .

Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika , 70(1):41-55.

Rubin, D. B. (1981). The Bayesian bootstrap. Annals of Statistics , 9(1):130-134.

Talts, S., Betancourt, M., Simpson, D., Vehtari, A., and Gelman, A. (2018). Validating Bayesian inference algorithms with simulation-based calibration. arXiv:1804.06788 .

Vandegar, M., Kagan, M., Wehenkel, A., and Louppe, G. (2021). Neural empirical Bayes: Source distribution estimation and its applications to simulation-based inference. In International Conference on Artificial Intelligence and Statistics .

Vehtari, A., Gelman, A., Simpson, D., Carpenter, B., and Bürkner, P.-C. (2021). Rank-normalization, folding, and localization: An improved ˆ R for assessing convergence of MCMC. Bayesian Analysis , 16(2):667718.

Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). Yes, but did it work?: Evaluating variational inference. In International Conference on Machine Learning .

Yu, X., Nott, D. J., Tran, M.-N., and Klein, N. (2021). Assessment and adjustment of approximate inference algorithms using the law of total variance. Journal of Computational and Graphical Statistics , 30(4):977-990.

Appendices to 'Discriminative Calibration'

The Appendices are organized as follows. In Section A, we provide several extra theory results for the main paper. In Section B, we discuss our experiment and implementations. In Section C, we provide proofs for the theory claims.

A Additional theory results

Theorem 7 (Closed form expression of the divergence) . Consider the label generating process in Section 4. Let π ( t, ϕ ) be the joint density of the label and features after mapping Φ , i.e., the distribution of the ( t, ϕ ) generated by the simulation process:

sample ( θ, y ) ∼ p ( θ, y ) ;

sample ( ˜ θ 1 , . . . , ˜ θ M ) from q ( θ | y ) ;

generate (( t 1 , ϕ 1 ) , . . . , ( t L , ϕ L ))) = Φ(( y, θ, ˜ θ 1 , . . . , ˜ θ M )) ;

sample an index l from Uniform (1 , . . . , L ) ;

return ( t l , ϕ l ) .

We define

to be the y -averaged law of ϕ | t = k . Note that y has been averaged out.

Then classification divergence defended through (7) in Theorem 1 has a closed form expression, a Jensen-Shannon divergence in this projected ϕ space.

This Theorem 7 gives a connection between our divergence and the familiar Jensen-Shannon divergence, which is well known to be linked to IID sample classification error (note that the classification examples we generate are not IID for they share y ).

As a special case, when K = 2 , w 0 = w 1 = 1 / 2 , we recover the symmetric two-group Jensen-Shannon divergence, or JSD. In general, between two densities π 1 ( x ) and π 2 ( x ) , the JSD is

As before, the standard notation [5] of conditional divergence is defined as taking expectations over conditional variables:

Theorem 8 (weighting) . The binary label mapping generates M paris label1 examples and one pair of label-0 examples per simulation. We can reweight the binary classifier by letting the ELPD be E [ C M +1 ✶ ( t = 1) log p ( t = 1 | c ( ϕ )) + CM M +1 ✶ ( t = 0) log p ( t = 0 | c ( ϕ ))] , where C = ( M +1) 2 2 M is a normalizing constant. That is if we modify the training utility function or the validation data LPD to be:

then the resulting binary classification divergence in Thm. 1, i.e.,

is the conditional Jensen Shannon divergence (15) .

A.1 SBC test and generative classifier

̸

In Example 3, we stated the intuitive connection between the SBC rank test and using rank as the feature in a ' generative ' classifier (in contrast to our 'discriminate' approach). Now we make such comparison more precise: Assuming the parameter θ is one-dimensional. This is what SBC does: From p we obtain θ, y , and for each θ we compute the rank of it in the paired q samples who shared the same y : r = ∑ M m =1 ✶ ( θ ≤ ˜ θ m ) . Repeating this simulation many times, we obtain a sample of r , and SBC will test if such r is discrete-uniformly distributed. This test could be done by simulating a reference distribution that matches the null. We do so by generating the ranks in the q samples: ˜ r n = ∑ m = n ✶ ( ˜ θ n ≤ ˜ θ m ) + ✶ ( ˜ θ n ≤ θ ) . To be clear, there are other ways to generate reference samples, but all it matters is that the reference distribution matches the distribution of r under the null. The uniformity test in SBC now becomes to test if r and ˜ r have the same distribution. For example, we can do a two-sample Kolmogorov-Smirnov test or a permutation test.

In comparison, this is what naive Bayes would do: the naive Bayes classifier first estimates the two conditional distributions: Pr( r | t = 0) from the empirical distribution of the prior rank r , and Pr( r | t = 1) from the empirical distribution of ˜ r . Then to test if Pr( r | t = 0) = Pr( r | t = 1) becomes the same to testing if the empirical distribution of r is the same as the empirical distribution of r -the same task as in SBC. The point is that given any two-sample based test,

is operationally the same as only use rank + naive Bayes classifier + density estimated by histogram + sample-based test.

B Implementations

B.1 A cheat sheet of useful pre-learned features

Table 1 summarizes how to use the statistical feature (whenever available) in both the binary and multiclass classifiers.

B.2 Code and data

We share the python and Jax implementation of our binary and multiclass calibration code in https://github.com/yao-yl/DiscCalibration .

In the MLP training we include a standard L 2 weight decay (i.e., training loss function = cross entropy loss + tuning weight ∗ L 2 penalization). We tune the weight of the decay term by a 5 fold cross-validation in the training set on a fixed grid { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } .

In the cosmology experiment in Section 5, we use one-hidden-layer MLP with 64 nodes to parameterize the classifier with the form (11), with additional pre-learned features such as log q ( θ | y ) added as linear features. The approximate log density log q ( θ | y ) is known (i.e., we can evaluate the for any θ and y , or at least least up to a constant) in either the normalizing flow or the ensemble of the normalizing flows. One classification run with roughly one million examples took roughly two hour

cpu time on a local laptop. It would be more efficient to run the classification on GPU, thought the classification cost is overall negligible compared with the simulation step y | θ which is pre-computed and stored.

In the closed-form Gaussian experiment in Section 5, we consider the easiest setting: θ ∈ R d ∼ MVN( 0 , Id d ) and a normal data model y | θ ∼ MVN( θ, Id d ) . Indeed, we have kept both the true distribution and the sampling corruption mean-field to make the calibration task easier for the traditional SBC rank-test. The power of the traditional SBC rank-test would be even worse if we add some interaction terms in the corrupted samples q ( θ | y ) , while our method leans the joint space sampling quality by default. The exact posterior p ( θ | y ) is known but we pretend we cannot evaluate it. We set d = 16 to be comparable with the real data example. The right panel validates Thm. 3 fixed the simulation size S = 5000 and vary M . We have tried other fixed M = 1000 but it seems the resulting graph is very similiar. The confidence band in Figure 3 and Figure 4 is computed using 1.96 times standard error from repeated experiments.

Table 8 shows the naive classifier: using the bandbox MLP without the linear features. The estimate is rather loose compared with the table we obtained in Section 5 which used statistically-meaningful features.

C Proofs of the theorems

We prove the theorems in the following order: Thm. 7, Thm. 1, Thm. 8, Thm. 6, Thm. 3, Thm. 5, Thm. 4, and finally Thm. 2.

Contents

C.1 The classification divergence

To prove our Theorem 1, we first state an equivalent but a more general theorem. Note that we are allowing non-IID classification examples for the shared z variables in this theory.

Theorem 9 (The general relation between divergence and classification error) . Define a simulation process π ( z, k, x ) through the following process, among the three random variables, z ∈ R n , x ∈ R d can have any dimension, and k ∈ { 1 , 2 , . . . , K } is an integer.

Sample z ∼ π ( z )

Sample k ∼ Categorical( w ) , where w is a simplex.

Sample x ∼ π k ( x | z )

Return ( z, k, x )

Define U( π ) to be the expected log probability or cross-entropy of an optimal classifier trained to predict k from ( x, z ) , i.e.

where U ( k, c ) = log c ( k ) . Then 3

where π ( x | z ) = ∑ K k =1 w k π k ( x | z ) is the mixture over all groups.

We define D ( π ) = U( π ) -∑ K k =1 w k log w k . Then this D has the following properties:

(lower bound). D ( π ) ≥ 0 . It achieves zero if and only if all p k are the same, i.e., Pr( X k ∈ A ) = Pr( X j ∈ A ) for any measurable set A .

(upper bound). D ( π ) ≤ ∑ K n =1 w k log w k . This maximum is achieved is p k are disjoint, i.e., Pr( X k ∈ A ) Pr( X j ∈ A ) = 0 for any measurable set A .

(alternative upper bound when the difference is small). D ( π ) ≤ max j,k KL( π k ( x | z ) , π j ( x | z )) .

(reparametrization invariance). For any fixed bijective transformation x ↦→ g ( x ) , let p g k be the law of g ( x k ) , then D ( p g k , . . . , p g k ) = D ( p k , . . . , p k ) .

(transformation decreases divergence). For any fixed transformation x ↦→ g ( x ) , let p g k be the law of g ( x k ) , then D ( p g k , . . . , p g k ) ≤ D ( p k , . . . , p k ) .

Proof. The (unknown) optimal conditional classifier is the Bayes classifier:

Let r ( x | z ) ∑ K j =1 w k π j ( x | z ) be mixture density marginalized over the group index k .

The expected log predictive density of the optimal classification is then

When ELPD -∑ K k =1 w k log( w k ) = 0 , ∑ K k =1 w k π k ( x | z ) = 0 , hence these K conditional densities π k ( x | z ) are the same almost everywhere.

The reparametrization invariance is directly inherited from KL diverge.

The upper bound is the consequence of ELPD ≤ 0 for category classification and the lower bound is that KL divergence ≥ 0 .

Let's emphasize again that we are following the notation of conditional divergence in [5], such that for any two joint density p ( theta, y ) and q ( θ, y ) , the conditional KL divergence is KL( p ( θ | y ) ∥ q ( θ | y )) := E p ( y ) E p ( y | θ ) log p ( y | θ ) q ( y | θ ) , an expectation is taken over y .

The procedure in Theorem 9 can reflect many different generating processes. Let's recap the examples we show in Section 2.

In Example 1 of the main paper, the binary case, K = 2 , and we generate M samples from q . π 1 = p ( θ | y ) , π 2 = q ( θ | y ) , w 1 = 1 /M , w 2 = ( M -1) /M . Our simulation process in example 1 is equivalent to

Sample y ∼ marginal p ( y ) ,

Sample k ∼ Categorical( w ) , where w 1 = 1 /M , w 2 = ( M -1) /M .

Sample x ∼ π k ( x | z ) , where π 1 = p ( θ | y ) , π 2 = q ( θ | y ) .

Return ( z, k, x )

As a direct consequence of Theorem 9, the resulting divergence in the binary case is

where r = w 1 p + w 2 q .

In Example 2 : Binary label with no y , we generate M samples from q . K = 2 , π 1 = p ( θ ) , π 2 = q ( θ ) , w 1 = 1 / ( M +1) , w 2 = M/ ( M +1) . Our simulation process in example 2 is equivalent to

Sample k ∼ Categorical( w ) , where w 1 = 1 /M , w 2 = ( M -1) /M .

Sample x ∼ π k ( x ) , where π 1 = p ( θ ) , π 2 = q ( θ ) .

Return ( k, x )

From Theorem 9, the resulting divergence reads

In the multivariate Example 4 , K = M +1 , and the individual density is

̸

.

From Theorem 9, the resulting divergence reads

We give more results on this D 4 in Section C.4.

The discriminative calibration is designed to detect and gauge the difference between p ( θ | y ) and q ( θ | y ) . From now on we use a long vector θ 1: M +1 (the index starts by 1) to represent the simulation draw. Each simulation draw first generates one θ 1 from the prior, one data point y from the sampling distribution p ( y | θ 1 ) , and finally M draws from the approximate inference θ 1 ,...,M +1 ∼ q ( θ | y ) .

The 'label generating process' Φ takes a simulated draw as its input and returns a vector of features ϕ and labels t .

The classifier will then see this feature and labels (the index k is disregarded) across all simulations.

We only ask that the map Φ be within in set : satisfying that:

Proof of Theorem 1.

Proof. The classifier sees features ϕ and labels t . Given a t , ( ϕ, y ) are IID draws from p ( y ) π ( ϕ | y, t ) . In case y is not observable by the classifier, ϕ | t are IID draws from ∫ y p ( y ) p ( ϕ | y, t ) .

To use Theorem 9, let x k = { ( ϕ j , y ) | t j = k } and w k = Pr( t = k ) . The divergence from Theorem 9: reads

in which π k ( ϕ ) = p ( ϕ | t = k ) = ∫ y p ( y ) π ( ϕ | y, t = k ) dy is the y -averaged ϕ margin. This is essentially Theorem 7.

If p = q , from ❋ , the law of ϕ, y is independent of t ( y is always independent of t marginally), so that π k ( ϕ ) = ∫ y p ( y ) π ( ϕ | y, t = k ) dy is also independent of t , as it only depends on y and ϕ , hence the divergence remains 0.

C.2 Reweighting (Theorem 8) leads to symmetric divergence

We now prove Theorem 8. The binary label mapping generates M pairs label1 examples and one pair of label-0 examples per simulation. We can reweight the binary classifier by letting the ELPD be E [ C M +1 ✶ ( t = 1)log p ( t = 1 | c ( ϕ )) + CM M +1 ✶ ( t = 0)log p ( t = 0 | c ( ϕ ))] , where C = ( M +1) 2 2 M is a normalizing constant. We want to prove that after weighting, the classification divergence is symmetric Jensen shannon distance, as if we have balanced classification data.

Proof. After the label-reweighing, the optimal classifier is the balanced-class classifier

The re-weighted ELPD in the population is

̸

With an appropriate normalizing constant C = ( M +1) 2 2 M ,

That is, the reweighted ELPD + log 2 is the conditional Jensen Shannon divergence (15) between p ( θ | y ) and q ( θ | y ) .

C.3 MCMCwithout thinning (Theorem 6)

To diagnose MCMC, we sample ( θ, y ) ∼ p ( θ, y ) , and sample (potentially autocorrelated) draws ( ˜ θ 1 , . . . , ˜ θ M ) from a MCMC sampler whose stationary distribution we believe is q ( θ | y ) (i.e., marginally ˜ θ i is from q ( θ | y ) ), and generate examples (( t 1 , ϕ 1 ) , . . . , ( t M +1 , ϕ M +1 )) , from the multiclass permutation (see definition in Example 4), such that

Then we run an exchangeable classifier parameterized by

where g is any Θ ×Y → R mapping to be learned.

Note that here Pr( t = k | ( θ 0 , θ 1 , . . . , θ M , y )) is the classifier model we restrict it to be, not the true population π . In general π ( t = k | ( θ 0 , θ 1 , . . . , θ M , y )) = = Pr( t = k | ( θ 0 , θ 1 , . . . , θ M , y )) Roughly speaking, the optimal classifier should be the Bayes classifier projected to the restricted space (16), and we need to proof that this projected restricted solution turns out to be the same as the IID case 10, which is not trivial in general.

̸

Proof. Intuitively, this separable network design (16) avoids the interaction between ˜ θ m with ˜ θ m ′ and disallows the network to predict t based on the autocorrelation or clustering of ˜ θ .

Because of the permutation design and we define q to be the marginal distribution of ˜ θ , first, we know that

̸

in the MCMC population π (there is no need to address the joint for now).

From this we obtain the conditionals in the population

and further the ratios for any m , k withtin the index set { 0 , 2 , . . . , M } and m = k :

̸

Here q 12 ( θ k , θ m ) is the joint distribution of two out of M draws from the MCMC sampler q ( θ | y ) , which would often not be the same as q ( θ k | y ) q ( θ m | y ) because of the autocorrelation of Markov chains. We do not need to specify the form of this joint. The key blessing is that when θ k is from p ( θ | y ) and θ m is from q ( θ | y ) , then they are independent (the numerator).

Next, from the line above we obtain the ratio estimate in the true population π :

Nowweproject the restricted classifier (16). Intuitively, the separable classifier 'almost' only depends on θ t , except for the normalizing constant. We can remove the dependence on the normalizing constant by specifying the ratio

Marginalizing out all other components, we obtain

Matching the restriction (18) with the true population (17), the restricted projection is attainable if and only if

so that the optimal classifier needs to be

It happens that this MCMC restricted optimal classifier matches the IID optimal classifier (10). It follows from the proof in Theorem 9 that the classification divergence using the restricted classifier is still D 4 ( p, q ) , as if { ˜ θ m } M m =1 are IID samples from q ( θ | y ) .

C.4 Large sample limit and rate as M →∞ (Theorem 3)

In the multivariate Example 4 , from each draw θ, y, ˜ θ 1: M , we generate M + 1 examples from K := M +1 classes; one label from each. We will use the index starting from 1 in this subsection. For the k -th example, t k = k , and the permutation for classifier reads ϕ k is a vector including y and K copies of θ , where the k -th copy is the prior draw, and the remaining θ are from q ( | y ) . Slightly abused the notation, in this subsection, we call this long feature vector as θ 1: K ; it is a vector in the Θ M +1 space. We now prove Theorem 3.

Proof. First, we write the true conditional label probability in this process:

̸

̸

Plug it as the classifier, we obtain the optimal ELPD or negative cross entropy: ELPD = E log π ( t | θ 1: K , y ) ,

The first term above is simply

According to our definition (4), the divergence is ELPD offset by an entropy term,

We now derive the limit of D 4 -KL( p ( θ | y ) || q ( θ | y )) when M →∞ (or equivalently K = M +1 → ∞ )

where

̸

Given any label value 1 ≤≤ K , θ t ∼ p ( ·| y ) , and all the remaining θ j ∼ q ( ·| y ) for j = t . Let

̸

The first term

̸

The second term, call it ∆ 2 is the mean of IID means as θ j ∼ q ( ·| y ) for j = t , the law of large number yields

̸

which finished the first part of the proof.

Now let's derive the rate. Under regularity conditions, for example, the variance of density ratio is bounded, i.e., there exists a constant C < ∞ , such that for all y ,

then CLT holds, such that the second term above has a normal limit,

which is the definition of the conditional chi-squared divergence.

Consider a Taylor series expansion, log(1 + x ) = x -1 2 x 2 + o ( x 2 ) . Using the Delta method to express the log function and the Slutsky theorem to ignore the zero term, we get

Plug this in the definition of ∆ we obtain the desired convergence rate,

This proves the claims in Theorem 3.

This proves that

Hence,

0

0.5

1

our permutation

0

0.5

1

naive permutation

p-values under null

C.5 Valid hypothesis testing (Theorem 4)

It is not trivial to have a valid permutation test since we do not have IID examples. One naive permutation is to permutate all the labels t (across batches i ). The resulting permutation testing is not uniform under the null (Figure 9).

In contrast, when we permute the label, we only ask for within-batch permeation (permute the index { t 1 , . . . , t L } in each batch). See Table 2 for an illustration. Let's prove this permutation is valid.

Proof. Under the null, p ( θ | y ) = q ( θ | y ) almost everywhere. According to the label mapping Φ ∈ ❋ , label t is independent of ϕ given y . We first show that label t is independent of ϕ .

In general conditional independence does not lead to unconditional independence. But because here we design the generating process by π ( y, t, ϕ ) = π Y ( y ) π t ( t ) π ( ϕ | y, θ ) . Under the null we have π ( y, t, ϕ ) = π t ( t )( π Y ( y ) π ( ϕ | y )) . Hence t needs to be independent of ϕ .

For any classifier c , because c ( ϕ ) is a function of ϕ , c ( ϕ ) and t are also independent. Let π Φ ( ϕ ) be the marginal distribution of ϕ in π .

Now we are computing the cross entropy (with respect to the population π ). LPD = ∑ N j =1 U ( c ( ϕ n ) , t n ) where U is the log score U ( P, x ) = log P ( x ) . It is a random variable because we have a finite validation-set. Now we are conducting a permutation of label t , the permuted label ˜ t is an independent draw from the same marginal distribution of t , π t ( t ) . Because of the independence, E ϕ,t U ( c ( ϕ ) , t ) = E t E ϕ U ( c ( ϕ ) , t ) , hence the permuted LPD b d = LPD , where LPD b is the computed LPD in the b -th permutation. Therefore Pr(LPD ≤ x ) = Pr(LPD b ≤ x ) . In expectation (with an infinitely amount of repeated random permutation), this p -value will be uniform on [0,1].

In practice, the computed p-value from finite permutation can only take values on a finite set { 0 , 1 /B,.. . , ( B -1) /B, 1 } . More precisely, under the null hypothesis, this permutation test p-value is discreetly-uniform on this set. This is because for any 0 ≤ m ≤ B

Hence, the permutation p -value based on B random permutations is uniformly distributed on the set { 0 , 1 /B,.. . , ( B -1) /B, 1 } .

C.6 Why does classifier help test? (Theorem 5)

Let's recap the binary label-generating process:

Sample y ∼ marginal p ( y ) ,

Sample k ∼ Bernoulli( w ) , where w 1 = 1 /M , w 2 = ( M -1) /M .

Sample θ ∼ π k ( θ | z ) , where π 0 = p ( θ | y ) , π 1 = q ( θ | y ) .

Return ( z, k, x )

Theorem 5 state that the optimal classifier has a sufficiency propriety in that (a) let ˆ c be the probability of label 1 in the optimal classifier as per (7), and let π p c and π q c be the one-dimensional distribution of this ˆ c ( ϕ ) when ( θ, y ) is sampled from p ( θ, y ) or from p ( y ) q ( θ | y ) respectively, then (i) Conditional on the summary statistic ˆ c , the label t is independent of all features ϕ = ( θ, y ) . (ii) There is no loss of information in divergence as the joint divergence is the same as the projected divergence, D 1 ( p, q ) = D 1 ( π p c , π q c ) .

Proof. There are three random variables, θ , y , and t . The optimal classifier ˆ c is the probability in this true joint:

To show sufficiency or conditional independence, all we need to show is that, conditional on any given value of ˆ c , Pr( t = 1 | ( θ, y ) , c ) does not depend on ( θ, y ) (in other words, Pr( t = 1 | ( θ, y ) , c ) is a function of c only). This becomes obvious as

Now we prove that there is 'no loss' in divergence.

We express the first term in ˆ c

This steps uses the chain rule of KL divergence: KL [ p ( x, y ) | q ( x, y )] = KL [ p ( x ) | q ( x )] + KL [ p ( y | x ) | q ( y | x )]

Using conditional independence:

Hence KL( π ( θ, y | ˆ c, t = 0) || π ( θ, y | ˆ c )) = 0 . Therefore,

Similarly,

This proves D 1 ( p, q ) = D 1 ( π p c , π q c ) .

C.7 The maximum discriminative generator (Theorem 2)

We save the proof of Theorem 2 in the end for its length.

The generator ϕ contains a few degrees of freedom: the number of classes K , the number of examples L , and how to design and the label-feature pairs. In the binary labeling: t 1 = 1 , ϕ 1 = ( θ 1 , y ) and the remaining t k = 0 , ϕ k = ( θ k , y ) for 2 ≤ k ≤ M +1 . The multi-class Φ ∗ assigns labels 1: K as

Before the main proof that the multiclass permutation creates the largest divergence, let's first convince that the multi-class classification produced higher divergence than the binary one.

In the binary classification with M = 1 (one draw from p and one draw from q )

In the multi-class classification with M = 1 ,

Use the joint KL > marginal KL, we have

Add these two lines we obtain

Likewise, where the m -th mixture probability is

C is chosen such that ∑ L m =1 p m = 1 .

Now that x 1 , . . . , x L are L IID random variables from this π , f 1 ( x 1 , . . . , x L ) is also a mixture of delta functions, For any sequence of input indices ( u 1 , . . . , u L ) ∈ { 1 : L } L ,

To prove that this multi-class permutation produced the uniformly largest divergence (across M , K , p , q ) optimal, we organize the proof into lemmas 5 to 9. For notation brevity, we denote ˆ M := M +1 in these lemmas to avoid using index M +1 .

Lemma 10. For an arbitrary integer L , any given output space Y , and any input space X that has at least L elements, if there are two functions mapping X L to Y

satisfying the following propriety:

for any probability distribution π on X , when x 1 , . . . , x n are L iid random variables with law π , f 1 ( x 1 , . . . , , x L ) has the same distribution as f 2 ( x 1 , . . . , , x L ) ,

then there must exist a permutation of 1 : L , denoted by σ (1 : L ) , such that

Proof. For any L distinct values a 1 , . . . , a L , a i ∈ X , let π be a mixture distribution of L delta functions:

in which the power index can be written as

as an L -decimal-integer.

Next, we study the law of f 2 ( x 1 , . . . , , x L ) :

Because f 2 ( x 1 , . . . , , x L ) and f 1 ( x 1 , . . . , , x L ) have the same distribution, f 2 ( a 1 , . . . , , a L )) needs to match the value at which f 1 ( x 1 , . . . , , x L ) has probability C L ( 1 2 ) (1 , 1 ,..., 1) L to attain. Comparing with (20), this probability is only attained when ∑ L j =1 1( u j = m ) = 1 , ∀ m . That is, there exists a σ , a permutation of 1 : L , such that u 1 , . . . , u L = σ (1 , 2 , . . . , L ) .

Matching the value of f 2 ( x 1 , . . . , , x L ) we obtain

Because the choice of vector ( a 1 , . . . , a L ) is arbitrary, we have

Because augmentation increases divergence, for the purpose of finding the largest divergence, to produce the largest divergence, we only need to consider an augmented generator that includes all y

.

It is enough to consider the generator of which ϕ k = ϕ k ( θ 1 , . . . , θ L ) are K functions of ( θ 1 , . . . , θ L ) Lemma11. For any augmented generator Φ arg satisfies ❋ , the null, there must exists K permutations σ 1 (1 : L ) , . . . , σ K (1 : ( L )) , with the convention σ 1 (1 : L ) = 1 : L , such that

Proof. Use Lemma (10) for ( K -1)

Lemma 12. For any augmented generator Φ arg satisfies ❋ , all feature-label generator can be replaced by a permutation, ϕ k ( θ 1 , . . . θ L ) = ( θ σ k (1) , . . . θ σ k ( L ) ) , while the divergence does not decrease.

Proof. From the previous lemma,

The augmented feature is now ( ϕ 1 ( θ σ k (1: L ) ) , y ) , a transformation of ( θ σ k (1: L ) ) , y ) . Using the raw feature ( θ σ k (1: L ) ) , y ) keeps the divergence non-decreasing.

Now that we only want to consider permutation-based generators: there exists a K , and K permutations σ k (1 : L ) .

̸

Given a p = q , any permutations θ σ i (1: L ) contains one copy from p and ( L -1) copies from q . It suffices to only consider those permutation θ σ i (1: L ) whose distributions are unique.

̸

Lemma 13. Given any p = q and L fixed, assuming all θ σ i (1: L ) has different distributions, then D ( q, p, Φ arg ) is an increasing function on K .

Proof. Using that joint KL is always not smaller than the KL divergence of sub-coordinates.

Lemma 14. Among all permutations σ ( θ 1 , . . . θ L ) , the maximum number of distinct distributions are K = L .

̸

Proof. The total number of permutations is L ! . Because θ 2: L are IID given y , the permutation of index 2 : L does not change the distribution. When p = q , the total number of distributionally distinct permutations is L ! / ( L -1)! = L = M +1 .

It is clear that the proposed multi-class permutation Φ ∗ attains this maximum number of distinct distributions, which proves its optimality among all generators from all previous lemmas, thereby the proof of our Theorem 2.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a classifier based approach to provide a measure of miscalibration in Bayesian computation, including for methods such as Approximate Bayesian Computation (ABC) and Simulation-Based Inference (SBI) methods like neural posterior estimation. The method enables the test statistic to be learned from data and  provides an interpretable divergence measure. The method is a form of two sample testing applied in the amortized and simulation based inference settings. Beyond standard approaches to using classifiers for two sample testing, the authors develop several classifier based approaches which use a form a \"label mapping\", and provide theoretical work on the validity and efficacy of these approaches. Experiments are provided to test the method for posterior inference in cases where the posterior is known, and provide empirical verification of the theoretical results. The paper also compares to the commonly used simulation based calibration method, and show clear improvements with the methods proposed in the paper. Experiments on cosmological data are also explored.\n\nSTRENGTHS:\nThis is a good paper, it provides:\n-  a clear description of the challenge they hope to address, specifically providing a better and more statistically interpretable measure of miscalibration over simulation-based calibration\n- a clear description of the proposed methods, including how to use label mapping to develop calibration based diagnostics, and how this differs from other approaches\n- theoretical grounding for the calibration based measures, including the expected behaviors in large sample limits\n- useful discussions on implementation details\n- useful discussion on the legitimacy and power of the tests.\n\nWEAKNESSES:\nWhile the idea of using classifiers for two sample testing is not new, the authors do develop new methods based on label mapping. While the new classification approaches are discussed and will likely be quite useful for simulation based inference, some attempts at using classifiers for testing posterior inference quality have been performed before (albeit not directly with some of the new classification methods discussed in the work). For instance, Vandegar et. al \"Neural Empirical Bayes: Source Distribution Estimation and its Applications to Simulation-Based Inference\" AISTATS 2021, use the AUC of a classifier as a diagnostic in a simulation based setting.  \n\nFurther development of the experiments would be quite useful. A very simple experiment is shown, allowing an exploration of the methods, but some experiments more clearly showing the quality of the method in a controlled but more complex simulation based inference setting could provide useful insights to the readers. For instance, a common example is the Simple Likelihood, Complex Posterior (SLCP) problem from Papamakarios et. al, \"Sequential neural likelihood: Fast likelihood-free inference with autoregressive flows\", AISTATS 2019. As it stands now, it is difficult to asses the quality of the method in the more complex cosmological data set experiment.\n\nQUESTIONS:\nCould you provide more details on the posterior model in the multi-variate Gaussian example?\n\nCould you provide experiments on more complex but controlled examples, for instance the SLCP problem from Papamakarios et. al? This could provide more evidence on the utility of the method in SBI settings.\n\nHow do these methods scale with feature and parameter dimension? Does this affect the ability to attain a useful diagnostic?\n\nYou mention that the presence of nuisance parameters do not affect the quality of the diagnostics. However, the Neyman-Pearson lemma does not guarantee that the likelihood ratio is the uniform most powerful test for a given size in the presence of nuisance parameters. Perhaps I am not fully understanding the text, could you explain if this has any ramifications for the proposed tests?\n\nCould you add references to other simulation based inference work using classifiers for diagnostics? In addition, recent work on using neural networks for two sample testing (e.g. Grosso et. al, https://arxiv.org/abs/2305.14137) may be relevant to discuss in the related work.\n\nLIMITATIONS:\nThe authors provide clear discussions of limitations. It would be interesting to also know if there are limitations dependent on the dimension of the features or parameters, and how this may impact the number of samples needed to attain a useful test.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors focus on the challenge of comparing two conditional distributions p, q from their samples. In particular, this is useful as a check for bayesian computations.\nTo achieve this, the authors propose the use of a probabilistic classification approach where they create a new dataset combining samples from p and q and labels related to the particular distribution. Here, the authors propose different approaches and make connections to previous work.\nGiven the combined dataset, a classifier tries to predict the label from the features. Failing to do so, indicates that the classifier could not discriminate between the two distributions.\nThe performance of the probabilistic classifier is then used to estimate the divergence between the distributions and for hypothesis testing. \nIn the empirical section, the authors compare against SBC and show improvements in power given a reduced number of samples.\n\nSTRENGTHS:\nThe paper analyses different approaches for the generation of samples to be used by discriminators. They then present theoretical developments and demonstrate their results empirically. The sampling approaches and theoretical treatment are novel, of high quality and well written.\nThe main advantage here is that even for a sub-optimal classifier, one gets improvements in data efficiency.\n\nWEAKNESSES:\nThe main weakness of the paper is that in the empirical evaluation, the authors do not explore the case where the samples present more challenging behaviors such as auto-correlations, different types of imbalances, etc.\n\nOther minor suggestions:\n- The colors of the thetas in the introduction are hard to see. \n- Describing why the bounds are tight could further help the reader in section 2.\n\nQUESTIONS:\nI'm thinking of your paper as an approach to test whether the labels and the generated features are connected. This reminds me of [1] where they evaluate the mutual information via neural networks. I'm wondering both if it makes sense in your setting to estimate the divergence via the Donsker-Varadhan representation, and in their setting if your feature generation approach is beneficial. Do you have any opinions on that?\n\nThe same has applications for other approaches on independency testing via classifiers where your feature generation approach also has potential.\n\n[1] Belghazi, Mohamed Ishmael, et al. \"Mutual information neural estimation.\" International conference on machine learning. PMLR, 2018.\n\nLIMITATIONS:\nThere are no potential negative societal impacts from this work. The authors mention limitations in a future work section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider conducting frequentists tests for simulation-based inference. Specifically, the authors want to test if an inference engine q(theta | y) produces samples from the true posterior p(theta | y). The key idea is to use classifiers to determine similarities between samples generated from the true and approximated posterior (similar to GAN). \n\nThe foundation of the algorithm is laid in section 3, where the authors show the negative cross-entropy is associated with a generalised divergence between p and q. In particular, D4 becomes KL divergence between p and q as the number of posterior draws goes to infinity. Empirical study validate the theoretical results and shows the effectiveness of the algorithm.\n\nSTRENGTHS:\n1. This paper is generally well-written and the idea is intuitive and sound.\n2. This paper addresses an understudied problem (SBC) in the literature, and the expansion from a single-dimensional rank statistic to a multidimensional test statistic seems to be a significant improvement\n3. The algorithm provides theoretical support (Section 3) to the proposed algorithm.\n\nWEAKNESSES:\n1. From the methodological point of view, using a classifier to compare distributions is not a new idea (as the authors have discussed in Section 6. \n2. When training a classifier, it seems authors are facing very imbalanced classes and training an imbalanced classifier itself can be difficult. I am wondering if this would cause issues in the tests performed later.\n\nQUESTIONS:\nIf one wants a divergence KL(p(theta|y, q(theta|y)), one can directly approximate it without using a classifier, right? \n\nKL(p(theta|y), q(theta|y) = KL(p(theta, y), q(theta|y)p(y)). \n\nSuppose you can sample from p(theta, y), q(theta|y) and p(y). Then we can directly approximate above KL using f-GAN,  and perform a permutation test or compute CI using bootstrap.  \n\nThis approach sounds like a simpler way to obtain a divergence measure between p and q, so I wonder why the authors did not explore this idea nor use it as a benchmark method. \n\n----------\n\nRevision after author's response. \n\nThis approximation cannot be easily applied due to the IID-ness and the sequential nature of SBI as authors mentioned.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work generalizes the well known Simulation-Based Calibration method to a setting where the tests on the posterior approximation consistency are based on classifiers. The paper has a nice balance between theoretical discussion (Section 3) and more practical issues (Section 4) making it easy for readers to appreciate different aspects of their proposal.\n\nThe paper is solid and tells a nice story from start to finish, but I would be tempted to point out the limitation of assessing only the quality of the posterior approximation **in average**. The authors mention this downside but don't give any insight regarding in which situations this might be a problem and when it could be OK to abandon local information.\n\nFurthermore, it comes somewhat as a surprise the fact that the authors use a **discriminative** approach for assessing the quality of the posterior approximation, but make absolutely no mention to the well known classifier two-samples test (C2ST).\n\nIn all, I'm rather satisfied by the paper as a contribution to the SBI literature, but I'm not certain of how wide is its scope for the NeurIPS conference. I'm giving it, therefore, a score of 6 (borderline accept), and would not be surprised if in the end it was not accepted to the conference main track.\n\nSTRENGTHS:\n- The authors managed to recast the simulation based calibration (SBC) method in a discriminative framework. This is very nice as it helps better understanding such method and see how it compares to other approaches.\n- The three theorems presented in Section 3 are of great practical utility and ensure that even though we only estimate a classifier between samples from a fixed-size dataset, we can be sure that its performance will serve as a lower bound to other theoretical quantities. In other words, the fact of actually having to abide with the \"real life\" setting of training a classifier from limited data is not a problem.\n\nWEAKNESSES:\n- The mathematical notation can be sometimes a bit confusing. For instance, it is not always easy to discern what is a scalar input variable to a function with what is a random variable samples from a pdf.\n\n- There's no comment on assessing the quality of the classifier trained on calibration data and then used to build the statistical tests. \n\n- It could have been of interest to consider more examples on simulated data to illustrate the procedure. For instance, using examples from the `sbi-benchmark` which are becoming more and more used in the simulation-based inference literature.\n\nQUESTIONS:\n- Why is your Figure 2 **after** the set of Figures 3-6? This is very disturbing...\n\n- How useful it is to have a diagnostics to $q(\\theta \\mid y)$ which is only valid in average over $y$? Can you tell a bit more about this?\n\n- The class imbalance in your Example 1-4 will probably make your classifier have a lot of trouble to discriminate one class from the other. Indeed, it will become more of an \"anomaly detection\" kind of thing than an actual classification. You don't make much comments on this but I think it would be very important to explain whether this is a difficulty or not.\n\n- Can you add a reference to your claim in Line 82?\n\n- Line 188: What do you mean with \"classifier $c$ is good enough\" ?\n\nLIMITATIONS:\nI've mentioned the limitations in the previous fields.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}