Validation Diagnostics for SBI algorithms based on
Normalizing Flows

Julia Linhart
Université Paris-Saclay, Inria, CEA
Palaiseau 91120, France
julia.linhart@inria.fr

Alexandre Gramfort
Université Paris-Saclay, Inria, CEA
Palaiseau 91120, France
alexandre.gramfort@inria.fr

Pedro L. C. Rodrigues
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK
Grenoble 38000, France
pedro.rodrigues@inria.fr

Abstract
Building on the recent trend of new deep generative models known as Normalizing Flows (NF), simulation-based inference (SBI) algorithms can now efficiently
accommodate arbitrary complex and high-dimensional data distributions. The
development of appropriate validation methods however has fallen behind. Indeed,most of the existing metrics either require access to the true posterior distribution,or fail to provide theoretical guarantees on the consistency of the inferred approximation beyond the one-dimensional setting. This work proposes easy to interpret
validation diagnostics for multi-dimensional conditional (posterior) density estimators based on NF. It also offers theoretical guarantees based on results of local
consistency. The proposed workflow can be used to check, analyse and guarantee
consistent behavior of the estimator. The method is illustrated with a challenging
example that involves tightly coupled parameters in the context of computational
neuroscience. This work should help the design of better specified models or drive
the development of novel SBI-algorithms, hence allowing to build up trust on theirability to address important questions in experimental science.

1

Introduction

Recent advances in computing have led to a new generation of expressive simulators used to study
complex systems in many scientific fields [4]. They implicitly encode the intractable likelihoodp(x | θ) of the underlying mechanistic model which relates observed data x ∈ Rd to scientificallymeaningful internal parameters θ ∈ Rm . To perform statistical inference in this setting, one canrecur to simulation based inference (SBI) [4] to approximate the posterior distribution p(θ | x) using
samples from the joint pdf p(x, θ). In this work, we consider SBI methods based on normalizing
flows [14], which are invertible neural networks that can be trained via maximum likelihood. Oncethe flow is trained, and for any new observation x, one can directly evaluate the estimated density
over the parameter space (i.e. θ-space) [6], draw samples to construct confidence regions [11], etc.
Flow-based SBI has been used in many recent applied works [10, 1, 6], but it lacks an importantfeature before becoming a technology for experimental science: validation. Ideally, one would
like to have a method that provides finite-sample guarantees of nominal coverage (or calibration)of the estimated posterior regions, but also ensures that the approximation fits the true underlying
posterior of the model when new data is observed. While solutions have been proposed to provide
such finite-sample guarantees [5, 11], assessing the convergence and consistency of the underlying
Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

inference remains a challenging task [10]. Simulation based calibration (SBC) [18] is arguably themost popular metric for validating posterior approximations in the applied SBI community [3], but it
only provides necessary conditions for consistency and fails to give any insight on the local behaviorof the estimator. Moreover, it is a univariate procedure, thus ignoring any information about the
coupling between parameters. Zhao et al. [19] recently proposed local coverage tests (LCT). Theyleverage machine learning to evaluate their test quantities on different locations of the feature space(i.e. x-space). In 1D, the probability integral transform (PIT) provides necessary and sufficientconditions for consistency, which is not the case for the proposed multivariate extensions (e.g. HPD).
In this work, we present a multivariate version of LCT for density models based on normalizing
flows. Our method comes with the same theoretical guarantees on local consistency and interpretablediagnostics as provided by [19] in 1D. We also present a workflow that can be used as a practicaluser guide. Lastly, we provide numerical illustrations on a well known model from computational
neuroscience [8], and demonstrate the importance of trustworthy diagnostics for multivariate posterior
distributions when correlations between parameter variables can play an important role.

2

Methods

Our conditional density estimator qϕ is a normalizing flow defined for samples θ ∈ Rm and isconditioned on observations x ∈ Rd . It uses a Gaussian base distribution p(z) = N (0, Im ) and abijective transformation defined for every x, Tϕ (.; x) := (Tϕ,1 (.; x), . . . , Tϕ,m (.; x)), with Jacobian−1
JTϕ (.; x), such that qϕ (θ | x) = p(Tϕ (z; x)) = p(z) det JTϕ (z; x).
Our goal is to evaluate the local consistency [9] of qϕ with respect to the true posterior density, i.e.
whether for a given x the following null hypothesis holds:
H0 (x) : qϕ (θ | x) = p(θ | x),∀θ ∈ Rm .

(1)

We define the multivariate probability integral transform (PIT) of θ at x and associated to qϕ as thevector of m one-dimensional projections:
PITm (θ, x, qϕ ) = [P1 (θ, x), . . . , Pm (θ, x)],−1
Pi (., x) = FN (0,1) ◦ Tϕ,i(.; x), ∀i ∈ [1, m](2)

where FN (0,1) is the cumulative distribution function (c.d.f.) of the univariate normal distribution.
Theorem 1: Local Consistency and multivariate PIT. For any x ∈ Rd , the null hypothesis H0 (x)holds if, and only if, the covariates of PITm (θ, x, qϕ ) conditioned on x are mutually independent anduniformly distributed over (0, 1). We refer to Appendix A for a detailed proof of this result.
p(θ | x) = qϕ (θ | x) ⇐⇒ p(Tϕ−1 (θ, x) | x) = p(z)−1−1⇐⇒ p(Tϕ,1(θ, x), . . . Tϕ,m(θ, x) | x) = N (0, Im )(p(Pi (θ, x) | x) = U(0, 1) ∀i ∈ [1, m] and⇐⇒{Pi (θ, x) | x}i=1,...,m are mutually independent

We can now verify the null hypothesis of local consistency H0 (x) (1) via m statistical tests for theuniformity of the 1D local PIT covariates and an additional test for their mutual independence.
Multivariate Local Coverage Tests (LCT). Noting that for every i = 1, . . . , m we have
Pi (θ, x) | x ∼ U(0, 1) ⇐⇒ ∀α ∈ [0, 1]

ri,α (x) = P(Pi (θ, x) ≤ α | x) = α

and following the same approach as in [19], we propose m test statistics
2
1 X
Ti (x) :=rbi,α (x) − α∀i ∈ [1, m] ,
|G|

(3)

(4)

α∈G

where G is a grid of α-values and the estimators rbi,α are obtained by regressing I{Pi (θ,x)≤α} on x,
which is optimal for E[I{Pi (θ,x)≤α} | x] = ri,α (x) (when using appropriate loss-functions [12]). We
refer to Algorithm 1 (resp. 2) in [19] for computing the p-values (resp. confidence bands) associatedto each test. Since there are m independent tests, we recur to a Bonferroni correction of the p-values2(resp. confidence levels) [2]. These tests come with interpretable graphical diagnostics such as
PP-plots or histograms that depict distributional deviations in 1D, as shown in Figure 1. If any of theuniformity tests is rejected, we reject H0 (x). If not, we proceed to the mutual independence test.
We now assume that the covariates of PITm conditioned on x are uniformly distributed over (0, 1),
which also means that every coordinate of the flow-transformation Tϕ−1 (θ, x) follows a normal
distribution given x (cf. Theorem 1). Their mutual independence is thus characterized by a covariancematrix equal to the identity Im . We are currently working on how to perform this check in practice.
The workflow in practice. Let D = {xn , θn }Nn=1 be a calibration dataset with (xn , θn ) ∼ p(x, θ)which were not used to train qϕ . We use D to calculate the PIT values PITm (θn , xn , qϕ ) (2) andestimate our local test quantities as functions of x. We investigate the consistency of qϕ in two parts:(1) Global consistency check: first, we look at the global PIT-distribution, i.e. on average over theentire x-space. More specifically, we directly compute the empirical approximation ofPNri,α = P(Pi (θ, x) ≤ α) ≈ N1 n=1 I{Pi (θn ,xn )≤α} , ∀i ∈ [1, m]
with samples from D to test the global uniformity of each PIT-covariate and check which one(s)might be responsible for making qϕ deviate from the true posterior distribution. Note that
passing such global test is only a neccessary condition for consistency of qϕ , as it is insensitive tocovariate transformations in x-space [19] (and ignores the condition on mutual independence).
(2) Local consistency check: we construct m × |G| transformed datasets from D
Di,α = {(xn , Wni,α )}N
n=1 ,α ∈ G,∀i ∈ [1, m]

where Wni,α = I{Pi (θn ,xn )≤α} . We can then compute the test statistics defined in (4) for any new
observation x and check whether H0 (x) should be rejected or not. In the latter case we proceed
to the mutual independence test. Only then can we conclude on the validity of H0 (x) according
to Theorem 1. If the check in (1) passes, this allows us to guarantee (or reject) consistency
anywhere in x-space. Even if the check in (1) does not pass, analyzing local consistency allows
to ‘open the box’ and better understand why and where in x-space the estimator fails. In suchsituations – as the goal is not to guarantee local consistency – it can be enough to test forthe covariate-wise uniformity of PITm , putting the test for mutual independence aside. Thisapproach was adopted for our numerical illustrations in Section 3.

3

Numerical illustrations

We apply our method to check the validity of a posterior estimate qϕ of the Jansen and Rit neural massmodel (JRNMM) [8]. This well known model from computational neuroscience takes parametersθ = (C, µ, σ, g) ∈ R4 as input and generates time series x ∈ R1024 with properties similar to brain
signals obtained in neurophysiology. Parameter C influences the oscillatory behavior of the signalsand (µ, σ) characterize their amplitude. The gain factor g rescales the signals and models the effectsof the amplifier used for measuring them in practice. Note that the coupling-effect of g and (µ, σ) on
the amplitude leads to intrinsic indeterminacies in the inversion of the model [16]. Our approximation
qϕ is a conditioned masked autoregressive flow (MAF) [13] with 10 layers and implemented in thesbi package [20]. Based on the setup in [16], we train qϕ on 50 000 simulations from the JRNMM
with a uniform prior defined on physiologically relevant values (cf. pair-plots in Figure 2).
Results and discussion. All test quantities are computed with a calibration dataset D containing
10 000 simulations and a grid of |G| = 100 α-values in [0, 1]. Following the workflow describedin Section 2, we first check the global consistency of qϕ . In the left part of Figure 1, our graphicaldiagnostics illustrate how the c.d.f. for every PIT-covariate deviates from the identity function (black
dashed line), outside of the 95%-confidence region (in gray), thus rejecting the null hypothesis ofglobal consistency. We compare our method to SBC (right part of Figure 1) as implemented in thesbi package. We observe that SBC is unable to detect any inconsistencies in qϕ .
We proceed to investigate the local consistency of qϕ on different locations of x-space. Note thatsince the global test for consistency has not passed, we may focus on the uniformity of the local
PIT-covariates (cf. (2) of the workflow in Section 2). We consider a 1D subspace in θ-space, where(C0 , µ0 , σ0 ) are fixed and the gain g0 varies in [−20, 20] to generate observations x0 . The upper-leftpart of Figure 2 shows how the test statistics evolve with g0 . We observe a strong deviation from3

Figure 1: Global Consistency of the JRNMM
posterior estimator: PP-plots and corresponding histograms obtained for every global PIT
covariate Pi (θ) (right) and for SBC applied
to every element θi of θ (left). The gray zone
indicates the 95%-confidence region of acceptance outside of which the uniformity test is
rejected. SBC fails to reject the null hypothesis of global consistency.

Figure 2: Local Consistency Analysis. (Left) Diagnostics obtained for every PIT-covariate, whenevaluated for different x0 , simulated via the JRNMM with fixed parameters (C0 , µ0 , σ0 ) and variable
g0 in [−20, 20]. The test-statistics are plotted as a function of g0 and indicate inconsistent behavior(values are not constantly close to zero). Below, the PP-plots report the nature (bias/dispersion) of
these inconsistencies for g0 ∈ {−20, 0, 20}. The gray zone indicates the 95%-confidence region of
acceptance outside of which the uniformity test is rejected. (Right) Pair-plots of qϕ (θ | x0 ) withground-truth parameters θ0 = (C0 , µ0 , σ0 , g0 ) (black dots and dashed lines) for g0 ∈ {−20, 0, 20}.

uniformity for covariates P2 and P3 : they vary smoothly in a ‘U-shape’, with higher values as g0
deviates from zero. We also generate local PP-plots (lower-left part of Figure 2) and observe positive(resp. negative) bias for small (resp. high) values of g0 . Our procedure shows that there are certain
locations in x-space (here g0 = 0) where qϕ performs well (i.e. test statistics are close to zeroand PP-plots show little deviations from the 95%-confidence region in gray), even though global
consistency does not hold. These observations can be compared to the pair-plots (right part of Figure2) which shows the estimated posterior density over the entire θ-space. We observe that gain values
at the boundary of the prior support (g0 = −20 and g0 = 20) induce discrepancies (cf. marginals
plotted in the diagonals of the pair-plots in Figure 2) that are detected by our PIT-based diagnostics.
The above results are obtained by directly applying the method from [19] using their default regressor,
MLPClassifier from scikit-learn [15]. Indeed, this model is well suited for the binary target variables W i,α and, based on neural networks, is able to scale to high-dimensional x-spaces. Furthermore,
it has been shown [12] that for a large enough dataset D (and number of training iterations), thecross-entropy loss converges to the optimal solution of our regression problem. Note that althoughsuch method allows to capture and visualize local discrepancies, it does not cope well with our
high-dimensional, complex data-distributions (high variances represented by large confidence regions
computed over 100 trials). Also, it requires the training of a large number (m × |G|) of regressors(one for each θ-dimension and α-value). Finally, this method is only applicable for the uniformity
tests. We are currently investigating algorithms that are computationally more efficient and moreaccurate in estimating the full multivariate local PIT-distribution.
4

Conclusions. Numerical illustrations demonstrate that our diagnostics capture well the inconsistencies in qϕ with respect to the true JRNMM posterior. To be precise, the proposed validation method isstatistically more powerful and computationally more efficient than SBC and, importantly, it allows
for a local analysis that reveals where in x-space the estimations should be improved. Our method
exploits useful properties of modern normalizing flows. Indeed, contrarily to other SBI-strategies [7],they allow for efficient density evaluation and use bijective transformations involving the mixing
between elements of θ. These transformations can be interpreted as a set of 1D-projections of θ,
including information about its joint p.d.f. (i.e. interactions between its elements θi ). The covariatesof our multivariate PIT therefore define fast and easy to compute 1D test-quantities on which we canperform univariate LCTs that do not completely ignore correlations in the θ-space. Combined with atest to check their mutual independence, this would provide theoretical guarantees for consistency,
which is not the case for HPD, the existing multivariate version of LCT proposed in [19].
Ongoing work will investigate the link between PIT-covariates and their deviations from uniformity
with the actual parameters of the model. This could reveal the true nature of their coupling which canbe used for the development of better specified models (e.g. hierarchical posterior estimation [16, 17]).
Ultimately, the goal is to use our diagnostics as a tool for model selection.

Broader Impact
This work tackles an important open question in simulation based inference. We introduce theoretically valid and interpretable validation diagnostics that scale to both high-dimensional data andparameter spaces. Our contribution should help to further improve SBI methods and drive the designof better specified models, hence allowing to build up trust on their ability to address importantquestions in experimental science. Moreover, our work is in line with many other tentatives in themachine learning community of ensuring the quality and calibration of complex models based onneural networks.

Acknowledgments
Julia Linhart is recipient of the Pierre-Aguilar Scholarship and thankful for the funding of theCapital Fund Management (CFM). Alexandre Gramfort thanks the support of the ANR BrAIN(ANR-20-CHIA0016) grant.

References[1] Sean R. Bittner, Agostina Palmigiano, Alex T. Piet, Chunyu A. Duan, Carlos D. Brody, Kenneth D. Miller, and John Cunningham. Interrogating theoretical models of neural computation
with emergent property inference. eLife, 10, 7 2021. ISSN 2050084X. doi: 10.7554/eLife.56265.
[2] J. Martin Bland and Douglas G. Altman. Multiple significance tests: the bonferroni method. BMJ(Clinical research ed.), 310:170, 1 1995. ISSN 0959-8138. doi: 10.1136/BMJ.310.6973.170.
[3] Jan Boelts, Jan-Matthis Lueckmann, Richard Gao, and Jakob H. Macke. Flexible and efficientsimulation-based inference for models of decision-making. bioRxiv, page 2021.12.22.473472,
6 2022. ISSN 2050084X. doi: 10.1101/2021.12.22.473472.
[4] Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation-based inference.
Proceedings of the National Academy of Sciences (PNAS), 117:30055–30062, 2020. ISSN
0027-8424. doi: 10.1073/pnas.1912789117.
[5] Niccolò Dalmasso, David Zhao, Rafael Izbicki, and Ann B. Lee. Likelihood-free frequentist
inference: Confidence sets with correct conditional coverage. 7 2021. doi: 10.48550/arxiv.2107.
03920.
[6] Pedro J. Gonçalves, Jan Matthis Lueckmann, Michael Deistler, Marcel Nonnenmacher, Kaan
Öcal, Giacomo Bassetto, Chaitanya Chintaluri, William F. Podlaski, Sara A. Haddad, Tim P.
Vogels, David S. Greenberg, and Jakob H. Macke. Training deep neural density estimators to
identify mechanistic models of neural dynamics. eLife, 9:1–46, 9 2020. ISSN 2050084X. doi:
10.7554/ELIFE.56261.
5[7] Joeri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free MCMC with amortizedapproximate ratio estimators. In Hal Daumé III and Aarti Singh, editors, Proceedings of the37th International Conference on Machine Learning, volume 119 of Proceedings of Machine
Learning Research, pages 4239–4248. PMLR, 13–18 Jul 2020. doi: 10.48550/arxiv.1903.04057.
[8] Ben H. Jansen and Vincent G. Rit. Electroencephalogram and visual evoked potential generation
in a mathematical model of coupled cortical columns. Biological Cybernetics 1995 73:4, 73:
357–366, 9 1995. ISSN 1432-0770. doi: 10.1007/BF00199471.
[9] Ilmun Kim, Ann B. Lee, and Jing Lei. Global and local two-sample tests via regression.
Electronic Journal of Statistics, 13:5253–5305, 12 2018. ISSN 19357524. doi: 10.48550/arxiv.
1812.08927.
[10] Jan-Matthis Lueckmann, Jan Boelts, David S Greenberg, Pedro J Gonçalves, and Jakob H
Macke. Benchmarking simulation-based inference. Proceedings of The 24th International
Conference on Artificial Intelligence and Statistics (PMLR), 130:343–351, 4 2021. doi: 10.
48550/arxiv.2101.04653.
[11] Luca Masserano, Tommaso Dorigo, Rafael Izbicki, Mikael Kuusela, and Ann B. Lee.
Simulation-based inference with waldo: Perfectly calibrated confidence regions using any
prediction or posterior estimation algorithm. 5 2022. doi: 10.48550/arxiv.2205.15680.
[12] John W. Miller, Rod Goodman, and Padhraic Smyth. On loss functions which minimii to conditional expected values and posterior probabilities. IEEE TRANSACTIONS ON INFORMAITON
THEORY, 39, 1993.
[13] George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density
estimation. Advances in Neural Information Processing Systems (NeurIPS), pages 2339–2348,
12 2017. ISSN 10495258. doi: 10.48550/arxiv.1705.07057.
[14] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji
Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of
Machine Learning Research, 22:1–64, 2021. ISSN 15337928. doi: 10.48550/arxiv.1912.02762.
[15] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,
P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher,
M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine
Learning Research, 12:2825–2830, 2011.
[16] Pedro L C Rodrigues, Thomas Moreau, and Gilles Louppe. Hnpe: Leveraging global parameters
for neural posterior estimation. Neural Information Proceedings (NeurIPS), 12 2021. ISSN
10495258. doi: 10.48550/arxiv.2102.06477.
[17] Louis Rouillard and Demian Wassermann. ADAVI: Automatic Dual Amortized Variational
Inference Applied To Pyramidal Bayesian Models. In ICLR 2022, Virtual, France, April 2022.
doi: 10.48550/arXiv.2106.12248.
[18] Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating
bayesian inference algorithms with simulation-based calibration. 4 2018. doi: 10.48550/arxiv.
1804.06788.
[19] David Zhao, Niccolò Dalmasso, Rafael Izbicki, and Ann B. Lee. Diagnostics for conditional
density models and bayesian inference algorithms. PMLR, 2 2021. ISSN 2640-3498.
[20] Álvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan,
Pedro J Gonçalves, David S Greenberg, Jakob H Macke, Computational Neuroengineering, andT U Munich. Sbi – a toolkit for simulation-based inference. Journal of Open Source Software,
5:2505, 7 2020. ISSN 2475-9066. doi: 10.21105/joss.02505.

6

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes] Theoretical results and workflow in Section 2, numericalillustrations and discussion in Section 3.
(b) Did you describe the limitations of your work? [Yes] See Section 3: last paragraph in"results and discussion" and "conclusions".
(c) Did you discuss any potential negative societal impacts of your work? [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] At the beginningof Section 2, right before introducing the multivariate PIT.
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] The exact script
for results reproduction is not provided. However, we refer (in section 3) to the codefrom different works that were directly used to simulate data, train the normalizing
flow and estimate local test-quantities (modula minor changes like python package
updates in sbi).
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] Data splits (size of training and calibration datasets) and numberof layers for the normalizing flow are explicitly mentioned. All other hyper parameters
are implicit by referring to the used implementations.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] As our work does not compare ML-models, no such
results are needed.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [No]4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] This includes theimplementation of the simulator, the normalizing flow, posterior estimation and theunivariate LCT method.
(b) Did you mention the license of the assets? [N/A](c) Did you include any new assets either in the supplemental material or as a URL? [No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A] No need, we work with simulated data and the implementation
for the simulator is publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A] No need, we work with simulated data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]7

A

Appendix

Proof of Theorem 1. Let x ∈ Rd . We consider the random variables defined on Rm : Θ ∼ p(θ | x)and Θϕ ∼ qϕ (θ | x). Our local null hypothesis (cf. Equation 1), can be rewritten asH0 (x) : Prob Θ ∈ Ωθ = Prob Θϕ ∈ Ωθ∀Ωθ ⊂ Rm
(5)
From the definition of our normalizing flow qϕ with bijective transformation Tϕ and normal basedistribution, there exists Z ∼ p(z) = N (0, Im ), such that Θϕ = Tϕ (Z, x).
Replacing Θϕ by Tϕ (Z, x) in (5) and defining Ωz = Tϕ−1 (Ωθ ; x), we have








Prob Θ ∈ Ωθ = Prob Tϕ (Z; x) ∈ Ωθ⇐⇒ Prob Tϕ−1 (Θ; x) ∈ Ωz = Prob Z ∈ Ωz
−1Considering Tϕ,i, the i-th coordinate of Tϕ−1 , and Z1 , . . . , Zm the mutually independent and normally
distributed covariates of Z ∼ N (0, Im ), we can rewrite the right part of the above equivalence as:
m

 Y


−1
−1
Prob Tϕ,1(Θ; x) ∈ Ωz,1 ∩ · · · ∩ Tϕ,m
(Θ; x) ∈ Ωz,m =Prob Zi ∈ Ωz,i

(6)

i=1

where the Ωz,i are projections of the set Ωz on each of its dimensions.
Note that since the above equivalence is true for any choice of Ωθ , hence of Ωz , we can write what
happens when we fix a given Ωz,1 and let Ωz,2 = · · · = Ωz,m = R:




−1
Prob Tϕ,1(Θ; x) ∈ Ωz,1 = Prob Z1 ∈ Ωz,1
Doing the same for all other coordinates and using these equalities in (6), we end up with




−1
∀i ∈ [1, m], ∀Ωz,i ⊂ R, Prob Tϕ,i(Θ; x) ∈ Ωz,i = Prob Zi ∈ Ωz,i
and



−1
−1
Prob Tϕ,1(Θ; x) ∈ Ωz,1 ∩ · · · ∩ Tϕ,m
(Θ; x) ∈ Ωz,p =

m
Y

(7)



−1
Prob Tϕ,i(Θ; x) ∈ Ωz,i (8)i=1where (8) is the definition of mutual independence itself.
We now apply the c.d.f. FN (0,1) to each random variable in (7) and (8). Mutual independence stays
true and the probability integral transform theorem (in 1D) states that FN (0,1) (Zi ) ∼ U(0, 1). We
therefore get that the null hypothesis H0 (x) holds if, and only if,(−1
Pi (Θ, x) = FN (0,1) (Tϕ,i(Θ; x)) ∼ U(0, 1), ∀i = [1, m] and(9){Pi (Θ, x)}i=1,...,m are mutually independent
The result in Theorem 1 directly follows from rewriting (5) and (9) with initial notations from Section
2: remember that Θ ∼ p(θ | x), so Pi (Θ, x) ∼ p(Pi (θ, x) | x) and we get:(p(Pi (θ, x) | x) = U(0, 1) ∀i ∈ [1, m] andp(θ | x) = qϕ (θ | x) ⇐⇒{Pi (θ, x) | x}i=1,...,m are mutually independent
where Pi (θ, x) is the ith covariate of PITm (θ, x, qϕ ), the multivariate PIT of θ at x, associated to qϕ .
Conclusion: The null hypothesis H0 (x) holds if, and only if, the covariates of PITm conditioned on
x are mutually independent and uniformly distributed over (0, 1).

8