Contrastive Neural Ratio Estimation
Authors: Benjamin Kurt Miller
Christoph Weniger
Patrick Forré
Venue: NeurIPS 2022 Accept (2022)
Keywords: simulation-based inference
likelihood-free inference
implicit likelihood
posterior
likelihood-to-evidence ratio estimation
neural ratio estimation
contrastive learning

Abstract: Likelihood-to-evidence ratio estimation is usually cast as either a binary (NRE-A) or a multiclass (NRE-B) classification task. In contrast to the binary classification framework, the current formulation of the multiclass version has an intrinsic and unknown bias term, making otherwise informative diagnostics unreliable. We propose a multiclass framework free from the bias inherent to NRE-B at optimum, leaving us in the position to run diagnostics that practitioners depend on. It also recovers NRE-A in one corner case and NRE-B in the limiting case. For fair comparison, we benchmark the behavior of all algorithms in both familiar and novel training regimes: when jointly drawn data is unlimited, when data is fixed but prior draws are unlimited, and in the commonplace fixed data and parameters setting. Our investigations reveal that the highest performing models are distant from the competitors (NRE-A, NRE-B) in hyperparameter space. We make a recommendation for hyperparameters distinct from the previous models. We suggest a bound on the mutual information as a performance metric for simulation-based inference methods, without the need for posterior samples, and provide experimental results.

Contrastive Neural Ratio Estimation
Benjamin Kurt Miller University Amsterdam b.k.mille mil.leruuva nl
Christoph Weniger University of Amsterdam с. w.wenggeruua..ll .nl
Patrick Forré University of Amsterdam p pod.forrequva.n nl
Abstract
seeeihood-to-evidenee ratio estimation is usually cast either binary (NRE-A) A) or a multiclass (NRF-B) classification task. contrast to the binary classification framework, the current formulation of the multiclass version has intrinsic and unknown bias term, making otherwise informative diagnostics unreliable. We propose multiclass framework free from the bias inherent to NRE-B at optimum, leaving us in the position to run diagnostics that practitioners depend on. It also recovers NRE-A in one corner case and NRE-B in the limiting case. For fair comparison, we benchmark the behavior of all algorithms in both familiar and novel training regimes: when jointly drawn data is unlimited, when data is fixed but prior draws are unlimited, and in the commonplace fixed data and parameters setting. Our investigations reveal that the highest performing models are distant from the competitors (NRE-A, NRE-B) B)in in hyperparameter space. We make recommendation for hyperparameters distinct from the previous models. We suggest bound on the mutual information as a performance metric for simulation- inference methods, without the need for posterior samples, and provide experimental results.
Introduction
We begin with a motivating example: Consider the task of inferring the mass of exoplanet O, from the light curve observations v. a distant star. We design computer program that maps hypothetical mass 0 to a simulated light curve using relevant physical theory. Our simulator computes æ from 0, but the inverse mapping is unspecified and likely intractable. Simulation- based inference (SBI) puts this problem in probabilistic context [3 64 Although we can- not analytically evaluate it, we assume that the simulator is sampling from the conditional prob- ability distribution p(x: 0). After specifying prior (0), the inverse amounts estimating the posterior p(0) ). This problem setting occurs
0.90
NRE-B
0.89
wo ammele
Y (gamma)
0.88
Classif 0a0.0mmmmm......mmmmmmm....
NRE-A
0.87
0.86
10-1
0.85
0.84
NRE-C
0.83
(ours)
0.82
10-3
0.81
10a
10a
Number of contrastive parameters
Figure 1: Conceptual, interpolated map from in- vestigated hyperparameters proposed algorithm NRE-C
measurement of posterior exactness using the Classifier Two-Sample Test. Best 0.5, worst 1.0. Red dot indicates NRE-A'S hyperparam- eters, = and K [30]. Purple line implies NRE-B [16 with 7 = oo and K Z NRE-C cov- ers the entire plane, generalizing other methods. Best performance occurs with K > and 7 Rs 1, in contrast with the settings of existing algorithms.
across scientific domains 7 10 29] where 0 generally represents input parameters of the simulator and æ the simulated output observa- tion. Our design goal is to produce surrogate model p(0 x) approximating the posterior for any data while limiting excessive simulation.
Density estimation
directly; however, an appealing alternative for can the like-
lihood 2 15
56
posterior 6
42
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

practitioners is estimating a ratio between distributions 12
16
30
Specifically, the 34
69
likelihood-to-evidennee ratio p(0 p(0[=) æ) p( Ø) p(0.))
p(r) p(0)p(=) Unlike the other methods, ratio esti-
n(0)
mation enables easy aggregation of independent and identically drawn data æ. Ratio and posterior estimation can compute bounds on the mutual information and an importance sampling diagnostic.
Estimating p(a 10) can be formulated as a bi- p(æ))
Independent 0 ..... (0 22
Dependent & 2
nary classification task [30), where the classi- fier a o fw x)) distinguishes between pairs (0) sampled either from the joint distribution p(0, x) or the product of its marginals pi (()p(z)) We call it NRE-A. The optimal classifier has
(
] p.0- (o(a))
p(() (.(8))..0)) p- p(p (8))
Simulation
p(0 x
(w, x) log
p(0)
Here, a represents the sigmoid function, o im- plies function composition, and fw is neural network with weights w. As a part of an effort unify different SBI methods and to improve simulation-efficiency Durkan et al. [66] refor- mulated the classification task identify which of K possible 0₂ was responsible for simulating x. We refer as NRE-B. At optimum
Paramater 8
Parameter 92
NRE-C (ours)
txp ohww (o(0) s0111)
Pg log
Kpx log.
K ?
exp oh (a()))
K- exp whol (1)))
NRE-A 7= 1, 11 =1
loo'1
fwl0 (2(1
lorlo - Aw (all
p(0 ax)
NRE-B 00,
Fw (0, a) x log
+ Geo (xx),
(2)
p(0)
log
where an additional bias, Cw Cw (x), appears. Iw
Figure 2: Schematic depicts how the loss is com- puted in NRE algorithms. (0, x) pairs are sampled from distributions ing the loss functions as depicted. NRE-C controls the number of contrastive classes with K and the weight of independent and dependent terms with Po and px- NRE-C generalizes other algorithms. Hyperparameters recovering NRE-A and NRE-B are listed next to the name within the dashed areas. Notation details are defined in Section 2.1
represents another neural network. The cuo (a:) term nullifies many of the advantages ratio esti- mation offers. Cuo (z) can be arbitrarily patholog- ical in æ, meaning that the normalizing constant can take on extreme values. This limits the appli- cability of verification tools like the importance sampling-based diagnostic in Section 2.2
the top of the figure, enter-
The ow (x) term also arises in contrastive learn- ing [1] with Ma and Collins (45) attempting to estimate it in order to reduce its impact. We
will propose a method that discourages this bias instead. Further discussion in Appendix [D]
There is a distinction in deep learning-based SBI between amortized and sequential algorithms which produce surrogate models that estimate any posterior p(0) æ²) or a specific posterior p(0 a.) respectively. Amortized algorithms sample parameters from the prior, while sequential algorithms use an alternative proposal -inebctiinnnncrranning efficiency at the expense of flexibility. Amortization is usually necessary to compute diagnostics that do not require samples from p(0 xx) and amortized estimators are empirically more reliable [3]] Our study therefore focuses amortized algorithms.
Contribution We design more general formulation of as a multiclass problem in which the bias inherent to NRE-B B is discouraged by the loss function and it does not appear at optimum. Figure diagrram the interpolated performance as function of hyperparameters. shows which settings recover NRE-A and NRE-B, also indicating that highest performance occurs with settings distant from these. Figure 2 shows the relationship of the loss functions. We call our framework NRE- and expound the details in Section 2
-eviiddenc ratio estimation
An existing importance sampling diagnostic (30 tests whether a classifier can distinguish p(æ) 0) samples from from samples from p(æ)) weighted by the estimated ratio. We demonstrate that, when estimating accurate posteriors, our proposed NRE-C passes this diagnostic while NRE-B B does not.
'The code for our project be found at htthtps//ggiuub..oo//bmii//////e under the Apache License 2.0.

---

Taking inspiration from mutual information estimation [58]. we propose applying a variational bound on the mutual information between and æ in a novel way-as as an informative metric measuring a lower bound on the Kullback-Leiib Leibler divergence between surrogate posterior estimate Pw (f a:) and p(0) æ:), averaged over p(æ:). Unlike with two-sample testing methods commonly used in machine learning literature [44], our metric samples only from p(0, æ),, which is always available in SBI, and does not require samples from the intractable p æ). Our metric meaningful to scientists working on problems with intractable posteriors. The technique requires estimating the partition function, which can be expensive. We find the metric to be well correlated with results from two-sample tests.
We evaluate NRE-B and NRE-C in fair comparison in several training regimes in Section 3 We perform hyperparameter search on three simulators with tractable likelihood by benchmarking the behavior when (a) jointly drawn pairs (0, x) are unlimited or when jointly drawn pairs (0, x are fixed but we (b) can draw from the prior p(0) without limit or (c) are restricted to the initial pairs. We also perform the SBT benchmark of Lueckmann et al. 44 with our recommended hyperparameters.
2
Methods
The ratio between probability distributions can be estimated using the "likelihood ratio trick" by training classifier distinguish samples [12 19
50 66 669. We first summarize the loss functions of NRE-A and NRE-B which approximate the intractable likelihood---eviddeee ratio r(x ()
27
30
pl 0) We then elaborate our proposed generalization, NRE-C. Finally, we explain piz)
how recover NRE-A and NRE-B within our framework and comment on the normalization properties.
NRE-A Hermans et al. 30 train a binary classifier to distinguish (0, x) pairs drawn dependently p(0,z) x) from those drawn independently p(0)p(z). This classifier is parameterized by neural network fw which approximates log r(z: 0). We seek optimal network weights
w' E arg min w
1
B
log 1 oo o (0 20)))
(0000
+
Σ log oo foo (0
B
, -00))))
(3)
2B
(b)
x (b) p()) and 006) (b() ~ p(0, x)) over B samples. NRE-A's 's ratio estimate converges
to fw log plæe 0) given unlimited model flexibility and data. Details can be found in Appendix A p(a))
NRE-B Durkan et al. 16 train a classifier that selects from among parameters (0.. k which could have generated in contrast with NRE-A's binary possibilities. One of these parameters 0 is always drawn jointly with x. The classifier is parameterized by a neural network Iw which approximates log r(x I). Training is done over B samples by finding
exp De (0 x000)
w" E arg min w
log
(4)
B
exp or (b)
x x(())
111
where ((8)) ..... 0%
p0)) and (t) p( ((())) Given unlimited model flexibility and data
NRE-B's ratio estimate converges to Iw- (0, x)
log p(e + Gw- (w-(x) Details are in Appendix
p(0)
2.1 Contrastive Neural Ratio Estimation
Our proposed algorithm NRE-C trains a classifier to identify which 0 among K candidates is responsible for generating a given æ, inspired by NRE-B, We added another option that indicates æ was drawn independently, inspired by NRE-A. The introduction of the additional class yields a ratio without the specific cw (x) bias at optimum. Define O = (01..., 0x) and conditional probability
(p(01) p(0x) (())
==0
PNRE Tane.c(,,a (0 k)
(5)
(p(0,) pp ( 0x) I...., K
3

---

We set marginal probabilities pΦy = k) = Pk for all k 2 1 and p(y= 0) Po. yielding the relationship Po Kpk. Let the odds of any pair being drawn dependently completely
independently be y Кpк We now use Bayes' formula to compute the conditional probability Po
p h
p(y = k
Σ "=
ply 2lu=th/eeaaly==///0e221=
= 10==
0)
(6)
K
k=0
K
スナッΣ T( T(
0.1
1=1....,, K
K+> > r( [0)
We dropped the NRE-C subscript and substituted in y replace the p(y) class probabilities. We train a classifier, parameterized by neural network No (0, x) with weights w, to approximate (6) by
K K+y Σ exp (0,,.)
qw - k
y exp olta 20)
I....,K.
K 2i exp (0..)
We note that 7 still satisfies Σ Tu ( my= = k O = no matter the parameterization.
Optimization We design loss function that encourages Nww (0, log p(a Ø)
p(a) convergence,
and holds at optimum with unlimited flexibility and data. We introduce the cross entropy loss (w) == e)) fw lw
x]]
Po Pr.p((e,a(==0)) log tw (y = 0 O x)
ee ly= [log two ( = le O x)
k=1
-PoBp(()=11=0))) log qw (y = 0 O æ>
Kpx ly= [log qw (y K O x)
and minimize it towards w" E arg min [(w). We point out that the final term is symmetric to permutation O, enabling the replacement the sum by multiplication with K. When y and K are known, Po It7 and pk
- under our constraints. Without loss of generality, we let 0....., 0K p00 and æ ~ p(æ 0k)- 3. An empirical estimate of the loss on B samples is therefore
B
(w)) = B
1
log qu
0 ( (b()
b=1
(9)
Σ
log qw y K 1066 =0(1))
In the first term, the classifier sees a completely independently drawn sample and θ while 0x is drawn jointly with in the second term. In both terms, the classifier considers K choices. In practice,
we bootstrap both (²)) ..... (/ and 0(6)) 0% x from the same mini-batch and compare them to the same similarly to NRE-A and NRE-B. Proof of the above is in Appendix B
Recovering NRE-A and NRE-B NRE-C general because specific hyperparameter settings recover NRE-A and NRE-B. To recover NRE-A one should set and K -1 in (9) yielding
exp oh (0(6)) (6())
1
(11(w)=
log
log
2B
1 + exp oh (0()) x((1)
+ exp o (0 x0(61)
W=1
(10)
B
log (1 1 r hew (0()) xb)))
Σ
log (a h hw (0(1') 22,(6))))
b=1

---

where we dropped the lower index. Recovering NRE-B requires taking the limit y - 0o in the loss function. In that case, the first term goes to zero, and second term converges the softmax function.
k0 6o.k(1)) (w)) lim (,,)(
Σ log
exp
oh (0k.
(11)
7-000
K E exp oh (0
determined by substitution into (9) Both equations are obviously the same as their counterparts.
Estimating normalized posterior In the limit of infinite data and infinite neural network capacity (width, depth) the optimal classifier trained using NRE-C (with r E "T) satisfies the equality:
o (0, x)) = log p(0 x))
(12)
p(0)
In particular, we have that the following normalizing constant is trivial:
((x) = exp (hw x)) p(0) d0 = p(0 x)) do =1
(13)
This is result of Lemma iin Appendix However, practitioners never operate in this setting, rather they use finite sample sizes and neural networks with limited capacity that are optimized locally. The non-optimal function exp(h (0, .)) does not have a direct interpretation as a ratio of probability distributions, rather as the function to weigh the prior p(0) to approximate the unnormalized posterior. In other words, we find the following approximation for the posterior p(0 x):
Pw (01z)= exp(1 (0, x)) (()),
(x) = exp (N 0 x p())d0 d0, (14)
Zu (x)
where in general the normalizing constant is not trivial, i.e. 7um * 1. As stated above, the NRE-C (and NRE-A) objective encourages Zuo (ar) to converge to 1. This is in sharp contrast to NRE-B, where at optimum with an unrestricted function class non-trivial æ-dependent bias term can appear.
There is no restriction on how pathological the NRE-B bias ow (w) can be. Consider a minimizer of (4). the NRE-B loss function, /:w-. + Cw (x). Adding any function d(x)) cancels out in the fraction and also a minimizer of (4). This freedom complicates any numerical computation of the normalizing constant and renders the importance sampling diagnostic from Section 2.2 generally inapplicable.
We report Monte Carlo estimates of (x) on a test problem across hyperparameters in Figure
2.2 Measuring performance & ratio estimator diagnostics
SBI is difficult to verify because, for many cases, the practitioner cannot compare surrogate Pw (0) x) to the intractable ground truth p(0 x). Incongruous with the practical use case for SBI, much of the literature has focused measuring the similarity between surrogate and posterior using two-samples tests on tractable problems. For comparison with literature, we first reference two-sample exactness metric which requires tractable posterior. We then discuss diagnostics which do not require samples from p(0 æ). commenting on the relevance for each NRE algorithm with empirical results. Further, we find that known variational bound to the mutual information is tractable to estimate within SBI, that it bounds the average Kullback-Leibler divergence between surrogate and posterior, and propose
for model comparison on intractable inference tasks.
Comparing to tractable posterior with estimates of exactness Assessments of approximate posterior quality are available when samples can be drawn from both the posterior p(0 x)) and the approximation 0 ~ ( æ). In the deep learning-based SBI literature, exactness is measured as function of computational cost, usually simulator calls. We investigate this with NRE-C in Section 3.3
Based on the recommendations of Lueckmann et al. 44 our experimental results are measured using classifier is trained to distinguish samples
the Classifier Two-Sample Test (C2ST) 11 from either the surrogate or the ground truth posterior. An average classification probability on holdout data of 1.0 implies that samples from each distribution are easily identified; 0.5 implies either the distributions are the same the classifier does not have the capacity to distinguish them.
5

---

Approxmmate likelihuzd dersity, various
Truth MIE-C
NRE-C
NRE-B
3.0
Poztrir conkks
2.5
1.5
Diagonal
1.0
NRE C
0.5
-1.
-1.0
-0.5
0.0
0.5
1.0
1.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
Data
False Positivn
Farameter
(b) Likelihood and reweighted marginal generative models.
(c) Receiver operating character- istic.
(a) Posteriors
Figure 3: The figures visualize the importance sampling diagnostic ratio estimators trained using NRE-B and NRE-C. (a) Both methods produce satisfactory posterior estimates that agree with p(0 az). (b) p(æ 0) is shown along with p(æz) samples weighted by NRE-A expo (0, a:) and NRE-B exp ogw (0, 2²). Each plot corresponds to a different 9. Despite high posterior accuracy, the NRE-B estimates are distinct from p(æ 0). (c) Two classifier's 's ROC curves, each trained to distinguish p( 0) samples from p(z) samples weighted by the corresponding NRE'S estimate. The classifier failed distinguish likelihood samples from the NRE-C weighted data samples, but successfully identified NRE-B weighted samples. NRE-B accurately approximates the posterior, but fails the diagnostie. NRE-C produces an accurate posterior surrogate and passes the diagnostic.
Importance sampling diagnostic An accurate teatthoo--t--vvienee weight transforms the data distribution into the likelihood by p(æ () -(<x)(< 0). Since NRE necessitates simulator access, we can test the ratio estimator by training a classifier distinguish unweighted p(x: 0) samples from weighted p(z) F()))(( 0) samples, where r implies an estimate. Indistinguishability between samples implies either that the approximate ratio accurate for parameter 0 or that the classifier does not have sufficient power to find predictive features. Issues with classification power can be detected by assessing the classifier's 's ability to distinguish p(æ) from p( (). The performance can be visualized in a receiver operating curve (ROC) measured by the area under the curve (ROC AUC). This diagnostic has been used for ratio estimators before [2] 30 but it comes from training models under covariate shift [62]. It is particularly appealing because it does not require samples from p(O x)).
Durkan et al. [16] do not mention this diagnostic in their paper, but due to its intrinsic bias NRE-B B does not fulfill the identity necessary for this diagnostic to hold at optimum. The unknown factor that depends on æ implies p(x: ()) # p(æx) exp ogw (ar 0). We provide empirical evidence of this issue in Figure 3 Although NRE-B accurately approximates the true posterior, it demonstrably fails the diagnostic. Given the limited options for verification of SBT results, this presents a major problem by significantly limiting the trustworthiness of NRE-B any problem with an intractable posterior. In Appendix B we show that the unrestricted NRE-B- B-spccccc ow (x) bias means approximating p(æ æ ()) with normalized importance weights will not solve the issue.
Mutual information bound Selecting the surrogate model most-similar to the target posterior remains intractable without access to p(0 x.). Nevertheless, practitioners must decide which surro- gate should approximate the posterior across training and hyperparameter search. Unfortunately, the validation losses between different versions of NRE and different K and settings are not comparable. good heuristic is to choose the model which minimizes the Beilbacck-Liible divergence on average over possible data p(æ). In Appendix D we prove the relationship between [(0; x), the mutual information with respect to p(0, ar), our models' variational bound 7((0) (0; a)), and the average KI.D
Ep(=)) [pLD(p(O 0 x)) Pw (0 [))] [(0xx) - (0) (0: (0;z),
(15)
[!0) (0: x) = Ep) [log f(x [)] Ey((e) [log (o 0)]]
(16)
The negatiiiyy of all terms in (15) implies T(0:x)) 2 {
(0; .:); that means the model which minimizes - (0; .)) best satisfies our heuristic. We propose to approximate - ( (0; ar) with
Monte Carlo using held-out data as metric for model selection during training and across hyper- parameters. The expectation values average over p(0, x). p()), and p(x). We can sample from all

---

of these distributions in the SBI context. Since the second term in (16) computes the average log partition function, our metric can compare NRE-B- B-b-eed surrogates to NRE-C- C-based ones, However, the metric comes with the normal challenges of estimating the log partition function, which can be very expensive. Additionally, the presence of the ratio in the integrand can make this integral high variance. We treat a generally tractable bound in Appendix [D] We go into much more depth there and discuss of the relevance to Neural Posterior Estimation [16 21 42 [3). While the application to SBI is novel, bounds on the mutual information have been broadly investigated for contrastive representation learning and mutual information estimation
22 23 25 58 71].
Empirical expected coverage probability For a candidate distribution to qualify as the posterior, integrating over data must return the prior. A measurement that follows from calibration to the prior is called expected coverage probability. Expected coverage probability can be estimated with samples from p(0, a)) and any amortized SBI method. Although important, ability to compute this metric does not distinguish NRE-C. We refer the interested reader to Hermans et al. [31] We note that popular sequential techniques generally render this diagnostic inapplicable, with exceptions [T0 47 48
3 Experiments
We perform experiments in three settings to measure the exactness of surrogate models under various hyperparameters and training regimes. Section 3.1 aims to identify whether data or architecture is the bottleneck in accuracy by drawing from the joint with every mini-batch. The next experiments aim to determine how optimally extract inference information given limited amount of simulation data. Section 3.2 leverages a cheap prior by drawing new contrastive parameters with every mini-batch. Finally, Section 3.3 applies the commonplace training regime for deep learning-based SBI literature of fixed data and bootstrapped contrastive parameters. In this setting, we also benchmark NRE-C on ten inference tasks from Lueckmann al.
4 using our recommended hyperparameters.
On all hyperparameter searches we consider three simulators from the simulation based inference benchmark, namely SLCP, Two Moons, and Gaussian Mixture 44. SLCP is difficult inference task which has a simple likelihood and complex posterior [21, [6]. Parameters are five dimensional and the data are four samples from two dimensional random variable. Two Moons is two dimensional with a crescent- crescent-shaped shaped bimodal posterior. Finally, the Gaussian Mixture data draws from a mixture of two, wwo-dimensional normal distributions with extremely different covariances B 63 64 77].
Our surrogate models are parameterized by one of these architectures: Small NN is like the benchmark with 50 hidden units and two residual blocks. Large NN has 128 hidden units and three residual blocks. We use batch normalization, unlike the henchmark. We compare their performance on a grid of y and K values. We report post-training results using the C2ST, and mid-training validation loss for insight into convergence rate. We generally use residual networks (28) with batch normalization [33 and train them using adam 337. We also run NRE-B with the same architecture for comparison. To compare with NRE-B we set the number of total contrastive parameters equal.
What does fair comparison mean in our experimentation? We compare models across fixed number of gradient steps. This implies that models with more classes, i.e., greater K, evaluate the classifier- in-training on more pairs at given training step than a model with fewer classes. An alternative which we do not apply in this paper: Vary the number of gradient steps, holding the number of pair evaluations fixed, i.e., model with higher K sees the same number of pairs as a model with lower K but the model with lower has been updated more times. We leave this analysis for future work.
3.1 Behavior with unlimited data
What is responsible for inaccuracies of the surrogate model when training has saturated? (a) the amount of training data (b) the flexibility of the model? In this section we provide new simulation and parameter data with every mini-batch and train until saturation, thereby eliminating the possibility of (a). The newly drawn mini-bater parameters O are bootstrapped for the contrastive pairs.
The setting is similar to REJ-ABC where simulations are drawn until the posterior has converged. The results of this study will provide a baseline to compare with limited-data results and help us understand how the deep learning architecture's limitations are affected by our introduced hyperparameters. The results are reported in the top row of Figure 5
7

---

The trend is that increasing the number of contrastive examples helps NRE-B B and NRE-C. There is not a clear trend with respect to y in the C2ST. Study of the detailed validation losses in Appendix C reveals that high y is associated with higher variance validation loss during training, but here it does not seem to strongly affect convergence rate. Saturation reached before the maximum number of epochs have elapsed and Large NN converges to better result. Although obfuscated in the averaged results in Figure 5 the result is obvious on SLCP, the most difficult inference task that provides an opportunity for more flexible architecture to improve.
We argue that generally the performance bottleneck has to do with the number of contrastive parameters, network flexibility, or training details rather than the amount of data. Despite saturating validation losses, the C2ST improves based on these factors. Intuitively more flexible model could continue to extract information from this unlimited set of jointly drawn (0, x) pairs. This has some consequence because network flexibility may limit performance in the benchmark case. Appendix C contains more detailed results.
3.2 Leveraging fast priors (drawing theta)
In practice, the simulator is often slow but one can often draw from the prior at the same pace as training a neural network for one mini-batch. Our goal is to understand how our hyperparameter are affected by this setting and whether it is valuable use this technique in practice. To our knowledge, this training regime has not been explored in the deep learning- based SBI literature.
Initially we draw fixed set of around 20,000 samples (0,z)) p(0, æ)). For every mini-batch during training, we sample all necessary contrastive parameters from the prior. For each term in l,,k take the same batch of contrastive parameters and reshuffle them, thereby equalizing the number of samples seen by NRE-B and NRE-C, up bootstrap. The averaged C2ST results from the three inference tasks are reported in the middle row of Figure 5
The resulting estimators are markedly less sensitive to the number of contrastive parameters than in Section 3.1 The number of network parameters has a positive effect, although again this is best seen on the hardest task, SLCP, in the Appendix C Empirically, lower values of y improve performance with Large NN; however, there is still high variance and this result may be from noise. In this setting, Y # has a very small negative effect convergence rate compared to unity, validation loss plots in Appendix C Since training has saturated, we claim this result implies drawing contrastive parameters from the prior helps extract the maximum amount of information from fixed simulation data withou being strongly affected by the number of contrastive parameters K.
in the
N
y (gamma))
(gamma)
y (gamma) 100.0
12
paramstara
25 200
102
10'
107
103
10a
10
107
107
Epacho
Epochs
Epscho
(a) NRE-B
(b) NRE-C
Figure 4: Our proposed metric, a negative bound on the mutual information -100) (0; x)). for the SLCP task estimated over the validation set versus training epochs using (a) NRE-B and (b) NRE-C with various values of y and K, Large NN architecture. and fixed training data. The bound permits visualization of the convergence rates and pairwise comparison across models. When K fixed, small y slows convergences but leads to better optima than large 7. When y is fixed, small K slows convergence but leads to better optima than high K. Unlike when computing the validation loss directly, this metric does not exhibit and K dependent bias as seen in Figures 7 and 9
3.3 Simulation- inference benchmark
In this section, we assume the traditional literature setting of limited simulation and prior budget. Once we've 've selected hyperparameters based on grid search of subset of the SBI benchmark, we perform the entire benchmark with those hyperparameters.
8

---

Draw (8.x) Small NN
Draw (8, Large NN
Hyperparameter search To compare to the previous two sections, the amount of training data was fixed such that each epoch was com- parable to Section 3.1] this amounts to about 20,000 samples. We first inspect the C2ST re- sults in the last row of Figure The sensitivity to K appears less than in Section 3.1 but more than in Section 3.2 Smaller slightly improves Large NN's performance but the noise makes this result uncertain. The larger network per- forms better, see Appendix [
0.85
0.50
0.70
0.65
Draw NN
Drow Laarg
0.85
(gammm))
0.001
0.75
1.0
0.70
100.0
We do a Monte Carlo estimate of our proposed metric, denoted - /100 (0; æ)), to show perfor- mance on the SLCP task as function of epochs in Figure For both NRE-B and NRE-C. increas- ing K tends to positively affect the convergence rate. Generally, with fixed and high K, peak performance negatively impacted, in contrast to results from Figure 5 Jusing the C2ST. Gener- ally, with fixed K, small 7 slows convergences but leads to better optima than large y. When 7 fixed, small K slows convergence but leads to better optima than high K. See Figure 12
Algortthm
0.60
MRE-C
Fixed data Small
Fooe data Large
0.80
0.75
0.85
0.60
10:
10F
10
10f
Contractive parameter
Contrastive parsmeter
Figure 5: Exactness of NRE posterior surrogates is computed for various contrastive parameter counts, Y values, and architectures on an average of three tasks from the SB1 benchmark 44). C2ST assigns 1.0 to inaccurate and 0.5 accurate posterior ap- proximation. (top) p(0. x)) was sampled at every mini-batch during training. The accuracy strongly depends on K. (mid) A fixed number of dependent (0, x) pairs were drawn, but p(0) was sampled at every mini-batch during training. In this regime, K has a smaller effect. (bot) The training data is completely fixed. Contrastive parameters drawn in a bootstrap from the batih On the problems with fixed simulation data æt, higher K improves accuracy and small y with larger architec- tures slightly improves performance. The effects of the architecture are more clearly seen on diffi- cult problems like SLCP in Appendix
Our away that 7 E [0.1, 10] has small effect, otherwise learning can become unstable or very slow. 7 = has good compromise on high convergence rate without sacrificing too much performance. Generally, we saw im- proved performance on the C2ST by increasing contrastive parameters K. However, larger K also increases the magnitude of the normalizing constant, and extreme values reduced the perfor- mance on - x)) see Appendix D Since the benchmark compares C2ST, we optimized our architecture based on that metric. Due to bootstrapping parameters from the batched 0, the maximum K is B B/2 without reusing any 0 to compare with æ. This is considering both terms in our loss function.
Benchmark We performed the benchmark tasks from Lueckmann et 44 with NRE-C. Our architecture and hyperparameters deviate based on the results of the last paragraph. We applied the largest number of computationally practical contrastive parameters, namely K 99, and set y 1. Since we identified potential architecture bottlenecks in Sections 3.1 and 3.3 we used Large NN for our results. The averaged results are visible Table Sequential estimates are prepended with an s, all other estimates are amortized. We trained five amortized estimators per task with five seeds, then we predicted the posterior for each of the ten observations æi..... Z10- Lueckmann et al. 44 trained an estimator for every with new seed. The reported C2ST for our method is averaged across ten posteriors per seed with five seeds, i.c., fifty C2ST computations per task. The other methods are averaged across one posterior per seed with ten seeds, i.e., ten C2ST computations per task.
NRE-C performed better than NRE-B at all budgets and generally well among the amortized algorithms. The high performance at 10" simulation budget implies that our design has high capacity and may scale well. Sequential methods naturally lead to high exactness since they are tailored to the specific observation v., but they come with drawbacks since they cannot practically perform empirical expected coverage tests [3 48) nor bound the mutual information, as discussed in Section 2.2
Detailed results per task can be found in Appendix

---

4 Conclusion
We introduced generalization of NRE-A and NRE-B which discourages the NRE-B specific
Table I: Averaged C2ST for various budgets on the simulation-baie paasattoo---add inference benchmark 44 An implies sequential i.e. con-amortized method.
Cw (2) bias term and produces an unbiased es- timate of the likelihood-to- teeiidedd-- ratio at op- timum. This property has implications for the importance sampling diagnostic, which is criti- cal to practitioners. We suggested using a vari- ational bound the mutual information model-selcction metric that could replace the C2ST averaged over several observations. It is significantly more practical since does not require access to the ground truth posterior; how- ever, it remains a lower bound so we don't 't know the overall quality of our estimator. Further dis- cussions and derivations related to the mutual information available in Appendix D
C2ST 10
Simulation budget Algorithm
103
104
NRE-C (ours)
0.853
0.762
0.680
REJ-ABC
0.965
0.920
0.871
NLE
0.826
0.753
0.723
NPE
0.838
0.736
0.654
NRE (NRE-B)
0.867
0.811
0.762
SMC-ABC
0.948
0.873
0.816
SNLE
0.783
0.704
0.655
SNPE
0.796
0.677
0.615
SNRE (SNRE-B)
0.788
0.703
0.610
In the context of NRE-C, we found that increas- ing the number of contrastive parameters K im- proved the C2ST in most cases, at the price of increasing the normalizing constant. Setting Y R 1 was generally the fastest saturate and lower values seemed to slightly improve performance with the Large NN and fixed simulation data. These results indicate that better performance can be achieved by using NRE-C with these hyperparameters than the other version of NRE. When both training parameters 0 and simulation data æ are unlimited, the architecture size plays important role in the quality of surrogate model. We tried drawing contrastive parameters from the prior, a commonly available practical use-case, and found that damped the effect of K on the C2ST. According to the C2ST, for highest accuracy with fixed budget set K large, up to the mini-batch size divided by 2, and 7 R 1.0 for the best convergence rate. When the prior sampled for every mini-batch, applying higher K may be less valuable. In situations where the normalizing constant should be very close to unity, i.c., for practitioners who want to run diagnostics, drawing 0 from the prior with every mini-batch and using smaller K and y is best.
Broader Impact The societal implications of SBI are similar to other inference methods. The methods are primarily scientific and generally lead to interesting discoveries; however, one must be careful to use an accurate generative model and to carefully test empirical results. Model mismatch and untested inference can lead to incorrect conclusions. That is why we emphasize the importance of the diagnostics in our paper. This nuance can be missed by practitioners doing inference in any field; however, special care should be taken when producing inference results that may be used for making decisions in areas like predicting hidden variables that describe human behavior or determining what factors are responsible for climate change. This list non-comprehensive and not specific SBI.
Acknowledgments and Disclosure of Funding
We thank Noemi Anau Montel, James Alvey, Kosio Karchev for reading; Teodora Pandeva for mathematics consultation; Maud Canisius for design consultation; Marco Federici and Gilles Louppe for discussions. This work uses numpy 26). scipy (72). seaborn 73.. matplotlib [32. pandas
52 74], pytorch [57]. and jupyter
38
We want to thank the DAS-5 computing cluster for access to their TitanX GPUs. DAS-5 is funded by the NWO/CC (the Netherlands Organization for Scientific Research). We thank SURF (www.surf.n)) for the support in using the Lisa Compute Cluster.
Benjamin Kurt Miller is funded by the University of Amsterdam Faculty of Science (FNWI), Informat- ics Institute (Ivl), and the Institute of Physics (IoP). We received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (Grant agreement No. 864035 UnDark).
10

---

References
[1] Alsing, B. Wandelt, and Feeney. Massive optimal data compression and density estimation for scalable, freeeihood-free inference in cosmology. Monthly Notices of the Royal Astronomical Sociery, 477(3))2874-288) 2885, 2018.
[2] J. Alsing, T. Charnoc, S. Feeney, and B. Wandelt. Fast likelihood-free cosmology with neural density estimators and active learning. Monthly Notices of the Royal Astronomical Society, 488 (3):4440-4458, 2019.
[3] M. A. Beaumont, J.-M. Cornuet, J.-M. Marin, and C. P. Robert. Adaptive approximate bayesian computation. Biometrika, 964)):833990, 2009.
[4] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.0480 2018.
[5] C. M. Bishop. Pattern recognition and machine learning (information science and statistics),
2006.
[6] M. G. Blum and François. Non-linear regression models for approximate bayesian computa- tion. Statistics and computing, 20(1):63-73, 2010.
[7] J. Brehmer, K. Cranmer, G. Louppe, and J. Pavez. Constraining effective field theories with machine learning. Physical review letters, 121(11):111801 2018.
[8] J. Brehmer, G. Louppe, Pavez, and K. Cranmer. Mining gold from implicit models to improve likelihood-free frree inference. Proceedings of the National Academy of Sciences, 117(10): 2242,9249, 2020.
[9] J. Chan, V. Perrone, J. Spence, P. Jenkins, S. Mathieson, and Y. Song. A likelihood-free inference framework for population genetic data using exchangeable neural networks. Advances in neural information processing systems, 31, 2018.
[10] A. Cole, B. K. Miller, S. J. Witte, M. X. Cai, M. W. Grootes, F. Nattino, and C. Weniger. Fast and credible Akclihoodd-fre cosmology with truncated marginal neural ratio estimation. arXiv preprint
2021.
[11] A. Coogan, K. Karchev, and C. Weniger. Targeted likelihood inference of dark matter substructure in strongly-lensed galaxies. arXiv preprint arXiv:2010.07032, 2020.
[12] K. Cranmer, Pavez, and G. Louppe. Approximating likelihood ratios with calibrated discrimi- native classifiers. arXiv preprini arXiv: /506.0216, 2015.
[13] K. Cranmer, J. Brehmer, and G. Louppe. The frontier of simulation-base inference. Proc. Natl. Acad. Sci. S. A., May 2020.
[14] N. Dalmasso, R. Izbicki, and A. Lee. Confidence sets and hypothesis testing in a likelihood freee inference setting. In International Conference on Machine Learning, pages 2323-2334. PMLR. 2020.
(15) C. C. Drovandi, C. Grazian, K. Mengersen, and C. Robert. Approximating the likelihood in abc. Handbook of approximate bayesian computation, pages 321-368, 2018.
[16] C. Durkan, I. Murray, and G. Papamakarios. On contrastive learning for likelihood-free inference. In International Conference on Machine Learning, pages 2771-2781. PMLR, 2020.
[17] J. H. Friedman. On multivariate goodness-of-fit and two-sample testing. STATISTICAL PROBLEMS IN PARTICLE PHYSICS, ASTROPHYSICS AND COSMOLOGY, page 311, 2003.
[18] M. Glöckler, M. Deistler, and H. Macke. Variational methods for simulation-based inference. In International Conference on Learning Representations, 2021.
[19] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-144 2020.

---

[20] S. Gratton. Glass: A general likelihood approximate solution scheme. arXiv preprint arXiv:1708.08479, 2017.
[21] D. Greenberg, M. Nonnenmacher, and J. Macke. Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning, pages 2414. PMLR, 2019.
[22] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[23] M. U. Gutmann and A. Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13 (2), 2012.
[24] M. U. Gutmann, J. Corander, et al. Bayesian optimization for likelihood-free inference of simulator-based statistical models. Journal of Machine Learning Research, 2016.
[25] M. U. Gutmann, S. Kleinegesse, and B. Rhodes. Statistical applications of contrastive learning. arXiv preprint arXiv:2204. 13999, 2022.
[26] C. R. Harris, K. J. Millman, J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del R 10, M. Wiebe, Peterson, P. Grerdrd-Mo Mraard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant. Array programming with 011888/415860020-2220
NumPy. Nature, 585(7825) 357-336, Sept. 2020. doi:
URL
attps://dot org/ 10. 1 021-5558-000-244933
[27] T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
[28] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1109/CVPR.2016.990
[29] Hermans, N. Banik, C. Weniger, G. Bertone, and G. Louppe. Towards constraining warm dark matter with stellar streams through neural simulation-based inference. arXiv preprint arXiv:2011. 14923, 2020.
[30] Hermans, V. Begy, and G. Louppe. Likelihood meme with amortized approximate ratio estimators. In International Conference on Machine Learning, pages 4239 -248. PMLR, 2020.
[31] J. Hermans, A. Delaunoy, F. Rozet, A. Wehenkel, and G. Louppe. Averting a crisis in simulation- based inference. arXiv preprint 100.552, 2021.
[32] D. Hunter. Matplotlib: 2d graphics environment. Computing in Science & Engineering, (3):90-95, 2007. doi: 10. 109/MCSE.2007.55..
[33] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[34] R. Izbicki, A. Lee, and C. Schafer. High-dimensional density ratio estimation with extensions approximate likelihood computation. In Artificial intelligence and statistics, pages 429- PMLR, 2014.
[35] M. Järvenpää, M. U. Gutmann, A. Vehtari, and P. Marttine.. Parallel gaussian process surrogate bayesian inference with noisy likelihood evaluations. Bayesian Analysis, 16(1):147-178, 2021.
[36] N. Jeffrey and B. D. Wandelt. Solving high nigh-dimensional parameter inference: marginal posterior densities & moment networks. arXiv preprint arXiv:2011.0599) 2020.
[37] D. P. Kingma and J. I.. Ba. Adam: A method for stochastic gradient descent. In ICLR: International Conference Learning Representations, pages 1-15, 2015.
12

---

[38] T. Kluyver. B. Ragan-Kelley, F. Pérez, B. Granger, M. Bussonnier. J. Frederic, K. Kelley, J. Hamrick, J. Grout, S. Corlay, Ivanov, D. Avila, S. Abdalla, C. Willing, and J. development team. Jupyter notebooks a publishing format for reproducible computational workflows. In Loizides and B. Scmidt, editors, Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87-90, Netherlands, 2016. IOS Press. soton. uk/403913/
[39] A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019.
[40] E. L. Lehmann, J. P. Romano, and G. Casella. Testing statistical hypotheses, volume 3. Springer, 2005.
[41] D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations, 2017.
[42] J.-. Lueckmann, P. J. Gonçalves, G. Bassetto, K. Öcal, M. Nonnenmacher, and H. Macke. Flexible statistical inference for mechanistic models of neural dynamics. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1289-1299, 2017.
[43] J.-M. Lueckmann, G. Bassetto, T. Karaletsos, and J. H. Macke. Likelihood-free inference with emulator networks. In Symposium on Advances Approximate Bayesian Inference, pages 32-53. PMLR, 2019.
[44] J.-M. Lueckmann, J. Boelts, D. Greenberg, P. Goncalves, and J. Macke. Benchmarking simulation-base inference. In A. Banerjee and K. Fukumizu, editors, Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 343-351. PMLR, 13-15 Apr 2021. URL http:// proceedings .mw.iingg.mm.................... walr.presse www.prrsssveeee
[45] 7. Ma and M. Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3698-3707, 2018
[46] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[47] B. K. Miller, A. Cole, G. Louppe, and C. Weniger. Simulation-efficies marginal posterior estimation with swyft: stop wasting your precious time. arXiv preprint arXiv:2011.1399 2020.
[48] B. K. Miller, A. Cole, P. Forré, G. Louppe, and C. Weniger. Truncated marginal neural ratio estimation. Advances in Neural Information Processing Systems, 34:129-143, 2021.
[49] A. Mnih and Y. W. Teh. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 419-426, 2012.
[50] S. Mohamed and B. Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv: 2016.
[51] R. M. Neal. Slice sampling. Annals of statistics, pages 705-741, 2003.
[52] T. pandas development team. pandas-dev/pandas: Pandas, Feb. 2020. URL https://doi org/10. wew.//220/eeooooe33 3509134
[53] G. Papamakarios and I. Murray. Fast g-free inference of simulation models with bayesian conditional density estimation. Advances in neural information processing systems, 29, 2016.
[54] G. Papamakarios, T. Pavlakou, and I. Murray. Masked autoregressive flow for density estimation. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 2335-2344 2017.
13

---

[55] G. Papamakarios, E. Nalisnick, J. Rezende, S. Mohamed, and B. Lakshminarayanan. Nor- malizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.0276.. 2019.
[56] G. Papamakarios. D. Sterratt, and I. Murray. Sequential neural likelihood: Fast likelihood- free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 837-848. PMIR, 2019.
[57] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, I.. Antiga, Desmaison, Kopf, Yang. Vitoo, M. Raison, Tejani, S. C.Chiamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high- performance deep learning library.
Wallach, H. Larochelle, A. Beygelzimer, F.d. Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035, Curran Associates, Inc., 2019.
www.:neuuiiicoom 1ibrary,
http://www.....a.aaaaaaeeeeeeeea pdf
[58] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180. PMLR, 2019.
[59] D. Prangle. Distilling importance sampling. arXiv preprint
2019.
[60] Ramesh, J.-M. Lueckmann, Boelts, Á. Tejero-Cantero, D. S. Greenberg, J. Goncalves, and J. H. Macke. Gatsbi: Generative adversarial training for simulation-based inference. In International Conference on Learning Representations, 2021.
[61] G. Rodrigues, D. J. Nott, and S. A. Sisson. Likelihood-fier approximate gibbs sampling. Statistics and Computing, 30(4):0557-1073, 2020.
[62] H. Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
[63] U. Simola, J. Corander, and U. Picchini. Adaptive meme for synthetic likelihoods and correlated synthetic likelihoods. Buyesian analysis, 2020.
[64] S. A. Sisson, Y. Fan, and M. Beaumont. Handbook of approximaate Bayesian computation. CRC Press, 2018.
[65] J. Skilling. Nested sampling for general bayesian computation. Bayesian Anal., 1(4):833-859 Dec. 2006.
[66] M. Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
[67] S. Talts, M. Betancourt, D. Simpson, A. Vehtari, and A. Gelman. Validating bayesian inference algorithms with simulation-based calibration. arXiv preprint arXiv: 1 180. 06788, 2018
[68] A. Tejero-Cantero, J. Boelts, M. Deistler, J.-M. Lueckmann, C. Durkan, P. J. Gonçalves, D. S. Greenberg, and H. Macke, sbi: toolkit for simulation-based inference. Journal of Open Source Software, 5(52):2505, 2020. doi: 10.21 105/5oss.02505. https://dos org/10
21105/joss 02505
[69] O. Thomas, R. Dutta, J. Corander, S. Kaski, M. U. Gutmann, et al. Likelihood-free inference by ratio estimation. Bayesian Analysis, 2016.
[70] T. Toni, D. Welch, N. Strelkowa, A. Ipsen, and M. P. H. Stumpf. Approximate bayesian computation scheme for parameter inference and model selection dynamical systems. J. Soc. Interface, 6(31):187-202, Feb. 2009.
[71] A. Van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints, pages arXiv-1807, 2018.
14

---

[72] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, Peterson, W. Weckesser, J. Bright, S. van der Walt, M. Brett, J. Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, f. Polat, Y. Feng, Laxalde, Perktold, R. Cimrman, I. Henriksen, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272 2020, doi: 10. 100003//41159--19-00666
E. W. Moore, J. VanderPlas,
Quintero,
[73] M. L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60): 3021, 2021. doi: 10.21 105/joss.03021. URL htt.httss///di.rrgg.g. 10 21105/joss 00021
[74] Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56
61, 2010. doi: 222200000Maoorr--222199222---000
15

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We introduce the generalization in Section 22.] We show that it has the properties Section
claim in Appendix B We perform the experiments in 3 and make a hyperparameter recommendation in Section
(b) Did you describe the limitations of your work? [Yes] We discussed the behavior of ours and other algorithms in the finite sample setting, see Section 2.T and limitations of existing to test the convergence of our method, see Section 2.2
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We briefly mentioned them in Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] This work applies toy simulation data thus no ethics concerns were raised by the paper. Other matters of ethics during experimentation and writing conformed.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of theoretical results? [Yes] These are laid out in Appendix B
(b) Did you include complete proofs of all theoretical results? [Yes] We have only one real theoretical result and that is proven in Lemma Anything else follows algebraically and is shown in detail.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The code will be made available via a link in the supplemental material. It is a python package with example calls and self-explanatory installation instructions. The details will be shown in Section C Results will be released upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Additionally to the hyperparameter search which is the primary experimental result, seen Section 3 we included more details in Appendix C
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] The hyperparameter searches indeed included error bars in Section 3 We did not include error bars in the computation of the diagnostic since we emphasize that it is made possible by our method, not its statistical properties. The details of the benchmark, including some uncertainty in the last experimental section shown in Appendix C
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] This is discussed in Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the benchmark [44 several times in the paper. Also the sbi package (68)
16

---

(b) Did you mention the license of the assets? [Yes] The license is mentioned in Ap- pendix A
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] I will link to my code in the supplemental material discussed above. Upon acceptance results will be made available.
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A] The publicly available data was used in accordance with the aforementioned license in 44.. There was no need to get consent or mention it in the paper.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] The data is generated by toy mathematical models which abstractly relate parameters to data. highly unlikely any culture would find them offensive.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "fk-Qoh_W8M9", "reviewer": "Reviewer_1x3V", "summary": "Density-ratio estimation can be performed via classification tasks (binary, multisample) and learn a potentially unnormalized model for the posterior distribution. \n\nThis submission highlights a simple yet important flaw: when the posterior is parameterized as a softmax (this is the case for NRE-B a.k.a. InfoNCE), the scaling indeterminacy translates to a bias in the density-ratio estimate which trickles down to the rest of the analysis. The submission proposes a new loss that removes this bias. It carefully examines how that impacts posterior estimation in practice, through a series of controlled experiments. \n", "questions": "It is known from [1] that the NRE-B loss recovers the density ratio + a bias term. However, in [1] the bias is learnt as part of the estimation procedure. This way, the *un-normalized* density ratio (and by extension, its numerator the conditional density model, in your case the posterior) can be recovered without bias. \n\nCould you clarify why learning the bias does not solve the problem? Is it because you would only recover the *unnormalized* posterior whereas you need the *normalized* posterior in the diagnostics?", "limitations": "I see no particular negative societal impact. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis work is interesting, as it highlights a simple yet important flaw in a popular loss (InfoNCE) and shows how a simple fix can lead enable better posterior learning.\n\nThe connections with NRE-A (essentially NCE, depending on how the discriminator is parameterized) and NRE-B (InfoNCE) are appreciated. Given that the authors' contribution, NRE-C, can be seen as a generalization of the previous losses, it is interesting to see in Figure 1 what the best setup is: while it has been noted in previous work that (i) using many negative samples are advantageous, the insight that (ii) imbalanced classification is favorable when the tuples of entirely noise samples are over-represented, seems novel. \n\nThe diagnostics section is appreciated as it is an important and rather open question when estimating conditional density models, such as the posterior distribution. \n\n**Weaknesses**\n\nAs I understand it, the simulations evaluate the quality of estimation by averaging across datasets (Figure 4). Is it common to report the variance *across* datasets? Could you report the variance *per* dataset across many (data-generation) *seeds*?\n\nThe effect of the hyperparameters on the convergence speed (Figure 5) is interesting, however it is not clear to me from the plot \"that increasing K leads to earlier convergence at fixed gamma\". Could the authors explain further?\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "MCrx4moNLnV", "reviewer": "Reviewer_fMcW", "summary": "Authors present a generalising framework of (what they call) NRE-A and NRE-B. They argue that is has clear advantages over especially NRE-B. In the experiment section they show some good values of hyperparameters K and gamma, found on a benchmark dataset.", "questions": "I really enjoyed your introduction and motivation; it helped me a lot. However, I could understand that equation 2 is not influenced by k? Can you elaborate here?\n\nIn general, I think the motivation to have multiple k could be more elaborate. You mention it improves simulation-efficiency; is there more to it?\n\nLine 89: \"In contrast with NRE-A, one of these parameters θ_k is always drawn jointly with x.\". As far as I could tell one of the two classes in NRE-A is exactly drawn jointly with x? So why is it in contrast?\n\nIn Equation 14: is Z_w computable? Or should it be approximated, and how good can such an approximation be? And, in data-limit (only) how quick is converge to 1? Vague answer is accepted here.\n\n\n", "limitations": "The authors include a section on societal limitations. I agree completely with the authors here.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the major strength is the generalising framework the authors present. This type of work usually helps fields move forward. \nThe weakness, in my eyes, is the introduction of hyperparameters, especially gamma. I am unsure how easy it is to find good values of this if data is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HcHqPEXBJV", "reviewer": "Reviewer_gFMN", "summary": "This paper introduces contrastive neural ratio estimation (NRE-C) to develop a better estimator for the likelihood-to-evidence ratio p(x | \\theta)/p(x) for simulation-based inference (SBI). They first cast the parameter estimation problem as a classification problem, then augment the K-class classification problem (where the goal is to select the best parameter \\theta_k among K options) in Durkan et al. with another (K+1)th class which contains samples that were generated independently from p(\\theta)p(x). They demonstrate that NRE-C is consistent and outperforms existing baselines on simulated benchmarks.", "questions": "- When constructing this “hybrid” objective function (that blends NRE-A and NRE-B), what exactly are the limitations? For example, is it computationally expensive to generate samples x^{(b)} \\sim p(x)p(\\theta) to sZet up the classification problem? Given that the simulator takes a prior p(\\theta) and a way to sample from the likelihood p(x|\\theta), it seems like sampling from p(x) would be very hard). The paper also mentioned that sampling from the prior is slow. If we take into consideration this samping procedure for constructing the (K+1)th class, is NRE-C still more efficient than NRE-B?", "limitations": "Broader Impact statement is too broad, should be more specific about what “decisions about matters that affect living creatures” are, etc.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- NRE-C is a clean and straight-forward generalization of Durkan et al. and the binary classification “trick” for estimating likelihood ratios.\n- The authors also performed extensive experiments and empirical evaluations on the SBI benchmarks suite.\n\nWeaknesses:\n- It’d be helpful if the paper was a bit more self-contained. There is a lot of information in the supplementary that would benefit from being moved to the main text (e.g. related works, further details on the SBI benchmark). As another example, l. 35 (equivalence of the 2 ratios) should be better explained. Why do we want these two quantities to be equivalent? The text should explicitly mention that it is to try to make posterior inference easier in SBI.\n- The presentation was also unclear. For example, the figures are pretty confusing – I’m not sure what we are supposed to be taking away from this image/caption. Figure 1 was unclear, and Figure 2 was also not clear with the different notations (e.g. b, b’). \n- There is notation (e.g. f_w, x, theta, theta_0, p, etc.) that is not explained. It’d be clearer if this notation was collected in the following section (or introduced properly before its usage) to make it easier to follow. This causes conduction, such as \\sigma \\circ f_w(\\theta, x) on l. 41.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept