Neural Approximate Sufficient Statistics for Implicit Models
Authors: Yanzhi Chen
Dinghuai Zhang
Michael U. Gutmann
Aaron Courville
Zhanxing Zhu
Venue: ICLR 2021 Spotlight (2021)
Keywords: likelihood-free inference
bayesian inference
mutual information
representation learning
summary statistics

Abstract: We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of the likelihood function is intractable but sampling data from the model is possible. The idea is to frame the task of constructing sufficient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. The infomax learning procedure does not need to estimate any density or density ratio. We apply our approach to both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on a range of tasks. 

Published as a conference paper at ICLR 2021
NEURAL APPROXIMATE SUFFICIENT STATISTICS FOR IMPLICIT MODELS
Yanzhi Chen'; Dinghuai Zhang Michael U. Gutmann', Aaron Courville?, Zhanxing Zhu The University of Edinburgh, MIIA,, BBeijing Institute of Big Data Research
ABSTRACT
We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of the likelihood function is intractable but sampling data from the model is possible. The idea is frame the task of constructing sufficient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. The infomax learning procedure does not need to estimate any density or density ratio. We apply our approach both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on range of tasks.
INTRODUCTION
Many data generating processes can be well-described by a parametric statistical model that can be easily simulated forward but does not possess analytical likelihood function. These models are called implicit generative models (Diggle & Gratton, 1984) or simulator-basee models (Lintusaari et al., 2017) and are widely used in science and engineering domains, including physics (Sjöstrand et al., 2008), genetics (Järvenpää et al., 2018), computer graphics (Mansinghka et al., 2013), robotics LLopez-Guevate et al., 2017), finance (Bansal & Yaron, 2004), cosmology (Weyant et al., 2013), ecology (Wood, 2010) and epidemiology (Chinazzi et al., 2020). For example, the number of infected/healthy people in outbreak could be well modelled by stochastic differential equations (SDE) simulated by Euler-Maruyama discretization but the likelihood function of SDE is generally non-analytical. Directly inferring the parameters of these implicit models often very challenging.
The techniques coined as likelihood-free inference open us door for performing Bayesian inference in such circumstances. Likelihood free inference needs to evaluate neither the likelihood function nor its derivatives. Rather, it only requires the ability to sample (i.e. simulate) data from the model. Early approaches in approximate Bayesian computation (ABC) perform likelihood-fii by repeatedly simulating data from the model, and pick a small subset of the simulated data close to the observed data build the posterior (Pritchard et al., 1999; Marjoram et al., 2003; Beaumont et al., 2009; Sisson et al., 2007). Recent advances make use of flexible neural density estimators to approximate either the intractable likelihood (Papamakarios et al., 2019) or directly the posterior (Papamakarios & Murray, 2016; Lueckmann et al., 2017; Greenberg al., 2019).
inference
Despite the algorithmic differences, a shared ingredient likelihood free inference methods the choice of summary statistics. Well-chosen summary statistics have been proven crucial for the performance of likelihood-free inference methods (Blum et al., 2013; Fearnhead & Prangle, 2012; Sisson et al., 2018). Unfortunately, in practice is often diffieult to determine low-dimensional and informative summary statistic without domain knowledge from experts. In this work, we propose a novel deep neural network-basse approach for automatic construction of summary statistics. Neural networks have been previously applied to learning summary statistics for likelihood-free inference (Jiang et al.. 2017: Dinev & Gutmann, 2018; Alsing et al., 2018; Brehmer al., 2020). Our approach is unique in that our learned statistics directly target global sufficiency. The main idea is to exploit the link between statistical sufficiency and information theory, and to formulate the task of learning sufficient statistic as the task of learning information-maxiiiiing representations of data. We achieve this with distribution-fit mutual information estimators or their proxies (Székely et al., 2014; Hjelm
Equul contribution. Correspondence Yanzhi Chen (rhythm, cyzégma i com) Dinghuai Zhang (di nghuai. zhang@mi @mi l quebeec). Codes available http:://gihbb.cmmmcyyyssammmmmmp

---

Published as a conference paper at ICLR 2021
et al., 2018). Importantly, our statistics can be learned jointly with the posterior, resulting in fast learning where the two can refine each other iteratively. To sum up, our main contributions are:
We propose a new neural approach to automatically extract compuct, near-sufficien statisties from raw data. The approach removes the need for careful handcrafted design of summary statistics.
With the proposed statistics, develop two likelihood-free inference methods namely SMC- ABC+ and SNL+. Experiments on tasks with various types of data demonstrate their effectiveness.
BACKGROUND
Likelihood-free inference. LFI considers the task of Bayesian inference when the likelihood function of the model is intractable but simulating (sampling) data from the model is possible:
ा ( x.) x π p(x,
(1)
where x, is the observed data, x (0) is the prior over the model parameters 0, p(x,,0) is the (possibly) non-analytical likelihood function and :: (.))x.) is the posterior over 9. We assume that, while we do not have access to the exact likelihood, we
still sample (simulate) data from the model with a simulator: X ~ p0) The task is then to infer ग xo) given x, and the sampled data: necessarily the prior =(0).
D (0;, x= where 0; p(0), x; p(x(0.). Note that p(0)
Curse of dimensionality. Different likelihood-free inference algorithms might learn ग (00)x, x.) in different ways, nevertheless most existing methods suffer from the curse of dimensionality. For example, traditional ABC methods use small subset of D> closest to X. under some metric to al., 2003; Beaumont et al., 2009; Sisson
build the posterior (Pritchard al., 1999; Marjoram et al., 2007), however in nigh-dimensional space measuring the distance sensibly is notoriously hard (Sorzano al., 2014; Xie et al., 2017). On the other hand, recent advances (Papamakarios et al., 2019; Lueckmann al., 2017: Papamakarios & Murray, 2016; Greenberg et al., 2019) utilize neural density estimators (NDE) to model the intractable likelihood or the posterior. Unfortunately, modeling digh-dimensional distributions with NDE accurately is also known to be very difficult (Rippel & Adams, 2013; Van Oord et al., 2016), especially when the available training data is scarce.
Our interest here is not to design new inference algorithm, but to find low-dimensional statistic s = sxx) that is (Bayesian) sufficient:
(0)x,) = (fls,) x (0)p(s,,0)
where s : X' - S is a deterministic function also learned from D. We conjecture that the learning of s(.) might be an easier task than direct density estimation. The resultant statistic could then be applied to a wide range of likelihood inference algorithms as we will elaborate in Section 3.2.
METHODOLOGY
3.1 NEURAL SUFFICIENT STATISTICS
Our new deep neural network-bass approach for automatic construction of near-sufficient statistics based on the infomax principle, as illustrated by the following proposition (also see Figure 1): Proposition 1. Let ~ p(0), x ~ p(x (),, and s : X - S be a deterministic function. Then sx) is sufficient statistic for pl x() and only if
arg 1(0; S(x),
S-X-1S
where S is deterministic mapping and I(::.) the mutual information between random variables.
Proof. We defer the complete proof to the appendix. This proposition is a variant of Theorem 8 in (Shamir et al., 2010) with adaption to the likelihood-fre inference scenario.
This important result suggests that could find the sufficient statistic s(x) for likelihood function p(x(0) by maximizing the mutual information (MI) I(0;s) = KL[pp(e ())l()))pppp))p())))) be- tween 0 and Moreover, our interest is in maximizing MI rather than knowing its precise value,
2

---

Published as a conference paper at ICLR 2021
updale
updale
~ p(0)
0 ~ p(0)
10,s)
expert knowledge
((0)so)
=(0)sa)
posterior solver
statistic network
posterior solver
p(x(0)
p(x
Figure 1: Left. Traditional likelihood- inference algorithm needs handcrafted design of summary statistic, which requires expert knowledge. Right. Our method automatically mines a low dimensional, near-sufficient statistic of via the infomax principle, which removes the need for careful summary statistic design. Furthermore, this statistics can be re-learned as the posterior inference proceeds.
we can maximize non-KI. surrogate, which may have an advantage in e.g. estimation accuracy or computational efficiency (Székely al., 2014; Hjelm et al., 2018; Ozair al., 2019). To this end, utilize the following two non-KL estimators:
Jensen-Shannon divergence (JSD) (Hjelm al., 2018): this non-KL estimator is shown to be more robust than KL-based ones. More specifically, is defined as:
sup fiSD T-OXS-E
Eyp0.s) |- sp( (00 s))] Ep())o(s) (sp(T((,,s) s))],
where sp(t) = log(1 (1 + exp(t)) the softplus function. With this estimator, we set up the following objective for learning the sufficient statistics, which simultaneously estimates and maximizes the MI:
max S,T
C(S,T) (e xx |- sp (-T(0, S(x))) (xx(1p(x) [sp (T(0,S(xx))
(4)
where the two deterministic mappings S and T are parameterized by two neural networks. Note that we have used the law of the unconscious statistician (LOTUS) from equation 3 to equation 4. The mini-batch version of this objective is given in the appendix.
Distance correlation (DC) (Székely et al., 2014): unlike the JSD estimator, this estimator does not need to learn an additional network T, and can be learned much faster. It is defined as:
(0 (e, (1000''(((ss))))
iDC (0:s) v 1
(5)
(1
where h(a, b) ||a bll Ep ||| a b'll F P [|lal bll] (bb) [l]a' b'll]- Similar the case of the JSD estimator, we set up the following objective for learning the sufficient statistics:
x (4 S(')
max C(S) = / E 1- (T V (xxx)x)) (Sxx,,
6
where the deterministic mapping parameterized by neural network. Again LOTUS is used from equation 5 to equation 6. The mini-batch version of this objective is given in the appendix.
comparison between the accuracy and efficiency of these two MI estimators (as well as other estimators (Belghazi et al., 2018; Ozair et al., 2019)) for infomax statistics learning is in the appendix.
With enough training samples and powerful neural networks, can obtain near-sufficient statisties with either 8 = arg max maxT L(S, 7) or arg max L(S), depending on the estimator. The statistic of data x is then given by
s= sx).
In the above construction, we have not specified the form of the networks S and T. For S, any prior knowledge about the data x could in principle be incorporated into its design. For example, for sequential data can realize S as transformer (Vaswani et al., 2017), and for exchangeable data we can realize as a exchangeable neural network (Chan et al., 2018). Here we simply adopt a fully-connected architecture for S, and leave the problem-specifi design of S as future work. For T. we choose it to be a split architecture T(0, S(x) = T" T' (0) ( S(x) where (-..).. H.() are both MLPs. Therefore separately learn low-dimensional representations for and 0 before processing
3

---

Published as a conference paper at ICLR 2021
them together. This could be seen that incorporate the inductive bias into the design of the networks that x and f should not interact with each other directly, based on their true relationship (for example, consider the likelihood function of exponential family: L(0:x) (0) S(x))).
We left with the problem of how to select d, the dimensionality of the sufficient statistics. The Horman-Koopman Darmois theorem (Koopman, 1936) tells us that sufficient statistics with fixed dimensionality only exists for exponential family, there is no universal way to select d. Here, we propose to use the following simple heuristics to determine d:
d = 2K
where K is the dimensionality of 0 (which typically satisfies K < D). The rationale behind this heuristics is that the dimensionality of the sufficient statistics in the exponential family is K, and exponential family has been proven reasonably accurate for posterior approximation (see e.g. Thomas et al., 2021; Pacchiardi & Dutta, 2020). By doubling the dimensionality of the statistics to 2K we are likely to have better representative power than the exponential family while still keeping d small.
Furthermore, we have the following proposition comparing our method to the existing posterior- mean-as-statistic approaches (Fearnhead & Prangle, 2012; Jiang al., 2017).
Proposition Let ~ p(0) and x ~ p(x(0). Let s(.) be a deterministic, duntcmiinissiiiiniiio that satisfies
arg
arg min [p(0x)11|S(x) 01 SXX+8
then s = s(x) is generally not a maximizer of T(S());0) and hence it is not a sufficient statistic.
Proof. We defer the proof the appendix.
This proposition tells that unlike method, the existing (posterior- )mean- sesaanas--ttine as-statistic approaches widely used in likelihood-free inference community lose information about the posterior, and is only optimal for predicting the posterior mean (Fearnhead & Prangle, 2012; Jiang et al., 2017). When using this statistics in inference, it may yield inaccurate estimates of e.g. the posterior uncertainty. Nonetheless which statistics to use depends on the task, c.g. full posterior vs. point estimation.
3.2 DYNAMIC CAATISTICS-POSTERIII LEARNING
The above neural sufficient statistic could, in principle, be learned via a pilot run before the inference starts, as, for example, done in the work by Drovandi et al. (2011); Fearnhead & Prangle (2012); Jiang et al. (2017). Such strategy requires extra simulation cost, and the learned statistic is kept fixed during inference. We propose dynamic learning strategy below to overcome these limitations.
Our idea is to jointly learn the statistic and the posterior in multiple rounds. More concretely, at round we use the current statistic s(-) to build the j-th estimate to the posterior: q, s.)) x T (0)x.). and at round j+1, this estimate is used as the new proposal distribution to simulate data: p (0) < (0)0s.)) (0)s.), O Ps+1 (0), X; p(x(0+). We then re-learn s(.) and q(-) with all the data up to the new round. In this process, s(.) and g(-) refine each other: a good s(-) helps to learn q(-) more accurately, whereas an improved q(-) better proposal in turn helps to learn s(-) more efficiently.
The theoretical basis of this multi-rounds strategy is provided by Proposition 1, which tells us that the sufficiency of the learned statistics is insensitive to the choice of p(0). the marginal distribution of in sampled data D (x,,0,)'' 2 This means that we are indeed safe to use any proposal distribution (i) at any round in multi-rounds learning, and in such case p(0) after round will be mixture distribution formed by the proposal distributions of the previous rounds, i.e. (e.p(() 1/ E (0)
4

---

Published as a conference paper at ICLR 2021
Algorithm 1 SMC-ABC+
Algorithm 2 SNL+
Input: prior (0), observed data Output: estimated posterior x0
Input: prior (0), observed data x,
Output: estimated posterior 7 (0
Initialization: D = Ø, (0) (0)
Initialization: Ø, pr (0) = (0)
for j in 1 to r do
for j in 1 to do
repeat
repeat
sample 0 ~ Pj (0)
sample 0: Pj (0())
simulate x; px(()
simulate x; ~ p(x(0,):
until n samples
until n samples
D < DU ( x
D < [0,, x
fit statistic net s(-) with D by equation 4;
fit statistic net s(.) with D by equation 4;
sort D according to [x, s(x,)ll:
fit q with converted D by equation 11; convert D with the learned s(-);
fit p(0 (s)) with the top m Os in D;
q (f) s.) p(0 (0)) Σ (0));
Li ( S.) x T()) .g(ss();;
Pj+1 (0) + (s)s)) end for
Pj+1 (0) < q, (0)ss))
end for
return (00 gr (01so)
return At (0 xx)x qr (0
In practice, any likelihood- free inference algorithm that learns the posterior sequentially naturally fits well within the above joint statistic- pottttiir learning strategy. Here we study two such instances:
Sequential Monte Carlo ABC (SMC-ABC) (Beaumont et al., 2009). This classical algorithm learns the posterior in a non- non-parametric way within multiple rounds. Here, consider variant of it to better make use of the above neural sufficient statistic, and to re-use all previous simulated data. The new SMC-ABC algorithm estimates the posterior qj (0]s.) at the j-th round follows. We first sort data in D {x,, 0: H according to the distances []=(x;) s(x,)]] [].
We then pick the top-n Os whose corresponding distances are the smallest, The picked Os then follow f ~ p(0 S,) as below:
p 8o) O
Σ pr()) Pr([Is s.ll <())),
where the threshold is implicitly defined by the ratio nj (which automatically goes zero as j + oo). We then fit p(0)s,) with the collected Os by flexible parametric model (e.g. a Gaussian copula), with which we can obtain the j-th estimate to the posterior by importance (re-)weigh
q O s.)) v(0 S..) (0)/ (0).
(10)
1=1
The whole procedure of this new inference algorithm, SMC-ABC+, is summarized in Algorithm 1.
Sequential Neural Likelihood (SNL. (Papamakarios et al., 2019). This recent algorithm learns the posterior in a parametric way, also in multiple rounds. The original SNL method approximates the likelihood function p(x)0) by a conditional neural density estimator qx)(0), which could be difficult to learn if the dimensionality of x is high. Here, we alleviate such difficulty with our neural statistic. The new SNL algorithm estimates the posterior 9, (0)
(s.) at the j-th round follows. At round j. where we have ni simulated data D = (0,, x, 1 we fit a neural density estimator (ss(0) as:
g(s 0) arg max Q
Σ log Q(s(x:) 0.).
(11)
where s(-) is the current statistic network and Q is neural density estimator (e.g. Durkan et al. (2019); Papamakarios et al. (2017)). With ný being moderately large and Q flexible enough, this would yield us g(s|0) x p(s(0). We then obtain the j-th estimate of the posterior by Bayes rule:
q 0 s²) x 0) qs"
(12)
The whole procedure of this new SNI algorithm, denoted SNL+ is summarized in Algorithm 2.
5

---

Published as a conference paper at ICLR 2021
4 RELATED WORKS
Approximate Bayesian computation. ABC denotes techniques for likelihood-free inference which work by repeatedly simulating data from the model and picking those similar to the observed data to estimate the posterior (Sisson et al.. 2018). Naive ABC performs simulation with the prior, whereas MCMC-ABC (Marjoram et al., 2003; Meeds et al., 2015) and SMC-ABC (Beaumont et al., 2009; Sisson al., 2007) use informed proposals, and more advanced methods employ experimental design or active learning to accelerate the inference (Gutmann & Corander, 2016; Järvenpää et al., 2019). To measure the similarity the observed data, often wise to use low-dimensional summary statisties rather than the raw data. Here we develop a way to learn compact sufficient statistics for ABC.
Neural density estimator-based inference. Apart from ABC, recent line of research uses conditional neural density estimator to (sequentially) learn the intractable likelihood (e.g SNL Papa- makarios et al. (2019); Lueckmann et al. (2019)) or directly the posterior (e.g SNPE Papamakarios Murray (2016); Lueckmann et al. (2017); Greenberg et al. (2019)). Likelihood-targe -trgeetigg approaches have the advantage that they could readily make use of any proposal distribution in sequential learning, but they rely on low-dimensional, well-chosen summary statistic. Posterior-targeting methods on the contrary need no design of summary statistic, but they require non-trivial efforts to facilitate sequential learning. Our approach (e.g SNL+) can be seen as taking the advantages from both worlds.
Automatic construction summary statistics. A set works have been proposed to automatically construct fow-dimensional summary statistics. Two lines of them are most related to approach. The first line (Fearnhead & Prangle, 2012: Jiang et al., 2017; Chan et al., 2018; Wiqvist et al., 2019; Dinev& & Gutmann, 2018) train neural network to predict the posterior as the summary statistic. These mean-as-statistic approaches, as analyzed previously in Proposition indeed do not guarantee sufficiency. Rather than taking the predicted mean, the works (Alsing et al., 2018; Brehmer et al., 2020) take the score function Ve log p(x @= around some fiducial parameter 0" as the summary statistic. However, these core-as-statisste methods are only locally sufficient around 0 Our approach differs from all these methods as it is globally sufficient for all O.
and use this prediction
MI and ratio estimation. has been shown in the literature that many variational MI estimators [(X;Y Y) also estimate the ratio p(X, ()/p(X))pp up to a constant (Nowozin et al., 2016; Nguyen et al., 2010). Therefore our MI-based statistic learning method is closely related to ratio estimating approaches to posterior inference (Hermans et al., 2020; Thomas et al., 2021). The differences are 1) we decouple the task of statistics learning from the task of density estimation for LFI, which grants us the privilege to use any infomax representation learning methods that are ratio-free (Székely et al., 2014; Ozair et al., 2019); 1); and 2) even we do estimate the ratio, we do this in the low- -iw-dimensional space based on sufficient statistics perspective, which is typically easier than in the original space.
5 EXPERIMENTS
5.1 SETUP
Baselines. We apply the proposed statistics to two aforementioned likelihood-free inference methods: (i) SMC-ABC (Beaumont al., 2009) and (ii) SNL (Papamakarios et al., 2019). We compare the performance of the algorithms augmented with our neural statistics (dubbed as SMC-ABC+ and SNL+) to their original versions as well as the versions based on expert-designed statisties (details presented later; we call the corresponding methods SMC-ABC" and SNL'). We also compare to the sequential neural posterior estimate (SNPE) method which needs no statistic design, as well as the sequential ratio estimate (SRE) method (Hermans et al., 2020) which is closely related to MI-based method?. All methods are run for 10 rounds with 1,000 simulations each. The results presented below are for the JSD estimator; the DC estimator achieves similar accuracy (see appendix).
Evaluation metric. To assess the quality of the estimated posterior, we compare the Jensen-Shannon divergence (JSD) between the approximate posterior Q and the true posterior P for each method
Moore specifically, the version B. We compare with SNPE-B (Lueckmann al., 2017) rather than the more recent SNPF:-C (Greenherg et al., 2019) due to its similarity to SRE shown in (Durkan et al., 2020). For fair comparison, we control that the neural network in SRE has similar number of parameters/same optimizer settings in method. See the appendix for more details about the settings of the neural networks.

---

Published as a conference paper at ICLR 2021
03
-0.5
+1.0
50
(a)
(b)
(c)
Figure 2: Ising model. (a) The 64D observed data x,, E (-1,1)64, (b) The JSD between the true and
the learned posteriors. (c) The relationship between the learned statistics and the sufficient statistic.
SMC
SMC+
SNL
SNL+
SRE
SNPE
0.008 + 0.006
0.046 0.051 0.007 + 0.002 0.015 +0.011 0.083 + 0.029 0.058 + 0.039
Table 1:
Ising model. The ISD between the learned and true posterior with 10,000 simulations. Here SMC' and SNL' utilize the ground-trutt sufficient statistics guided by human prior knowledge.
(see appendix). For the problems we consider, the true posterior P is either analytically available, or can be accurately approximated by a standard rejection ABC algorithm (Pritchard al., 1999) with known cow-dimensional sufficient statistic (c.g sxx Ø. and extensive simulations (e.g 10%).
5.2 RESULTS
We demonstrate the effectiveness of our method on three models: (a) an Ising model; (b) a Gaussian copula model; (c) Ornstein-Uhlenbedd process. The Ising model does have an analytical likelihood but the posterior can be approximated accurately by rejection ABC due to the existence of low-dimensional, discrete sufficient statistic. The last two models have analytical likelihoods and hence analytical posteriors. These models cover the cases of graph data, i.i.d data and sequence data."
Ising model. The first model we consider is mathematical model in statistical physies that describes the states of atomic spins 8 x 8 lattice (see Figure I(a)). Each spin has two states described by discrete random variable x; {-1, 1-1,+11, +1}. and only allowed interact with its neighbour. Given parameters 0 {01,02) the probability density function of the Ising model is:
p(x(0) x exp( (x;; 0)),
H(x; 0) 01 Σ IaJj 02 Σ J4-
(3.))
where (i,j) denotes that spin i and spin j are neighbours. is also called the Hamiltonian of the model. The likelihood function of this model is not analytical due the intractable normalizing constant (0) = Exe Exx(-1,11 mm exp[-/f (x; 0)]. However, sampling from the model by MCMC is possible. Note that the sufficient statistics known for this model: s* (x) (E(b)) Zęzj; Ee ) The true posterior
easily be approximated by rejection ABC with the low-dimensional sufficient statistics and extensive simulations. Here, we assume that Og is known, and the task is to infer the posterior of f under uniform prior 01 (O. 1.5) (in this case the sufficient statistic becomes (1.3) J. The true parameters are 0 {0.3, 0.1}.
only ID: 8 (x) E(.,)
In Figure 1(c), we investigate whether the proposed statistic could achieve sufficiency. Ideally, the learned statistic in method does recover the true sufficient statistic s* well, the relationship between s and s* should be nearly monotonic (note that both and s* are here ID). To verify this,
'e chose to conduct experiments on these models rather than common tasks like M/G/1 and Lotka-Von Volterra since they lack known true likelihond, making hard to verify the sufficiency of the proposed statistics. How evaluate [. methods models without known likelihood still open problem (Lueekkann et al., 2021).
7

---

Published as a conference paper at ICLR 2021
0.23
Trubh
SMC
SMC-
0.13
0.5
09
SNI.
SNL+
(a)
(b)
Figure 3: Gaussian copula. (a) The observed data x., in this problem, which comprised of population of 200 i.id samples. (b) The JSD between the true/learned posteriors. (c) The contours.
SMC
SMC+
SNL'
SNL+
SRE
SNPE
0.183 = 0.014 0.047 + 0.009 0.054 + 0.016 0.042 + 0.006 0.052 = 0.032 0.037 + 0.018
Table Gaussian copula. The JSD between the learned and true posterior with 10,000 simulations. Here SMC' and SNL' utilize the hand-crafted summary statistics guided by human prior knowledge.
plot the relationship between s" and We see from the figure that learned by our method does, approximately, increase monotonically with s", suggesting that recovers s* reasonably well. In approach only comparison, the statisties learned with the widely-used indistinguishable for different s", In other weakly depends on the true sufficient statistic; it is nearly words, it loses sufficiency. The result empirically verifies our theoretical result in Proposition 2.
Figure ((b) further shows the JSD between the true and learned posterior for different methods across the rounds (the vertical lines indicates standard derivation, each JSD is obtained by calculating the average of 5 independent runs. The results shown in the below experiments have the same setup). can be seen from the figure that for this model, likelihood-fre free inference methods augmented with the proposed statistic (SMC-ABC+, SNL+) outperform their original counterparts (SMC-ABC, SNL) by a large margin. In Table 1, we further compare our statisties with the expert designed statistics, from which one can see their close performance (here the expert statistics is taken as the true sufficient statistics 5*). We further see that our method also outperforms SRE which directly estimates the ratio t(x, 0) p(x, 0) ((x)p(()) x L(0:x) in high-dimensional space (note that the true likelihood is of the form L(0; x) = exp(Os" (xxp00=(xx))/2> (0)) as well as SNPE (version B). The reason why SNPE(-B) does not perform more satisfactorily might be that it relies importance weights to facilitate sequential learning, which can induce high variance that makes the training unstable.
Gaussian copula. The second model consider is 2D Gaussian copula model (Chen & Gutmann, 2019). Data x for this model can be generated with aid of a latent variable as follows:
1.
0g
~ N
0,
1
Og
T1 = F. (
T2 = F2 F (T(2))
(1-00) (((1;01)= Beta(x1; 0), fz 02) = 02N (1- 4, 1/4)
where (--). Fi (z) F2(x2; 02) are the cumulative distribution function (CDF) of the standard
normal distribution, the CDF of (x1; 01) and the CDF of /2(x2;02) respectively. We assume that total number 200 samples are i.i.d drawn from this model, yielding a population X {x, 200 that serves as observed data. Note that the likelihood of this model can be computed analytically by the law of variable transformation. To perform inference, we compute a rudimentary statistic to describe X, namely (a) the 20-equally spaced quantiles of the marginal distributions of X and (b) the correlation between the latent variables 21, 22 in X, resulting in a statistic of dimensionality 41. A uniform prior is used: 8, (0.55, 12.5), 02 4(00 1), 0g U(0.4,0.8 and 0* (6, 0.5. 0.6).
In Figure 2(b), we demonstrate the power of our neural sufficient statistic learning method on the Gaussian copula problem. Overall, we see that the proposed method improves the accuracy of
8

---

Published as a conference paper at ICLR 2021
0.37
0.23
6.23
Truth
SMC
SMC-
6.5
2.2
SNL
SNL+
(a)
(b)
(c)
Figure 4: OU process. (a) The observed time-series data x, = f 150 (b) The JSD between the true and the learned posteriors. (c) The contours of the true posterior and the learned posteriors.
SMC
SMC+
SNL'
SNL+
SRE
SNPE
0.040 + 0.006 0.044 + 0.018 0.004 + 0.001 0.009 + 0.002 0.022 + 0.013 0.019 + 0.009
Table 3: OU process. The JSD between the learned and the true posterior with 10,000 simulations. Here SMC' and SNL' utilize the hand-crafted summary statistics guided by human prior knowledge.
existing likelihood-free inference methods, as well as their robustness, see e.g. the reduced variability for SNL+ (the high variability in SNL may be due to the lack of training data required to learn the 41-dimensional likelihood function well). This is also confirmed by the contours plots in Figure 2(c). In Table we further compare the proposed statistic with the expert-designed low-dimension statistic (here the expert statistic is taken to be the 5-equally spaced marginal quantiles and the correlations between zı; 22), from which see that our proposed statistic achieves better performance. For this model, the average performance of our method is slightly worse than that of SNPE. However, SNPE has a higher variability, so that the difference in performance is actually not significant.
Ornstein-Uhlenbeck process. The last model we consider is stochastic differential equation (SDE). Data {x)) in this model sequentially generated as:
Ft+1 = Zt + A.
Az, = 0, (exp((,) x, 2, 0.5c, C~N(c;0,At).
where D = 50, At = 0.2 and To 10. This SDE can be simulated by the Suler-Maruyamy method, and has analytical likelihood. It has wide application in financial mathematics and the physical sciences. Here, the parameters of interest are 0 = [0, 0g J. and uniform prior is placed on them: 0, ((, 0) ~ U( (-.0, 2.0). The true parameters are set be f (0.5, 1.0).
Figure 3(b) compares the JSD of each method against the simulation cost. Again, we find that the proposed neural sufficient statistics greatly improve the performance of both SMC-ABC and SNL. In Table 3, we compare our statistics to expert statistics (here the expert statistics are taken as the mean, standard error and autocorrelation with lag 2, time series). can be seen that our statistics perform comparably to the expert statisties. Our method also seems outperform SRE and SNPE.
CONCLUSION
We proposed a new deep learning-base based approach for automatically constructing low-dimensional sufficient statistics for likelihood-free inference. The obtained neural approximate sufficient statistics can be applied to both traditional ABC-based and recent NDE-based methods. Our main hypothesis is that learning such sufficient statistics via the infomax principle might be easier than estimating the density itself. We verify this hypothesis by experiments on various tasks with graphs, i.id and sequence data. Our method establishes a link between representation learning and likelihood- inference communities. For future works, we can consider further infomax representation learning approaches, well as principle ways to determine the dimensionality of the sufficient statistics.

---

Published as a conference paper at ICLR 2021
REFERENCES
Justin Alsing, Benjamin Wandelt, and Stephen Feeney. Massive optimal data compression and density estimation for scalable, likelihood frre inference in cosmology. Monthly Notices of the Royal Astronomical Society, 477(3))2874-2888) 2018.
Ravi Bansal and Amir Yaron. Risks for the long run: A potential resolution of asset pricing puzzles. The journal of Finance, 5984):148- 2004.
Mark A Beaumont, Jean-Marie Cornuet, Jean-Michel Marin, and Christian P Robert. Adaptive approximate Bayesian computation. Biometrika, 96(4):833-90, 2009.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540, PMLR, 2018.
Michael GB Blum, Maria Antonieta Nunes, Dennis Prangle, Scott A Sisson, et al. A comparative review of dimension reduction methods in approximate bayesian computation. Statistical Science, (8(2):189-208 2013.
Johann Brehmer, Gilles Louppe, Juan Pavez, and Kyle Cranmer. Mining gold from implicit models to improve likelihood fereeeeoodfeeeeeeeeeeeee inference. Proceedings of the National Academy of Sciences, 117(10): 9242-5249, 2020.
Jeffrey Chan, Valerio Perrone, Jeffrey Spence, Paul Jenkins, Sara Mathieson, and Yun Song. A likelihood-fres inference framework for population genetic data using exchangeable neural net- works. In Advances in Neural Information Processing Systems, pp. 8594-8605, 2018.
Yanzhi Chen and Michael U Gutmann. Adaptive gaussian copula abc. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1584-1592. PMLR, 2019.
Matteo Chinazzi, Jessica Davis, Marco Ajelli, Corrado Gioannini, Maria Litvinova, Stefano Merler, Ana Pastore y Piontti, Kunpeng Mu, Luca Rossi, Kaiyuan Sun, et al. The effect of travel restrictions on the spread of the 2019 novel coronavirus (covid-19) outbreak. Science, 368(6489):395-400, 2020.
Tm Cover, Ja Thomas, and J Wiley. Elements of information theory. Tsinghua University Pres, 2003.
Peter J Diggle and Richard J Gratton. Monte Carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Society, Series B. pp. 193-227, 1984.
Traiko Dinev and Michael Gutmann. Dynamic likelihood-fres inference via ratio estimation (dire).
arXiv preprint arXiv: arXiv:1810.0900 1810.0989, 2018
Christopher C Drovandi, Anthony N Pettitt, and Malcolm Faddy. Approximate Bayesian computa- tion using indirect inference. Journal of the Royal Statistical Society: Series C (Applied Statistics), 60(3):317-337, 2011.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. arXiv preprint arXiv: 04032, 2019.
Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood inference. arXiv preprint arXiv:2002.03712, 2020.
Paul Fearnhead and Dennis Prangle. Constructing summary statistics for approximate Bayesian computation: semi-automatic approximate Bayesian computation. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 74(3):419 2012.
David Greenberg, Marcel Nonnenmacher, and Jakob Macke. Automatic posterior transformation for likelihood-free inference. International Conference on Machine Learning, pp. 2404-2414, 2019.
Michael Gutmann and Jukka Corander. Bayesian optimization for lkelihood-free inference of Limulator-base statistical models. Journal of Machine Learning Research, 17(1):4256-433 4302, 2016.
10

---

Published as a conference paper at ICLR 2021
Joeri Hermans, Volodimir Begy, and Gilles Louppe. Likelihood-free meme with amortized approxi- mate ratio estimators. In International Conference on Machine Learning, pp. 4239-42492222 4248. PMLR, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Moroie-Mrrchiloon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2018.
M. Järvenpää, M.U. Gutmann, A. Vehtari, and Marttinen. Gaussian process modeling in approx- imate Bayesian computation to estimate horizontal gene transfer in bacteria. Annals of Applied Statistics, 2018.
M. Järvenpää, M.U. Gutmann, A. Vehtari, and Marttinen, Efficient acquisition rules for model- based approximate Bayesian computation. Bayesian Analysis, 14(2):595-622 2019. doi: doi: BA1121.
10.1214/1-BAI111.
Bai Jiang, Tung-yu Wu, Charles Zheng, and Wing H Wong. Learning summary statistic for ap- proximate bayesian computation via deep neural network. Statistica Sinica, pp. 1595-1618, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1:Xi::1412.6980 2014.
Bernard Osgood Koopman. On distributions admitting sufficient statistic. Transactions of the American Mathematical society, 39(3):399-409, 1936.
J. Lintusaari, M.U. Gutmann, R. Dutta, S. Kaski, and J. Corander. Fundamentals and recent developments in approximate Bayesian computation. Systematic Biology, 66(1)x666-822, January 2017.
T. Lopez-Guevara, N.K. Taylor, M.U. Gutmann, S. Ramamoorthy, and K. Subr. Adaptable pouring: Teaching robots not to spill using fast but approximate fluid simulation. Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference on Robot Learning (CoRL), volume 78 of Proceedings of Machine Learning Research, pp. 77-86, November 2017.
Jan-Matthis Lueckmann, Pedro J Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnenmacher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. In Advances in Neural Information Processing Systems, pp. 1289-1299, 2017.
Jan-Mathhis Lueckmann, Giacomo Bassetto, Thcofanis Karaletsos, and Jakob H Macke. Likelihood- free inference with emulator networks. In Symposium Advances in Approximate Bayesian Inference, pp. 32-53, 2019.
Jan-Matths Lueckmann, Jan Boelts, David S Greenberg, Pedro J Gonçalves, and Jakob H Macke. Benchmarking simulation-based inference. arXiv preprint arXiv:2101.04653, 2021.
Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. Approximate bayesian image interpretation using generative probabilistic graphics programs. In Advances in Neural Information Processing Systems, pp. 1520-1525 1528, 2013.
Paul Marjoram, John Molitor, Vincent Plagnol, and Simon Tavaré. Markov chain Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 100126);:5324-11522 2003.
Edward Meeds, Robert Leenders, and Max Welling. Hamiltonian abc. Proceedings of the Thirty-First Conference Uncertainty in Artificial Intelligence, pp. 582-591, 2015.
XuanI.on Long Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-586. 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
11

---

Published as a conference paper at ICLR 2021
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pp. 15604-15614, 2019.
Lorenzo Pacchiardi and Ritabrata Dutta. Score matched conditional exponential families for likelihood-free inference. arXiv preprint arXiv:2012.10003 2020.
George Papamakarios and lain Murray. Fast e-free inference of simulation models with bayesian conditional density estimation. In Advances in Neural Information Processing Systems, pp. 1028- 1036, 2016.
George Papamakarios, Theo Pavlakou, and Lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
George Papamakarios, David Sterratt, and lain Murray. Sequential neural likelihood: Fast likelihood- free inference with autoregressive flows. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 837-848. PMIR, 2019.
Jonathan K Pritchard, Mark Seielstad, Anna Perez-Lezauee and Mareus W Feldman. Population growth of human chromosomes: study of y chromosome microsatellites. Molecular biology 1999.
and evolution,
Oren Rippel and Ryan Prescott Adams, High-ddimensionl probability estimation with deep density models. arXiv preprint arXiv: 130.55130..5 2013.
Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theoretical Computer Science, (1112293)):266-22777 2010.
S.A. Sisson, Y Fan, and M.A. Beaumont. Handbook of Approximate Bayesian Computation, chapter Overview of Approximate Bayesian Computation. Chapman and Hall/CRC Press, 2018.
Scott A Sisson, Yanan Fan, and Mark M Tanaka. Sequential Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 17666):160-1165, 2007.
Torbjörn Sjöstrand, Stephen Mrenna, and Peter Skands. A brief introduction to pythia 8.1. Computer Physies Communicationns 178(1 ()221)2222822 867, 2008.
Carlos Oscar Sánchez. Sorzano, Javier Vargas, and A Pascual Montano. survey of dimensionality reduction techniques. arXiv preprint arXiv: 1403228777 2014.
Gábor J Székely, Maria L Rizzo, et al. Partial distance correlation with methods for dissimilarities. Annals of Statistics, 42(6):2382-2444 2014.
Owen Thomas, Ritabrata Dutta, Jukka Corander, Samuel Kaski, Michael U Gutmann, et al. Likelihood inference by ratio estimation. Bayesian Analysis, 2021.
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp- 1747-1756. PMLR, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv: TexXiv/77000 /00/ 03762, 2017.
Anja Weyant, Chad Schafer, and W Michael Wood- Wood-Vasey. Likelihood-fred cosmological inference with type ia supernovae: approximate Bayesian computation for a complete treatment of uncertainty. The Astrophysical Journal, 764(2), 2013.
Samuel Wiqvist, Fierre-Alexande Mattei, Umberto Picchini, and Jes Frellsen. Partially exchangeable networks and architectures for learning summary statistics in approximate bayesian computation. In International Conference on Machine Learning, pp. 6798-68800 2019.
Simon N Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466 (7310): 102, 2010.
Haozhe Xie, Jie Li, and Hanqing Xue. A survey of dimensionality reduction techniques based on random projection. arXiv preprint arXiv: (006.0/770603337... 2017.
12

---

Published as a conference paper at ICLR 2021
A THEORETICAL PROOFS
A.1 PROOF OF PROPOSITION
Proof. Firstly, assume s()) ) is sufficient statistic. By the definition of sufficient statistic know Then we have the Markov chain 0 - S - x for the data generating process. On the other hand, since x ~ p(x and 5 is deterministic function we have the Markov chain 0- + x + S. By data processing inequality we have sx)) < (00 x) for the first chain
p(x = p(x|s)p(s|@)
and (0; x) < I(0: s(x)) for the second chain. This implies that [ x) 1(0: sx)) i.e is the
maximizer of [0; (x)). For the other direction, since (()) = maxş [(0; S()), we have
[(:: s(x)) = [0;; x). Note that
- is Markov chain, from Theorem 2.8.1 of Cover et al. (2003) we can get 0 and X is conditionally independent given s. This implies s is sufficient.
M
A.2 PROOF OF PROPOSITION 2
Proof. We can write the objective as x - - p(0, log (x) -1) On the other hand we have r(0: S(x) = f ((O x) (()())))))((((()))) see they are generally not equivalent. Equivalence only holds in special cases (e.g. Gaussians). I1
By comparing them, we
B MORE EXPERIMENTAL DETAILS AND RESULTS
B.1 DETAILED EXPERIMENTAL SETTINGS
Neural networks settings. For the statistic network S in our method (for both JSD and DC estima- tors), we adopt a D-100- 1000d fully-comnected architecture with D being the dimensionality of input data and d the dimensionality of the statistic. For the network H used to extract the representation of 9, we adopt a K-100-106 100- K fully-connected architecture with K being the dimensionality of the model parameters 0. For the critic network, we adopt (d + K)- fully connected architecture. ReLU is adopted as the non-linearity in all networks. For SRE, which is closely related to our method, use a (D + - 110--44-144-10000 144-100-1 architecture. This architecture has similar complexity as our networks. All these neural networks are trained with Adam (Kingma & Ba, 2014) with a learning rate of 10 and a batch size of 200. No weight decay applied. We take 20% of the data for validation, and stop training if the validation error does not improve after 100 epochs. We take the snapshot with the best validation error as the final result.
For the neural density estimator in SNL/SNPE, which is realized by Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017), we adopt 5 autoregressive layers, each of which has two hidden layers with 50 tanh units. This is the same settings in SNL. The MAF is trained with Adam with learning rate 5 x 10-4 and a batch size of 500 and a slight weight decay (1 x 10-1). Similar to the case of MI networks, we take 20% of the data for validation. and stop training if the validation error does not improve after 100 epochs. The snapshot with the best validation error is taken as the result.
Sampling from the approximate posterior/learnt proposal. For fair comparison, adopt simple rejection sampling for all LFI methods (ABC, SNL, SNPE, SRE) when sampling from the learnt posterior, that each LFI method only differs in the way they learn the posterior. No MCMC is used.
Empirical version of objective functions. Recall that in the JSD estimator, the statistic network 8(.) is trained with the following objective together with the critic network T(-): maximizes,r C(S,T) = -Ep x) |- sp (e, S(x))) Ex())()) (T the mini-batch approximation to this objective is:
C(S,T) TI Σ -- sp S(x)))
Isp (T S(x)))
m v
where (f1, f2.,, .... Jn} the j-th random permutation of the indexes (1, 2..,, n and the pair x0,, are randomly picked from the data D {0,, x. J.N. Here we set = 400 and n is the batch size.
In the DC estimator, the statistic network is trained by the following objective:
Ep(0.x)p(0r... x') (h(p)/hh))))())))(__)__))))))")) (0)
maximizes L(S) /Epr0miii (12(0.0))
13

---

Published as a conference paper at ICLR 2021
where h(a,b) ||a l
( b'lll Bptar)llaa' blll + Bpta Sobr))((b)llll"" b'll]- The
mini-batch approximation to this objective is:
Σi h(0,, (10,0,))((((x))) S(x))
((S)
R. (2, 0,)
v V (((()
nnn where M(,, by) = laa (ai-byll- -ー Σ3 ||a |abbbyll - Σ baul (--1xx-2] 2n' ||ag by Here i, 1 j' are the indexes in the mini-batch. is again the batch size.
JSD calculation between true posterior and approximate posterior. The calculation of the Jensen- Shammon divergence between the true posterior P and approximate posterior Q. namely JSD( P, Q) = (P++)()) is done numerically by Riemann sum over 30 equally spaced grid points with K being the dimensionality of 0. The region of these grid points defined by the min and max values of 500 samples drawn from P. When we only have samples from the true posterior (e.g. the Ising model), we approximate / by mixture of Gaussian with 8 components.
[I][ /2 +
B.2 ADDITIONAL EXPERIMENTAL RESULTS
Comparison of different MI estimators. We compare the performances of four MI estimator for infomax statistics learning: Donsker-Varadhan (DV) estimator (Belghazi et al., 2018), Jensen- Shannon divergence (JSD) estimator (Hjelm et al., 2018), distance correlation (DC) Székely et al. (2014) and Wasserstein distance (WD) (Ozair et al., 2019). We highlight that the last two estimators (DC and WD) are ratio-free. We compare the discrepancy between the true posterior and the posterior inferred with the statistics learned by each estimator, as well as the execution time per each mini-batch. The results, which averaged over independent runs, shown in the figure and the table below.
From the figure we see that the JSD estimator generally yields the best accuracy among the four estimators. In terms of execution time, the DC estimator clearly the winner, with its execution time being only 1/15 of the other estimators. However, the accuracy of the DC estimator is still comparable to the JSD estimator, especially when the number of training samples is large (e.g. 10,000). According to these results, we suggest using JSD in small-scale settings (c.g. early rounds in sequential learning), and use DC in large-scale ones (e.g. later rounds sequential learning).
-0- C2
50
DC
-0- WD
DV
+-
150
-0- 0C -0- WD
150
0.20
0.52
3.07
030
2.06
0.58
246
0338
0.55
0.233
3.00
0.200
3.03
sunds
rounds
(a) Ising model
(b) Gaussian copula
(c) OU process
Figure 5: Comparing the accuracy of different MI estimator for infomax statistics learning.
Ising model
Gaussian copula
OU process
DV
JSD
DC
WD
DV
JSD
DC
WD
DV
JSD
DC
WD
115
124
6
230
154
167
10
288
143
158
13
256
Table 4: Comparing the execution time (ms) of different MI estimator for infomax statistics learning.
14

---

Published as a conference paper at ICLR 2021
Contrastive learning v.s. MLE. In the experiment in the main text, discover that method does not always achieve the best performance; it does not work better than SNPE-B on the Gaussian copula problem. Here would like to investigate why this happens.
Upon a closer look, we discover that SRE, which is closely related to our method when used with the JSD estimator, is outperformed by SNPE-B on the Gaussian copula problem. Remark that both SRE and our method, when used with the JSD estimator, uses contrastive learning rather than MLE. Since both of these SNPE-B, it makes us suspect the reason is due to imperfect contrastive learning. To verify this, we further conduct experiments for SNPE-C, which shares the same loss function with SRE but with different parameterization to the density ratio (SRE: fully-connected network; SNPE-C: NDE-based parameterization. This NDE is the same as SNI.). The result follows:
contrastive learning methods do not perform better than the MLE-based
Ising model
Gaussian cupula
OU process
SRE
SNPE-B
SNPE-C
SNL+
SRE
SNPE-B
SNPE-C
SNL+
SRE
SNPE-B
SNPE-C
SNL+ 0.009
0.083
0.058
0.030
0.017
0.052
0.037
0.047
0.042
0.022
0.018
0.016
Table 5: Commparing the the JSD of contrastive learning- based methods (SRE, SNPE-C, SNL+) and MI MI.E-based method (SNPE-B) on the three models considered in the experiments in the main text.
Surprisingly, we find that SNPE-C also perform less satisfactorily than SNPE-B on the Gaussian copula problem. This suggests that contrastive learning might be less preferable than MLE on the Gaussian copula problem, which might also explain the less satisfactory performance of our method.
15


Reviews:
[{"review_id": "gXDEsF4ooo", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThe paper presents a method for building sufficient statistic $\\mathbf{s}(\\mathbf{x})$ of observed data $\\mathbf{x}$ for a parameter $\\theta$. It focuses on using this statistic for performing posterior inference in a likelihood-free setting. In particular, the authors proposed to improve SMC-ABC and SNL with their statistics. The results suggest the method is able to learn good statistics and compares favorably with other LFI algorithms.\n# Major comments\n## Pros:\nThe paper is well written and easy to follow. Moreover, the introduction of SMC-ABC and SNL+ is interesting as it improves the performance of the classic algorithms. Although the training procedure in itself is not new I think (see comments below), finding a sufficient statistic is conceptually new and is very relevant.  \n## Cons:\n1) Formally, your method is very close to SRE. Indeed the MI loss used to learn the statistic is equivalent to the binary cross-entropy used by https://arxiv.org/pdf/1903.04057.pdf [1] to learn the discriminator. In the end, the network T can be seen as a discriminator (without the sigmoid activation in the computation of the loss) between (x, \\theta) pairs that only takes as input s(x), a compressed version of x. Thus the novelty is not really about how you learn the sufficient statistic but more the way you use it. In [1], they use the ratio whereas you directly plug the statistic into SMC-ABC and SNL.\nThus when discussing MI and RE just before the experiments section, I do not agree with your arguments 1) and 2) are just an architectural choice that can be made. \n2) I wonder whether discussing Bayesian Sufficient statistics couldn't be interesting?\n3) You don't write about minimal sufficient statistics whereas this is exactly what you are looking for as it is the one that will provide the best SNR.\n4) Why don't you compare to more standard tasks such as the one used by SNL or SRE?\n5) I did not get the point about monotonicity between two sufficient 1D statistics, could you explain why this should be correct? Indeed \n\"A statistic T = T (x) is said to be minimally sufficient for the parameter $\\theta$ if it is sufficient for $\\theta$ and for any other sufficient statistic S = S (x) there exists a function g(.) with T(x) = g(S(x)).\" Nothing prevents g to be something else than monotonic. The only case where this is maybe true is if S and T are minimal.\n\n## What could be made to address my comments\n\n1) You should clarify the similarities between SRE and your work, showing they share the same training procedure but that you are more specific about the architectural choice. I agree that your work is original and conceptually distinct from SRE. However, the only formal difference between the two is that you specify how to input the parameters and data into the neural network whereas SRE does not. The two algorithms introduced in this paper are original and seem very competitive and thus the novelty seems to be there more than anywhere else.\n2) This is just a question!\n3) I think the \"minimal sufficient statistic\" should be defined somewhere, also discussing the impact of just finding a sufficient statistic in contrast to a minimal one should be discussed. Probably, ABC and SNL are able to discard the \"useless\" noise from the statistic but I suppose this has somewhere an impact on the sample efficiency.\n4) Compare on SLCP, Lotka-Volterra, etc...\n5) Just clarify this point.\n\n*I will increase my score to 7 or 8 depending on how you address my comments (or convince me they don't make sense).*\n\n## Minor comments\nFigure 1: (b) What about SRE? (c) could you improve the quality of this plot, maybe creating it as a pdf?\npage 6: \"Note that the sufficient statistic\" -> \"a sufficient\"?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ObwLbeSDuVC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of coming up with summary statistics for ABC-type approximations.  The strategy is to find a function of the data S(X) parametrized by neural networks, with the objective of maximizing a mutual information criterion.  The motivation comes from an interesting classical result described by Shamir, Sabato and Tishby, relating sufficiency and mutual information, combined with a machinery inspired by Hjelm et al, \"Learning  deep representations  by  mutual  information estimation and  maximization\" published in ICLR 2019, and Monte Carlo algorithms to learn the ABC posterior and the statistics jointly. The method is illustrated on toy examples.\n\nThe topic is very interesting and the paper contains a real contribution. It is particularly remarkable as a lot of articles have been written on the topic of the summary statistics for ABC, without much progress in the last few years.  The use of information-theoretic consideration is fruitful to define a meaningful concept of \"near sufficiency\". Indeed we know that exact sufficiency is only attainable in exponential models, while ABC methods are intended for much more complex settings. The experiments are only dealing with simple models, not really typical of challenging settings for which ABC methods are useful, but they make a convincing case that the method is promising.\n\nIt might be worth reminding the readers of the Pitman-Koopman-Darmois theorem on sufficient statistics of fixed dimension in the exponential family, and perhaps a few more elements from Shamir et al 2008 could be recalled to motivate the relaxed notion of \"near sufficiency\".  There could also be more comments on the use of JSD instead of KL, as in Appendix A of Hjelm et al, as this seems important.  I would find these reminders more useful and informative than the authors' remarks on the curse of dimensionality, which seem less central to the manuscript. The authors mention a research direction around eq (7) related to the dimension of the statistic but leave it for future work. \n\nProposition 2 itself is not very surprising, given that the \"posterior mean as statistic\" is not designed to minimize the same information-theoretic loss, and given the lack of fixed-dimensional sufficient stats in generic models. On the other hand the posterior mean is optimal for another loss, as shown in Theorem 3 of Fearnhead & Prangle. I find the writing around this proposition to be a bit partial; it would be preferable to use neutral terms and to explain that the different approaches optimize different losses.\n\nThe SMC-ABC looks more like a population Monte Carlo algorithm as in Beaumont et al., or an adaptive importance sampling strategy than an SMC sampler. In SMC the particles are typically propagated across steps, weighted and resampled, whereas here, at each iteration of Algorithm 1, particles on the parameter space are drawn from a parametric distribution $p_j$; they are not propagated from the previous $\\theta$'s.\n\nI was not sure why Figure 1c was looking so wiggly? \n\nThe exact posterior in the Ising model could be obtained from the exchange algorithm or similar techniques for distributions with intractable normalising constants, instead of the suggested rejection sampler. See e.g.\nMurray, Ghahramani & MacKay, MCMC for doubly-intractable distributions.\nMoeller, Pettitt, Berthelsen & Reeves, An efficient Markov chain Monte Carlo method for distributions with intractable normalising constants.\n\nThe discretization of the OU process is also known as an autoregressive process of order 1, although the parametrization is typically different. I wonder if the choice of 'expert' statistics could be commented on further. Are the mean, sd and autocorrelation of order 1 actually sufficient here?\n\nOn the last page, \"For future works, we can consider other infomax approaches.\" sounds a bit shallow.\n\nIn the experiments, instead of selecting a subset of the order statistics, I wonder if the authors have tried to use all the order statistics. The distance between order statistics actually might be a very sensible way of comparing marginal distributions... the \"reduction of dimension\" associated with the selection of some order stats might not actually bring much gains.\n\nFinally, how does the method relate to theoretical works on ABC, such as Frazier et al or Li & Fearnhead? Do the automatically derived summary statistics satisfy the assumptions of these papers? If so, something could perhaps be said about asymptotic concentration of the resulting posteriors?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9QYLSbc-ROE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis study proposes a new method for computing summary statistics in the context of likelihood-free inference. The authors pose the problem as finding the summary statistics function (in the form of a neural network) that maximises the mutual information between parameters and summary statistics. The study is technically sound and will be of interest to the likelihood-free community and inspire further method development. However, the manuscript could be improved in terms of clarity and soundness of the claims.\n\n\n### Quality\nThe paper is technically sound, but some of the claims are not sufficiently backed by empirical evidence:\n\n-in the main section of the paper, tables 1-3, the authors report a significant improvement in performance compared to previous methods (SRE and SNPE) (with the exception of the Gaussian copula case, where SNPE seems to have a slightly better performance). However, in the appendix, the differences in performance (using the MMD metric) between algorithms are much more nuanced and in fact, it is unclear what algorithm is the best. It would be important for the authors to discuss these nuanced results in the main section of the paper, and be careful not to oversell their approach;\n\n-related to the above point, most results presented in the paper are averages across three independent runs for each algorithm. This is somewhat insufficient to assess the real difference in performance between algorithms, and increasing this to at least 10 rounds would be critical.\n\n\n### Clarity\n\nThe manuscript is for the most part clearly written. However, it sometimes lacks enough information for an expert reader to follow all the steps to evaluate and reproduce the results. Some comments to improve clarity:\n\n-when describing SNL in the main section of the manuscript, the authors write that after getting an estimate of the likelihood:\n\"We then obtain the r-th estimate of the posterior by Bayes rule\". However, the posterior is obtained via MCMC and this should be made explicit for clarity. This is not described at all in the manuscript, and I would urge the authors to also include a few more details on the MCMC procedure in the supplement;\n\n-what were the hyperparameters used for SRE and SNPE? Did the authors test different parameterisations of these for the comparison between algorithms?\n\n\n### Originality\n\nThe novelty of the study resides in framing the problem of computing the approximate sufficient statistics (in the context of likelihood-free inference) as a maximisation of mutual information between the parameters and the summary statistics.\n\n\n### Significance of the work\n\nThe results suggest that the developed approach is a promising step towards improving likelihood-free methodology, especially in settings with high-dimensional data. \n\n\n### Minor comments\n\nRegarding the choice of SNPE-B over SNPE-C, the authors write \"We select to compare with SNPE-B (Lueckmann et al., 2017) rather than the more recent SNPE-C (Greenberg et al., 2019) due to its equivalence to SRE shown in (Durkan et al., 2020).\" However, Durkan et al. 2020 only showed similarity between SRE and SNPE-C and not strict equivalence, so the authors should rephrase this.\n\n\n### Some typos:\n\n-instead of \"for automatically construction\", \"for automatic construction\";\n\n-instead of \"problem-specific design of S as future works\", \"problem-specific design of S as future work\";\n\n-instead of \"Likelihood-targeting approaches has the advantage that it could [...], but relies\", \"...have the advantage that they could...but rely...\";\n\n-instead of \"our method does increases\", \"our method does increase\";\n\n-instead of \"methods augmented with the proposed statistic [...] outperforms\", \"methods augmented with the proposed statistic [...] outperform\";\n\n-instead of \"In Figure 2.(b)\", \"In Figure 2 (b)\";\n\n-instead of \"in Figure 2.(c)\", \"in Figure 2 (c)\";\n\n-instead of \"discreterized\", \"discretized\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "5k3jHwkzhxE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThe paper introduces a new method for learning approximately sufficient summary statistics in the context of likelihood-free inference. To do so, neural networks are trained with a loss maximising mutual information, using a JSD surrogate estimator previously proposed in the literature. The authors show how their approach can for example be combined with SMC-ABC or SNL, leading to marked performance improvements on three problems.\n\n### Score\nI appreciate the core idea of the paper as well as the empirical improvements over SMC-ABC and SNL. As detailed below, my main concern is the comparison to existing LFI approaches that are able to automatically reduce data dimensionality (SRE, SNPE). I hope the authors will address the weaknesses and conceptual question during rebuttal. As it stands, I view the paper as marginally below acceptance.\n\n### Strengths\n+ The proposed method of learning summaries is novel and very relevant to many applications of LFI\n+ SMC-ABC+ and SNL+ show marked performance improvements\n+ Comparison of different mutual information estimators\n+ Concise discussion of their approach versus posterior-as-mean statistic used in some previous work\n+ Comparison to hand-crafted summary statistics (SMC-ABC', SNL')\n\n### Weaknesses\nAs mentioned above, I see the main weakness in the comparison to SRE and SNPE, which falls short in several ways: \n- The authors include comparison to SNPE-B rather than SNPE-C arguing that “We select to compare with SNPE-B (Lueckmann et al., 2017) rather than the more recent SNPE-C (Greenberg et al., 2019) due to its equivalence to SRE shown in (Durkan et al., 2020)”. While Durkan et al. 2020 indeed show an equivalence in the loss functions of both approaches, this does not mean that they are the same and comparison is not warranted: While SNPE-C/APT trains a density estimator to estimate the posterior directly, a classifier architecture is used for SRE/AALR, combined with MCMC sampling. While these two approaches share similarity in the loss, they perform differently in practice, due to different training problems/inductive biases/algorithmic steps. Therefore, results using the newer SNPE-C version should be reported (which generally outperforms SNPE-B)\n- No details on how SRE and SNPE-B were used are reported in the paper, i.e., it is unclear which network architectures were used, and whether these were appropriate for the problems at hand. I briefly checked the code in the supplementary material, but could not find hyperparameters for these methods in the respective notebooks either. In any case, this is central information which needs to be reported properly in the papers’ appendix\n- OU process: Judged by MMD instead of JSD, the performance of SRE and SNPE seems comparable to SNL+ on the OU model (Figure 6 Appendix), while this is not the case when looking at JSD results in Table 3. How can the discrepancy be explained? \n\nMore generally, regarding empirical comparisons:\n- In the figures reporting results it is not defined what error bars represent\n- Have the authors checked whether JSD calculation is stable, i.e., yields similar results, when using a finer grid and more than 500 samples?\n- Results on the MMD metric for the Ising model are missing from the appendix\n- It would have been good to include a truly high-d problem, e.g. involving images\n\n### Conceptual question\n\nIn the appendix, the authors show the posterior directly extracted from the critic. How exactly were the plots shown for comparison and the JSDs calculated, and more specifically, could the difference in JSD be explained by calculating the normaliser on coarse grid? As a control, I would like to see a sample-based comparison, i.e., a comparison of samples from the approximate posterior extracted from T (e.g., by MCMC) versus samples generated from the NDE refit. If it turns out that results are comparable, wouldn't it make sense to think of the algorithm as a new LFI algorithm in itself rather than a pre-processing step reducing dimensionality of data for other algorithms?\n\n\n### Additional comments\n\n- As a suggestion: It might make sense to first compare and discuss SMC, SMC+, SMC’ as well as  SNL, SNL+, SNL’, and second, go on to compare SMC+, SNL+ versus SRE, SNPE\n\n\n### Minor comments\n\n- Figure 5 and 6 (Appendix): SNP should be SNPE for consistency with main paper\n- Figure 6 (Appendix): OP should be OU for consistency with Figure 3 (main paper)\n- Ticks in Figures 2c and 3c are too small to be legible\n- The reference section should be checked carefully, to name a few examples: Hermans et al. (2019) has been published, Durkan et al. (2020) has been published, Kingma et al. (2014) has been published, Aaron Van Den Oord et al. (2016) misses a venue, Aaron van den Oord (2017) has been published, Rippel et al. (2014) misses a venue, Thomas et al. (2016) has been published\n\n\n### Update\n\nThe authors have addressed most of the above criticisms. Even though part of the definite answer about directly extracting the posterior from the MI network is left to future work and no test of their method on a high-d example was included, I believe that the paper is a valuable addition to the literature. I have increased my score accordingly to acknowledge the authors' replies.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a central problem in inference in implicit models-- classical approaches on such problems ('ABC') rely on computation of summary statistics, and multiple methods for automatically finding summary statistics have been proposed. This paper provides a fresh take on this classical problem, by providing a methods for finding information-maximising summary stats. The work is original, likely impactful, and carried out rigorously and carefully. The reviewers flagged some issues with empirical comparisons, as well as discussion or relevant work-- those issues mainly seem to have been resolved in the review process. Moreover, given the originality of the approach, and provided that the description of empirical comparisons and relationship with other work are carefully and conservatively worded, I believe this will be worth publishing even if it is not always the 'best' method on all problems. 
Decision: Accept (Spotlight)