Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
This study proposes a new method for computing summary statistics in the context of likelihood-free inference. The authors pose the problem as finding the summary statistics function (in the form of a neural network) that maximises the mutual information between parameters and summary statistics. The study is technically sound and will be of interest to the likelihood-free community and inspire further method development. However, the manuscript could be improved in terms of clarity and soundness of the claims.


### Quality
The paper is technically sound, but some of the claims are not sufficiently backed by empirical evidence:

-in the main section of the paper, tables 1-3, the authors report a significant improvement in performance compared to previous methods (SRE and SNPE) (with the exception of the Gaussian copula case, where SNPE seems to have a slightly better performance). However, in the appendix, the differences in performance (using the MMD metric) between algorithms are much more nuanced and in fact, it is unclear what algorithm is the best. It would be important for the authors to discuss these nuanced results in the main section of the paper, and be careful not to oversell their approach;

-related to the above point, most results presented in the paper are averages across three independent runs for each algorithm. This is somewhat insufficient to assess the real difference in performance between algorithms, and increasing this to at least 10 rounds would be critical.


### Clarity

The manuscript is for the most part clearly written. However, it sometimes lacks enough information for an expert reader to follow all the steps to evaluate and reproduce the results. Some comments to improve clarity:

-when describing SNL in the main section of the manuscript, the authors write that after getting an estimate of the likelihood:
"We then obtain the r-th estimate of the posterior by Bayes rule". However, the posterior is obtained via MCMC and this should be made explicit for clarity. This is not described at all in the manuscript, and I would urge the authors to also include a few more details on the MCMC procedure in the supplement;

-what were the hyperparameters used for SRE and SNPE? Did the authors test different parameterisations of these for the comparison between algorithms?


### Originality

The novelty of the study resides in framing the problem of computing the approximate sufficient statistics (in the context of likelihood-free inference) as a maximisation of mutual information between the parameters and the summary statistics.


### Significance of the work

The results suggest that the developed approach is a promising step towards improving likelihood-free methodology, especially in settings with high-dimensional data. 


### Minor comments

Regarding the choice of SNPE-B over SNPE-C, the authors write "We select to compare with SNPE-B (Lueckmann et al., 2017) rather than the more recent SNPE-C (Greenberg et al., 2019) due to its equivalence to SRE shown in (Durkan et al., 2020)." However, Durkan et al. 2020 only showed similarity between SRE and SNPE-C and not strict equivalence, so the authors should rephrase this.


### Some typos:

-instead of "for automatically construction", "for automatic construction";

-instead of "problem-specific design of S as future works", "problem-specific design of S as future work";

-instead of "Likelihood-targeting approaches has the advantage that it could [...], but relies", "...have the advantage that they could...but rely...";

-instead of "our method does increases", "our method does increase";

-instead of "methods augmented with the proposed statistic [...] outperforms", "methods augmented with the proposed statistic [...] outperform";

-instead of "In Figure 2.(b)", "In Figure 2 (b)";

-instead of "in Figure 2.(c)", "in Figure 2 (c)";

-instead of "discreterized", "discretized".
review_title: Interesting and technically sound study, but clarity and soundness of the claims could be improved

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary
The paper introduces a new method for learning approximately sufficient summary statistics in the context of likelihood-free inference. To do so, neural networks are trained with a loss maximising mutual information, using a JSD surrogate estimator previously proposed in the literature. The authors show how their approach can for example be combined with SMC-ABC or SNL, leading to marked performance improvements on three problems.

### Score
I appreciate the core idea of the paper as well as the empirical improvements over SMC-ABC and SNL. As detailed below, my main concern is the comparison to existing LFI approaches that are able to automatically reduce data dimensionality (SRE, SNPE). I hope the authors will address the weaknesses and conceptual question during rebuttal. As it stands, I view the paper as marginally below acceptance.

### Strengths
+ The proposed method of learning summaries is novel and very relevant to many applications of LFI
+ SMC-ABC+ and SNL+ show marked performance improvements
+ Comparison of different mutual information estimators
+ Concise discussion of their approach versus posterior-as-mean statistic used in some previous work
+ Comparison to hand-crafted summary statistics (SMC-ABC', SNL')

### Weaknesses
As mentioned above, I see the main weakness in the comparison to SRE and SNPE, which falls short in several ways: 
- The authors include comparison to SNPE-B rather than SNPE-C arguing that “We select to compare with SNPE-B (Lueckmann et al., 2017) rather than the more recent SNPE-C (Greenberg et al., 2019) due to its equivalence to SRE shown in (Durkan et al., 2020)”. While Durkan et al. 2020 indeed show an equivalence in the loss functions of both approaches, this does not mean that they are the same and comparison is not warranted: While SNPE-C/APT trains a density estimator to estimate the posterior directly, a classifier architecture is used for SRE/AALR, combined with MCMC sampling. While these two approaches share similarity in the loss, they perform differently in practice, due to different training problems/inductive biases/algorithmic steps. Therefore, results using the newer SNPE-C version should be reported (which generally outperforms SNPE-B)
- No details on how SRE and SNPE-B were used are reported in the paper, i.e., it is unclear which network architectures were used, and whether these were appropriate for the problems at hand. I briefly checked the code in the supplementary material, but could not find hyperparameters for these methods in the respective notebooks either. In any case, this is central information which needs to be reported properly in the papers’ appendix
- OU process: Judged by MMD instead of JSD, the performance of SRE and SNPE seems comparable to SNL+ on the OU model (Figure 6 Appendix), while this is not the case when looking at JSD results in Table 3. How can the discrepancy be explained? 

More generally, regarding empirical comparisons:
- In the figures reporting results it is not defined what error bars represent
- Have the authors checked whether JSD calculation is stable, i.e., yields similar results, when using a finer grid and more than 500 samples?
- Results on the MMD metric for the Ising model are missing from the appendix
- It would have been good to include a truly high-d problem, e.g. involving images

### Conceptual question

In the appendix, the authors show the posterior directly extracted from the critic. How exactly were the plots shown for comparison and the JSDs calculated, and more specifically, could the difference in JSD be explained by calculating the normaliser on coarse grid? As a control, I would like to see a sample-based comparison, i.e., a comparison of samples from the approximate posterior extracted from T (e.g., by MCMC) versus samples generated from the NDE refit. If it turns out that results are comparable, wouldn't it make sense to think of the algorithm as a new LFI algorithm in itself rather than a pre-processing step reducing dimensionality of data for other algorithms?


### Additional comments

- As a suggestion: It might make sense to first compare and discuss SMC, SMC+, SMC’ as well as  SNL, SNL+, SNL’, and second, go on to compare SMC+, SNL+ versus SRE, SNPE


### Minor comments

- Figure 5 and 6 (Appendix): SNP should be SNPE for consistency with main paper
- Figure 6 (Appendix): OP should be OU for consistency with Figure 3 (main paper)
- Ticks in Figures 2c and 3c are too small to be legible
- The reference section should be checked carefully, to name a few examples: Hermans et al. (2019) has been published, Durkan et al. (2020) has been published, Kingma et al. (2014) has been published, Aaron Van Den Oord et al. (2016) misses a venue, Aaron van den Oord (2017) has been published, Rippel et al. (2014) misses a venue, Thomas et al. (2016) has been published


### Update

The authors have addressed most of the above criticisms. Even though part of the definite answer about directly extracting the posterior from the MI network is left to future work and no test of their method on a high-d example was included, I believe that the paper is a valuable addition to the literature. I have increased my score accordingly to acknowledge the authors' replies.
review_title: Potentially impactful, weaknesses in empirical comparison, conceptual question

final_score: rating: 7: Good paper, accept
review_content: # Summary
The paper presents a method for building sufficient statistic $\mathbf{s}(\mathbf{x})$ of observed data $\mathbf{x}$ for a parameter $\theta$. It focuses on using this statistic for performing posterior inference in a likelihood-free setting. In particular, the authors proposed to improve SMC-ABC and SNL with their statistics. The results suggest the method is able to learn good statistics and compares favorably with other LFI algorithms.
# Major comments
## Pros:
The paper is well written and easy to follow. Moreover, the introduction of SMC-ABC and SNL+ is interesting as it improves the performance of the classic algorithms. Although the training procedure in itself is not new I think (see comments below), finding a sufficient statistic is conceptually new and is very relevant.  
## Cons:
1) Formally, your method is very close to SRE. Indeed the MI loss used to learn the statistic is equivalent to the binary cross-entropy used by https://arxiv.org/pdf/1903.04057.pdf [1] to learn the discriminator. In the end, the network T can be seen as a discriminator (without the sigmoid activation in the computation of the loss) between (x, \theta) pairs that only takes as input s(x), a compressed version of x. Thus the novelty is not really about how you learn the sufficient statistic but more the way you use it. In [1], they use the ratio whereas you directly plug the statistic into SMC-ABC and SNL.
Thus when discussing MI and RE just before the experiments section, I do not agree with your arguments 1) and 2) are just an architectural choice that can be made. 
2) I wonder whether discussing Bayesian Sufficient statistics couldn't be interesting?
3) You don't write about minimal sufficient statistics whereas this is exactly what you are looking for as it is the one that will provide the best SNR.
4) Why don't you compare to more standard tasks such as the one used by SNL or SRE?
5) I did not get the point about monotonicity between two sufficient 1D statistics, could you explain why this should be correct? Indeed 
"A statistic T = T (x) is said to be minimally sufficient for the parameter $\theta$ if it is sufficient for $\theta$ and for any other sufficient statistic S = S (x) there exists a function g(.) with T(x) = g(S(x))." Nothing prevents g to be something else than monotonic. The only case where this is maybe true is if S and T are minimal.

## What could be made to address my comments

1) You should clarify the similarities between SRE and your work, showing they share the same training procedure but that you are more specific about the architectural choice. I agree that your work is original and conceptually distinct from SRE. However, the only formal difference between the two is that you specify how to input the parameters and data into the neural network whereas SRE does not. The two algorithms introduced in this paper are original and seem very competitive and thus the novelty seems to be there more than anywhere else.
2) This is just a question!
3) I think the "minimal sufficient statistic" should be defined somewhere, also discussing the impact of just finding a sufficient statistic in contrast to a minimal one should be discussed. Probably, ABC and SNL are able to discard the "useless" noise from the statistic but I suppose this has somewhere an impact on the sample efficiency.
4) Compare on SLCP, Lotka-Volterra, etc...
5) Just clarify this point.

*I will increase my score to 7 or 8 depending on how you address my comments (or convince me they don't make sense).*

## Minor comments
Figure 1: (b) What about SRE? (c) could you improve the quality of this plot, maybe creating it as a pdf?
page 6: "Note that the sufficient statistic" -> "a sufficient"?

review_title: Nice paper but would benefit from a thorougher discussion of similar work.

final_score: rating: 7: Good paper, accept
review_content: The paper considers the problem of coming up with summary statistics for ABC-type approximations.  The strategy is to find a function of the data S(X) parametrized by neural networks, with the objective of maximizing a mutual information criterion.  The motivation comes from an interesting classical result described by Shamir, Sabato and Tishby, relating sufficiency and mutual information, combined with a machinery inspired by Hjelm et al, "Learning  deep representations  by  mutual  information estimation and  maximization" published in ICLR 2019, and Monte Carlo algorithms to learn the ABC posterior and the statistics jointly. The method is illustrated on toy examples.

The topic is very interesting and the paper contains a real contribution. It is particularly remarkable as a lot of articles have been written on the topic of the summary statistics for ABC, without much progress in the last few years.  The use of information-theoretic consideration is fruitful to define a meaningful concept of "near sufficiency". Indeed we know that exact sufficiency is only attainable in exponential models, while ABC methods are intended for much more complex settings. The experiments are only dealing with simple models, not really typical of challenging settings for which ABC methods are useful, but they make a convincing case that the method is promising.

It might be worth reminding the readers of the Pitman-Koopman-Darmois theorem on sufficient statistics of fixed dimension in the exponential family, and perhaps a few more elements from Shamir et al 2008 could be recalled to motivate the relaxed notion of "near sufficiency".  There could also be more comments on the use of JSD instead of KL, as in Appendix A of Hjelm et al, as this seems important.  I would find these reminders more useful and informative than the authors' remarks on the curse of dimensionality, which seem less central to the manuscript. The authors mention a research direction around eq (7) related to the dimension of the statistic but leave it for future work. 

Proposition 2 itself is not very surprising, given that the "posterior mean as statistic" is not designed to minimize the same information-theoretic loss, and given the lack of fixed-dimensional sufficient stats in generic models. On the other hand the posterior mean is optimal for another loss, as shown in Theorem 3 of Fearnhead & Prangle. I find the writing around this proposition to be a bit partial; it would be preferable to use neutral terms and to explain that the different approaches optimize different losses.

The SMC-ABC looks more like a population Monte Carlo algorithm as in Beaumont et al., or an adaptive importance sampling strategy than an SMC sampler. In SMC the particles are typically propagated across steps, weighted and resampled, whereas here, at each iteration of Algorithm 1, particles on the parameter space are drawn from a parametric distribution $p_j$; they are not propagated from the previous $\theta$'s.

I was not sure why Figure 1c was looking so wiggly? 

The exact posterior in the Ising model could be obtained from the exchange algorithm or similar techniques for distributions with intractable normalising constants, instead of the suggested rejection sampler. See e.g.
Murray, Ghahramani & MacKay, MCMC for doubly-intractable distributions.
Moeller, Pettitt, Berthelsen & Reeves, An efficient Markov chain Monte Carlo method for distributions with intractable normalising constants.

The discretization of the OU process is also known as an autoregressive process of order 1, although the parametrization is typically different. I wonder if the choice of 'expert' statistics could be commented on further. Are the mean, sd and autocorrelation of order 1 actually sufficient here?

On the last page, "For future works, we can consider other infomax approaches." sounds a bit shallow.

In the experiments, instead of selecting a subset of the order statistics, I wonder if the authors have tried to use all the order statistics. The distance between order statistics actually might be a very sensible way of comparing marginal distributions... the "reduction of dimension" associated with the selection of some order stats might not actually bring much gains.

Finally, how does the method relate to theoretical works on ABC, such as Frazier et al or Li & Fearnhead? Do the automatically derived summary statistics satisfy the assumptions of these papers? If so, something could perhaps be said about asymptotic concentration of the resulting posteriors?

review_title: A promising ABC method with selection of statistics optimizing an information-theoretic criterion

Meta-review: This paper addresses a central problem in inference in implicit models-- classical approaches on such problems ('ABC') rely on computation of summary statistics, and multiple methods for automatically finding summary statistics have been proposed. This paper provides a fresh take on this classical problem, by providing a methods for finding information-maximising summary stats. The work is original, likely impactful, and carried out rigorously and carefully. The reviewers flagged some issues with empirical comparisons, as well as discussion or relevant work-- those issues mainly seem to have been resolved in the review process. Moreover, given the originality of the approach, and provided that the description of empirical comparisons and relationship with other work are carefully and conservatively worded, I believe this will be worth publishing even if it is not always the 'best' method on all problems. 
Decision: Accept (Spotlight)