Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a novel algorithm for estimating the posterior distribution in simulation-based inference (SBI) settings, where the likelihood is not known but it is implicitly defined via a stochastic simulator. The work proposes truncated marginal neural ratio estimation (TMNRE), which is able to approximate the posterior distribution only for a small number of parameters of interest and making the sampling more efficient by sequentially truncating the prior for a given observed data. In addition, the posterior distribution is amortized and can be used to check the goodness of fit of the posterior distribution, albeit only in the areas which have not been truncated during training. The paper offers extensive experiments for different parameter dimensionality, as well as a comparison with the state of the art in SBI and theoretical analysis of how much error is introduced with the prior truncating process.
main_review: I found the paper clear and easy to read, with the main algorithm being presented clearly (the comments in the algorithm box are much appreciated), as well as the relationship with related work. The experimentation is extensive and tackles different settings that could be challenging: an asymmetric low-dimensional distribution and a high-dimensional multimodal example, in addition to a series of standard tasks in SBI which have been recently established by [1] and a physics-motivated example in Appendix E.
Additionally, the authors elaborate on how much the \epsilon impacts the approximation error in a couple of settings, which helps the reader get an idea of the tradeoff in posterior inference; having said that, the fact that the \epsilon is set to be very low likes make a negligible difference in inference (at least in the example shown in the paper).

Overall, I feel this is a solid paper and would be a great addition to the SBI community.
I am including some (minor) comments below:

- The amortization for every data and parameter value is a nice property, but it seems like it is restricted to the constrained region of the prior. Given the truncation in this paper seems to be pretty mild (in terms of tail trimming), I assume that checking the posterior fit for different observed data would still not be affected by the truncation. Can the author comment on how empirical check would be affected by a more aggressive trimming?

- The truncation idea is quite interesting and potentially beneficial to other methods. As the author observed, creating a hyper-rectangle is not the most efficient way of doing truncation, but I assume it's probably computationally faster as one does not need to consider the correlation between parameters. Would it be possible for the truncation mechanism to just consider the correlation between the parameters of interest and doing 1D truncation among the nuisance parameters as a tradeoff? In addition, can the authors comment on how many initial samples one should use in order to avoid missing a potentially relevant area of the parameter space and how much is this number affected by the dimensionality of the parameter space? 


[1] Lueckmann et. al, "Benchmarking Simulation-Based Inference", AISTATS 2021


limitations_and_societal_impact: Yes, limitations and societal impact have been addressed adequately.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This study contributes with a new simulation-based inference method, Truncated Marginal Neural Ratio Estimation (TMNRE). It builds on a previous method, Neural Ratio Estimation (NRE), by combining NRE (for directly inferring marginals of the posterior distribution) and the sequential constraining of the prior distribution by truncation (targeted to a particular observation). On standard tasks, TMNRE is shown to perform on par with state-of-the-art methods. Furthermore, the sequential truncation of the prior allows for local amortisation and thus makes it easy to perform (local) consistency tests.
main_review: ### Originality

The novelty of the method consists in the sequential constraining of the prior distribution by truncation. Previous work is for the most part appropriately cited and discussed, and it is clear how this work differs from previous contributions. However, I believe it would be important to also discuss the paper by Blum and François (2010 - Non-linear regression models for Approximate Bayesian Computation), as this proposes an ABC method also with prior truncation.


### Quality

The paper is technically sound, and claims are mostly well supported by empirical evaluation. However, I have a couple of comments I would appreciate the authors to address:

-the authors write "Targeting marginals directly, by estimating only the marginal posterior for the parameters of interest, is simpler and sufficient for most purposes." As a first minor point, I believe "most purposes" to be quite an overstatement, and would recommend the authors to instead write "many" or something similar (see also a related point under section "Limitations");

-throughout the manuscript, the authors put quite some emphasis on the fact that TMNRE is more efficient, given its sequential prior truncation feature. However, while this efficiency is convincingly shown against MNRE (no truncation), NRE and some other variants (Figures 2,3,5), it would have been important to also show this against other methods (e.g. SNLE, SNPE), and for several simulation budgets.


### Clarity

The paper is clearly written, and provides enough information for an expert reader to understand all the steps to reproduce the results. I would just like to point out a few typos:

-lines 161-162, rather than "r(x|theta)", "r(x,theta)";

-Figure 2, right panel, the x-axis is in units of the common logarithm, and that should be specified;

-the authors write "That means simulations from within this constrained region are likely informative while simulations from outside the region are likely uninformative". I would suggest the authors to rephrase this sentence, as the wording "(un)informative", without stating about what, is a bit imprecise and potentially misleading.



### Significance

This new method will be of interest to the simulation-based inference community and inspire further method development.


### After rebuttal

Thank you for the careful answer to my review. After reading all the reviews and authors' rebuttal, I believe that this is an interesting paper and that the authors made an effort to address all the reviews, and in my opinion, to satisfaction. While TMNRE is a method for the exploration of the posterior (or more concretely, its marginals) rather than a method for full Bayesian inference, I think this will be a valuable/useful addition to the SBI toolkit. I increased my score accordingly.
limitations_and_societal_impact: The authors have for the most part adequately addressed the limitations of their work. However, I would like to raise a point regarding the limitations of estimating marginals rather than the full posterior: arguably, in many cases the practitioner of simulation-based inference would indeed be interested in having good posterior predictives. These posterior predictives are not possible with TMNRE (as the authors acknowledge in line 345), and I would recommend the authors to more explicitly acknowledge this potentially impactful limitation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for performing likelihood-free inference in parametric simulators.  The method presented is thematically similar to, and builds upon, a recent tranche of work using neural function approximators to ameliorate the inherent difficulties in likelihood-free settings.  The method takes the interesting and unique approach of modelling an arbitrary set of many smaller marginals of the full posterior, instead of directly targeting the full posterior.  These marginals are estimated by using neural ratio estimation between the target marginal posterior and the full marginal.  The prior is then truncated (to a hyperrectangle smaller than the support of the original prior in each dimension) to concentrate simulations into regions of significant posterior mass.  These estimators (the ratio estimators and the truncated prior) can then be used to establish an estimate of the posterior distribution.  The method is evaluated on reasonably standard tasks, and an interesting cosmology example (although this was inexplicably demoted to the supplementary materials).  
main_review: ### Summary+

The authors posit that modelling many lower-dimensional marginal posterior distributions is a fundamentally easier task than modelling a high-dimensional posterior distribution.  This seems like an interesting and intuitive design choice.  It also seems to be a relatively novel formulation in this context (see comments below).  I can believe that learning multiple cheap classifiers/ratio estimators is computationally streamlined, and defines a fundamentally easier inference task.  

The prior art is well surveyed (although the nuts and bolts of the prior art are not explored and hence some familiarity is required).  The problem formulation paragraphs (S2.P1,2) are compact and neatly introduces the problem domain, and indicates the solution path that the authors will take.  

The algorithm block is beautifully presented and makes the method very easy to understand.  I believe that I could implement the method using pretty much just the algorithm block.  Chapeau.  

The authors benchmark their method on the LFI benchmark dataset, as well as applying their method to two further examples.  The authors TMNRE method performs slightly worse on the benchmark tasks compared to the baseline methods.  A major point in the SBI baselines paper is that the choice of metric is significant, and so I would have liked to have seen these methods compared using a slightly broader range of metrics than a single metric.  

I broadly like the submission, although I am not entirely convinced by it and its significance  (see comments below).  I have erred on the side of weak acceptance because I believe the work is sound, the paper is well put together and executed, and it has been well evaluated and the results thoroughly analysed.  The paper is generally well written, and is actually quite enjoyable to read.  The authors also release code, for which they are to be commended for.  However, there is no theoretical innovation or particularly inspiring or transferrable methodological innovation.  The performance on the standard benchmarks is not outstanding, and the baselines were not fully tested on the other experiments, making meaningful comparison difficult.  Finally, the extent to which this method advances our understanding of LFI more broadly is not especially clear to me.  For me, these points limit the reach and impact of this paper.  I would, however, consider upgrading my score if my concerns were successfully allayed.


### Major Comments:

1 - Can the authors comment on the fidelity of modelling a posterior distribution as a collection of marginals?  I am not convinced that this would transfer well to distributions with complex and richly structured posteriors.  I also do not agree with the author's assertion that a joint distribution can be “unnecessarily informative”.  I would rather “pay more” to recover the full joint, and then marginalize post-hoc, as opposed to recovering a simplified representation from which I cannot recover the full representation.  When targeting the full joint, I believe this method just reduces to NRE+truncation.  NRE is known to perform relatively poorly compared to more sophisticated methods, although the truncation _may_ help somewhat.

2 - There are two methods presented, MNRE and TMNRE, correct?  Why wasn’t MNRE tested in Figure 1?  It also strikes me that the torus model and the eggbox model were well-chosen experiments for this method -- the posterior does basically factorize as a product of marginals with well-concentrated mass, and hence I am not especially surprised that these methods perform well.  I would be interested to see how well something like SNL performs on the torus model.  I suspect it would perform exceptionally well, and provides tractable marginals as stipulated by the authors, and hence I do not see a major contribution from this method there.  

3 - Truncating the prior is a reasonably low-overhead method for increasing sample efficiency.  However, for the eggbox example, surely this truncation method leads to the hypervolume retained having a much larger (hyper)volume than one would desire?  This would drive down sample efficiency, especially as regions with posterior mass are driven towards the edge of this volume.  This will _dramatically_ reduce sample efficiency in high dimensions, in a way that more sophisticated sequential methods are not affected.  Furthermore, if the axis were rotated through 45 degrees (in all dimensions), then the volume of the hyper rectangle increases by a factor of 2^5=32, and hence the truncation will be a factor of 32x less efficient.  This seems to be a major drawback to me.

4 - It seems to me that truncation may be a step back from the SNL-esque methods.  I would prefer to see something along the lines of {SMC, MCMC}-MNRE, where instead of truncating the prior, a discrete/atomic representation is used.  This would allow computation to be concentrated more efficiently, would relax the dependence on axis-aligned truncations, and prevent regions of parameter space from being aggressively deleted early in training.  I realise this is not the paper that has been written, but it is a sticking point in my head that a set of strictly contracting axis-aligned truncations is just about the least flexible approach that could have been taken.  You allude to this in the discussion (Lines 349-352).  Do you have any further comments on this?  This seems like the biggest weakness of the method.  (I suppose you want to write Canonical TMNRE next…!)

5 - I do not understand these “consistency checks through local amortization”.  I figure you are suggesting that through the use of synthetic data we can test the inference result by recovering the ground truth.  If this is the case, this seems trivial/generally true to me and does not seem to be a real plus-point of your method in particular.  The real issues with methods such as this arise from model-mismatch in the generative model, and hence there are no real tests for this.  I think this claim is overstated, and unless I’ve gotten something wrong, I would rather see this removed.  (It could be referenced in passing as a sanity check or for debugging -- but nothing more).  I could not follow Section 3.3 as a result.

6 - According to Algorithm 1, the data points aren’t drawn from the prior, they are drawn from the truncated prior, and so is the ratio estimator targeting the correct quantity?  This may just be a typo, i.e. Line 5 and Line 7 are identical (modulo a dash), whereas I believe Line 7 should drop the indicator term.  

7 - Using the C2ST score seems a little biased to me.  Since the other methods are computing full posteriors, and yours is only computing a single marginal,  performing a two-sample test comparing yourself to the full posterior actually seems a little harsh on your own method -- even _incorrect_ as a metric!  I suppose this shows your method is working admirably, but also shows that maybe the solutions just happen to admit a factorized posterior (and hence are amenable to your method).  Either way, I find using this as the only performance metric a bit of a head-scratcher.  As mentioned above, I would like to see a more diverse suite of performance metrics examined.  

8 - You recover N^2 marginal pairs, and N marginals.  The N^2 + N marginals that you recover will all be dramatically different as a result of marginalizing over N-2 other variables.  Practically speaking, how do you select which distribution to draw from for a particular distribution?  Moreover, how do you actually go about generating a sample from the joint...? 


### Other Comments:

A - Are the authors able to comment on the difference between their method and that of Jeffrey & Wandelt?  I believe that the work is sufficiently different to be easily classified as novel throughout, but I think a more substantiated explanation of any similarities and differences would be useful.

B - Does truncating the prior lead to sections of parameter space being permanently deleted?  I don’t think the algorithm could recover if a section with mass were not covered.  I realise this is true for most sampling algorithms, and Fig 4 suggests that the distribution starts broad and then narrows.  But it strikes me as though it could be particularly dangerous in this setting if there is categorically no way to expand the distribution.  Maybe consider adding some discussion of entropy regularization to the prior to the conclusion.

C - There is a relatively substantial limitations section in the supplementary materials.  I would prefer it if some of these limitations were brought up and explored in the main text.  

D - The cosmology experiment is by far the most interesting of the experiments.  I do not understand why it was demoted to the supplementary materials.  

E - There also seems to be a missed opportunity using the marginals recovered to establish independencies or conditional independencies in the posterior, where this method explicitly enforces the efficiency gains that these independences can provide. 

F - An interesting point that is not strongly emphasised is that the same samples can be (re-)used to learn all of the ratio estimators.  

G - I am also a little confused about the “local amortization”.  This method is inherently sequential and conditioned on a single $x_o$, and so there is no real amortization going on.  Around Line 190, you say that you train “locally amortized posteriors that are valid for parameters $\theta \in \Gamma^{(m)}$”.  Are suggesting that the function approximators are going to be someway accurate/informative over the entire domain of $\Gamma^{(m)}$?  This should be explained more clearly if this is the case.


### Minor/Typographical Comments:

- Inline or wrapped figures should be avoided.  They are bad for readability and flow.  Figures should really be floated to the top (or bottom at a push) of the page.  
- I don’t understand the sentence beginning Line 30: “With existing…”.
- Line 129: I don’t understand how a likelihood ratio estimator can be overconfident...! 
- Line 147: Incorrectly rendered characters (I only see boxes).
- Some of the font sizes in the figures are TINY.  
- I don’t really understand what Figures 4(b-d) are showing.  
- Figure 6 does a much better job of showing the truncation compared to Figure 4a. 
- The discussion is very long and doesn’t add much (partly because the rest of the paper is pretty well written).  This could be cut down and used for the cosmology example instead, or, for Figure 6.
- The societal impact statement is too long.  I think it is safe to say here: “There are no societal impacts.” (as is explicitly permitted in the submission instructions). 

limitations_and_societal_impact: No societal impacts of note.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors suggest the use of neural ratio estimation to target the marginal posteriors directly, thereby circumventing the more difficult problem of obtaining the full Bayesian posterior.
The method comes with other benefits such as local amortization as well as a sequential truncation technique for targeted inference. 
The method is validated on several benchmark examples. 

main_review: __Originality:__ There is something to the idea to circumvent the trouble of full Bayesian inference and target the marginal posteriors directly. The authors also consider modifications that help to improve the performance by truncation. With its aims this paper is a different take on simulation-based inference than previous approaches. 

_Some minor comments:_
- References to related work have a recency bias. For example, when writing “...the traditional approach is Approximate Bayesian Computation (ABC) [7]...” the reference [7] is a survey from 2018. However, ABC techniques already emerged in the 1990s. If the authors don’t want to add any more references, then I suggest something like [see e.g. 7 for a reference], in latex `\cite[see e.g.][for a reference]{bibkey}`.

- Inference for simulators goes back at least to the 1980s: 
Diggle, Peter J., and Richard J. Gratton. "Monte Carlo methods of inference for implicit statistical models." Journal of the Royal Statistical Society: Series B (Methodological) 46.2 (1984): 193-212.

__Quality:__ 
While it might be true that many practitioners mostly consider marginal posteriors (be it just because of visualization) I think the paper’s claims are too strong in parts. 

For example, Table 1 is very suggestive, effectively showing the properties of the method but indicating that those are _the_ desirable properties of an algorithm. In particular, the “Simulator efficient direct marginals“ are not by necessity a desirable property as I would assume that most Bayesians would prefer to have access to the joint distribution if cost was not an inhibiting factor.

__Clarity:__ The paper is generally clearly written and well organized. I did find some minor points the authors might want to change in a revision to additionally improve the clarity of the paper. 

_Some minor comments:_
- May I suggest the authors use `\usepackage[sort]{natbib}`
- I am not a fan of the formula
$$
	f(x | \theta) = \int \delta(x - g(z, \theta)) f(z | \theta) dz
$$
and I feel it adds more confusion than clarity, not least because of its double abuse of notation. First, it abuses notation by using the same symbol for the distribution of $X$, $f( \cdot \mid \theta)$ and for $Z$, also $f( \cdot \mid \theta)$. They would be better references as $f_X$ and $f_Z$. Second, the use of the Dirac delta as a density requires some mathematics that are absent from the rest of the paper and might especially be confusing to non-physicists.  Third, I fail to see any additional content by writing the simulator likelihood in this way, effectively denoting an unknown quantity $f(x \mid \theta)$ as integral over another unknown density $f(z \mid \theta)$ and an additional unknown function $g$. 
“The full joint posterior can be unnecessarily informative.” I don’t think that makes sense as a criticism. The additional information of the posterior might not be worth the additional cost afforded, but “unnecessarily informative” is too blunt.
The notation $\{g(\theta, z) : \theta \in \Omega, \forall z\}$ is confusing. The quantifier “for all” ($\forall$) is usually followed by a predicate, but I don’t see what predicate should be satisfied by all $z$.


__Significance:__ 
Approaches that help reduce the computational burden of simulation-based inference are an important part of research. The current draft of the paper needs some minor improvement before it is ready to be published. However, but I am happy to increase my score if the author can address the points raised here. 

__Edit after rebuttal:__
The authors have addressed all my concerns on clarity and have suggested changes that in my opinion improve the quality of the manuscript. I have therefore decided to increase my score towards acceptance. 
limitations_and_societal_impact: Societal Impact has been adequately addressed.
review_title: null

Meta-review: The paper proposes a method for simulation-based inference that is based on neural ratio estimation and estimates marginals of the posterior distribution.

The reviewers are generally positive about the submission, citing the usefulness of the proposed method, the clarity of the presentation, and the good variety of experiments as strengths. There were some initial concerns, but the reviewers appreciated the extensive and thorough responses provided by the authors and most concerns were allayed. Overall, I'm happy to recommend acceptance.
Decision: Accept (Poster)