Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a simulation-based inference method for increasing the simulation-efficiency of Neural Posterior Estimation using an alternative proposal distribution. The proposal is constructed by truncating the prior to the HDR of the previously-estimated posterior for a particular observation of interest. They call their method Truncated Sequential Neural Posterior Estimation (TSNPE).

TSNPE:
- Overcomes issues with unstable training and leakage of posterior mass outside of the prior associated with other sequential version of NPE, namely so-called APT and SNPE-C.
- Enables a specialized coverage test, which is dubbed Simulation-based Coverage Calibration (SBCC), which exploits the tractability of the approximate posterior and general empirical expected coverage testing on the truncated region due to "local amortization."
- 's empirical performance is shown on six benchmark tasks from the simulation-based inference benchmark as well as two complex neuroscience problems. The performance on benchmarks is not significantly distinguished from other methods. The previous versions of Sequential Neural Posterior Estimation cannot robustly identify the posterior on the neuroscience problems.
strengths_and_weaknesses: ## Originality
### Strengths
- The combination of existing truncated proposals with existing Neural Posterior Estimation (NPE) is novel.
- Addressing the issue of posterior "leakage" into the region without prior support in APT / SNPE-C is an important advance in the development of NPE-related techniques.
- Although SIR is not novel, using it to sample from the truncated proposal in this context is new to me.

### Weaknesses
#### Relation to TMNRE
A core weakness of originality in the paper is that the proposed method is merely the combination of Neural Posterior Estimation with a simplified version (only rejection / SIR sampling from the prior; no iterative bounding of prior for faster proposal sampling) of the proposal algorithm of [Truncated Marginal Neural Ratio Estimation (TMNRE)](https://arxiv.org/abs/2107.01214), another deep-learning based and highly-effective simulation-based inference algorithm. This holds when the marginals are taken to be the entire set of parameters; a mode which is suggested in the TMNRE paper on page 6. Unfortunately, this current paper relegates the exceptional similarity with TMNRE to a paragraph which emphasizes that both techniques are locally amortized (agreed) and writes that their aims are orthogonal (a point on which I disagree). Furthermore, just as in the TMNRE paper, the current paper emphasizes empirical expected coverage testing in order to avoid truncating the prior too aggressively; however, the TMNRE paper is never cited in this context. To the reviewer's knowledge, the TMNRE paper is first to use tests of coverage in this way and even uses [a very similar algorithm](https://github.com/bkmi/tmnre/blob/main/tmnre/coverage/oned.py) for computing coverage as the current paper's proposed SBCC that doesn't require integrating on a grid as the current paper claims (consult their code, which is available on github). Finally, relevant to TMNRE, several examples of applications of simulation-based inference to scientific problems are cited; however, relevant work applying TMNRE (which uses the truncated proposals suggested by the current paper) is not cited. For example: [Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimation](https://arxiv.org/abs/2111.08030) and [Estimating the warm dark matter mass from strong lensing images with truncated marginal neural ratio estimation](https://arxiv.org/abs/2205.09126). Also the workshop paper which predated TMNRE, [Simulation-efficient marginal posterior estimation with swyft: stop wasting your precious time](https://arxiv.org/abs/2011.13951), is not cited either.

To address this issue, I suggest the authors:
- Clearly highlight the similarities between TMNRE and relationship to NPE. In particular, by emphasizing the similar methodology (in the case of parameters of interest = all parameters) applied to a different simulation-based inference technique. In my opinion, the connection is so close that it could be mentioned in the abstract along with the relationship to Blum and François [2010]. Also highlight the new key innovations contained in this paper.
- Clarify that the SBCC algorithm (as best as I can understand it given the state of its clarity in the current draft, see Clarity) is quite similar to techniques used in the computation of coverage by TMNRE and include TMNRE in citations in the discussions of using coverage as a diagnostic tool. (A very similar algorithm is also used in the averting a crisis in sbi paper, see github code.)
- Introduce the application / workshop paper citations suggested above.

#### SBCC Algorithms
The current paper suggests that coverage testing has only been done by integrating on a grid in the cited papers; however, this is not the case. An example is shown above in the previous section for TMNRE. Here is an example from the [Averting a crisis in SBI](https://github.com/montefiore-ai/averting-a-crisis-in-simulation-based-inference/blob/a53f34ccfa2b520c58f46158e52e235766da681f/workflows/SNL/mg1/snl.py#L116-L129) paper.. Both TMNRE and the crisis in SBI papers use a similar algorithm to SBCC and the crisis in SBI paper computes coverage for the entire set of parameters. In addition to the stated similarity to simulation-based calibration, this shows that the SBCC algorithm is not novel; although its explicit enumeration is useful for the community. 

To address the issue:
- The paper should be updated to reflect that coverage is not generally computed by these other papers based on integrating on a grid.
- The paper should be updated to reflect that the SBCC algorithm is not novel.

**Caveat: Algorithm 2, which corresponds to SBCC has several typos. If this means that I didn't understand how it was novel, I would change my opinion here. Either way these typos must be fixed (see Clarity).**

#### Nested Sampling
The paper does not mention nested sampling, although truncation is an important part of the process. Please consider citing the following review https://arxiv.org/abs/2101.09675, and these papers:
- https://projecteuclid.org/journals/bayesian-analysis/volume-1/issue-4/Nested-sampling-for-general-Bayesian-computation/10.1214/06-BA127.full
- https://arxiv.org/abs/0809.3437
- https://arxiv.org/abs/1904.02180
- https://arxiv.org/abs/1506.00171


## Quality
### Strengths
- Addressing the leakage issue with APT / SNPE-C is demonstrated.
- The neuroscience tasks are made possible using the algorithm.
- The use of coverage testing to determine whether or not it is safe to truncate is a good idea and in line with previous work on the subject (TMNRE / crisis in sbi).

### Weaknesses
- The improvement on the benchmark is marginal, but the benchmark does not usually produce a clear "winner" among methods. Perhaps this should be stated within the paper.
- In Figure 6 d (and corresponding text) the use of ensembles does not produce a conservative estimator of the posterior, although it improves matters compared to single neural networks. Truncating in this round would lead to an overconfident estimate of the HDR. This is not mentioned in the paper, but it should be highlighted as exactly the failure mode to avoid!
- It should be made clear that computing coverage only in the final round does NOT tell the user if the algorithm truncated too aggressively since parameter samples are drawn from the already truncated prior. In this setting, it is like computing the coverage of different problem where the prior has been replaced by a truncated version.
- The term coverage is generally misused throughout the paper and not well explained in the proposed SBCC. The paper computes the expected coverage. Coverage is a frequentist concept which characterizes the confidence sets, not the credible regions discussed in this paper. Expected coverage is discussed the averting a crisis in sbi paper and implies a coverage test for averaged over simulations from the proposal. (See questions)
- Truncating is somehow "dangerous" when the first rounds of NPE miss a mode of the posterior, since it will not be represented in the subsequent rounds due to truncation. This is not sufficiently addressed and should have its own section. (See the limitations response prompt for more discussion.)
- It looks like the parameter rejection rate for many benchmark problems goes to nearly zero at higher rounds. The amount of help SIR provides in this is not well quantified; does using K=1024 distort the proposal distribution due to high variance in the importance weights? Does it affect the final posterior? This is an importance nuance which is not sufficiently explored. (See questions.)

## Clarity
### Strengths
- The main text is well written and easy to follow.
- I am extremely familiar with this field of research. I feel that it has a satisfactory preamble and main paper description. I will give a positive score but look to other reviewers to see if it is clear for others who are less well acquainted. Namely, I believe the paper references concepts from the development of deep-learning based simulation-based inference which are clear to me but perhaps not to others.
- I have not looked at the code presented in the paper, but hydra has been used for documenting the details of every run for reproducibility. That's very good!

### Weaknesses
- The proof of convergence in Section 6.2 is a difficult to follow. It could benefit from fewer words and more terse mathematical description.
- Algorithm 2 uses symbols which are starred then removes the stars later. It also uses symbols which are not introduced. This makes the algorithm unclear. Perhaps a verbal description would be beneficial as a comment? (See questions.)
- Algorithm 3 uses symbols which are not introduced. This makes the algorithm unclear. Perhaps a verbal description would be beneficial as a comment? (See questions.)
- The statement about the region which is well calibrated in 6.12 is unclear to me. Is the implication that the estimated posterior is truncated based upon the most recent round, thereby the support of q handles any issues regarding sampling parameters from the prior? If this is the case, it is a nuance which I suspect to pass most readers by and should be clarified. (see questions)
- It is not made clear that training on data drawn from all rounds should produce an accurate posterior. (Section 6.11) Drawing data from all rounds means that the "implicit prior," due to the use of a mixed proposal, can be modulated by a sum of R step functions with arbitrary support. Consider that in the limit of large steps that implies the modulating function can be extremely expressive. (see questions)

## Significance
### Strengths
- Although I am not an expert in neuroscience, the author's claims imply that the expanded relevance of NPE to these problems is significant.
- Addressing leakage in APT / SNPE-C is quite practical.

### Weaknesses
- As discussed in the Originality section above, the novelty of the TSNPE is marginal.
- Unless it is shown that training on data from all rounds is acceptable, then the proposed algorithm has a fundamental flaw. (It is addressed by training only on data drawn from that lies within the previous round's proposal, but then the performance on various tasks is unknown.)
questions: - Please address the bullet points raised in the "weaknesses" section above.
- Would you please clarify Algorithm 2 such that I can check whether or not it is novel?
- Would you please quantify the effects of using SIR and opposed to rejection sampling for drawing from the proposal? An analysis of the variance of importance weights would help us to understand whether SIR produces useful samples for training.
- Why do you consider computing coverage at every round to be optional? Truncating with an overconfident posterior, as is empirically produced by even the ensemble of networks in Figure 6, means that the proposal truncates too aggressively for the hyperparameter setting.
- I am not convinced that training TSNPE based on data aggregated from all rounds during truncation converges to the HDR of the ground truth posterior. Could you please clarify where in the paper this is proven or prove it here (and include it in the paper)? Specifically, I am concerned by how a collection of data from many rounds implies a prior with mass modulated by the details of truncation regions of previous rounds. It also implies that the estimated posterior outside of the HDR is untrustworthy.
- In Section 3.3 the authors claim that having (expected) coverage implies that the HDR of q_phi(t | x_0) is a super set of p(t | x_0). I don't think this is true. Would the authors please clarify or prove it?
limitations: - The simulation efficiency gained by truncation is marginal if the posterior is nearly as wide as the prior. This is mentioned in the result section, but should be put into the limitations section.
- It is possible to miss modes with this method and inaccurately remove them from the analysis. This should be emphasized. Coverage alone does not stop this from happening, to my knowledge. (see questions / quality weaknesses.)
- The limitations of computing coverage using the SBCC algorithms in high dimensions should be mentioned. Yes, drawing parameters is cheap from this method but it may be that a significant amount of proposed parameters are below the contour line of interest. I expect this to suffer from the curse of dimensionality.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Given a generative model $\mathcal P(\theta, x)$ of the model parameters $\theta$ and data $x$, the goal is to solve the inverse problem $\mathcal P(x | \theta)$. Neural Posterior Estimation (NPE) uses simulations from $\mathcal P(\theta, x)$ to train an approximate posterior distribution $q_\phi (\theta | x)$ that is tractable and has learnable parameters $\phi$.
If one is interested in solving the inverse problem for a particular data realization $x_0$, Sequential Neural Posterior Estimation (SPNE) proposes a iterative sequence, starting at the prior, where the sampler for the model parameters is drawn from the previously learned $q_\phi$.

Rather than plugging in the previous proposal. this paper proposes a sequence of truncated prior distributions.
With Truncated Sequential Neural Posterior Estimation (TSPNE), the sampler for $\theta$ at each step is drawn from the prior distribution truncated at the highest-probability region (HPR) of the most recent approximate posterior.
This overcomes the drawback of SPNE that the maximizer of the loss function no longer converges to the true posterior.
The authors show that as long as the chosen HPR covers the support of the true posterior distribution, TSNPE provably converges to the true posterior.
The authors also introduce a metric called simulation-based coverage calculation (SBCC) that evaluates the coverage of their learned posterior approximation. This coverage is based on whether the quantiles of the learned log-density cover the log-density evaluated at the true parameter across repeated draws from the generative model.

The authors evaluate their procedure on six synthetic benchmarks and compare to the original NPE and APT, which is a specific variant of SPNE.
They show that their method mostly matches or outperforms APT and NPE, although APT seems to do better in two cases.
They then compare APT and their TSNPE on a simulation of pyloric networks in crabs.
Their TSNPE method does well in this example and is similar to previously known posteriors.
They also show that APT does poorly in this case.
Their final example is on a multicompartment model of a single neuron. Since only one proposal exhibited poor coverage based on their SBCC metric, they use an ensemble of 10 proposals and find this leads to much better coverage.
The competing APT method also does poorly in this case.

strengths_and_weaknesses: ## Strengths
This paper builds on the established Sequential Neural Posterior Estimation (SNPE) framework, but using a truncated prior distribution as a prior. The truncation is achieved by rejecting samples outside of the highest-probability region (HPR) of the previously-estimated posterior. This method is simple, yet it makes intuitive sense because it prevents sampling parameters that are not possible under the prior and parameters unlikely under the estimated posterior.
Moreover, it corrects the main problem with SPNE by ensuring that the minimizer of the TSNPE objective at eatch stage is the true posterior.
This property hinges on the HPR covering the support of the true posterior, which may not be guarenteed in practice, but the authors develop the SBCC metric to verify the validity of the learned posteriors.
This lends confidence to the proposed method.

The pyloric network and multicompartment experiments demonstrate that TSNPE can work in challenging scenarios that the competing method (ANP) fails in. In the multicompartment experiment, it was nice to see that the authors' SBCC metric identified that a single neural network led to poor coverage and was corrected by an ensemble method.

## Weaknesses
The main weakness to this work are the simple simulation experiments in section 4.1. While TSNPE appears to match the performance of other methods, it would be reassuring to re-create the failure of APT observed in the more complex neurological applications in a simple setting.

questions: + Instead of an ensemble mixture to solve the poor coverage in the multicompartment experiment, would it work to form a defensive mixture by sampling from the untruncated prior with some probability? Even if this probability were small, it may help get around the issue of the HPR not covering the true posterior support.
+ It has been suggested that the forward KL objective (used here) leads to an overdispersed posterior compared to the reverse KL typical of variational inference. Is this applicable to TSNPE?

limitations: The authors address the main limitation of their method that the estimated HPR may leave out the true posterior support, and they describe their proposed SBCC metric for checking poor coverage.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a new simulation-based inference (SBI) technique that sequentially learns a neural surrogate for the posterior. The key idea is to use a truncated version of the prior, rather than the current posterior, as a proposal distribution to generate new samples. This removes the need for an impractical importance weight during training, leading to two practical advantages: it solves a posterior leakage problem that other state-of-the-art sequential neural posterior methods (APT) have, and it simplifies the use of calibration diagnostics. There are some downsides, too: the truncation biases the posterior and sampling from the proposal distribution requires some rejection sampling, which can be inefficient.

This new method, TSNPE, is demonstrated on a suite of standard benchmarks and finally on two real-life problems from neuroscience. On the benchmarks it performs on par with other neural posterior estimation methods, while on the neuroscience problems it enables plausible solutions for the posterior where other methods failed.
strengths_and_weaknesses: ## Strengths

- Scaling SBI methods to more complex problem is an important (and to some extent unsolved) problem, with potential impact in many areas of science.
- The proposed method is, overall, sensible (with some caveats, see below) and can help make SBI work in more complex problems.
- The downsides of the presented methods (in particular the bias from $\epsilon > 0$ and the inefficiency of rejection sampling) are acknowledged and discussed (though this could be further improved, see below).
- The empirical evaluation is strong. I appreciate that the authors both present benchmark problems (for which true posterior samples are available) and more challenging real-world neuroscience problem. In the latter, the authors do a good job of discussing the plausibility of the learned posterior. While the presented method does not lead to a huge improvement on the benchmark tasks, it does appear effective in the neuroscience problems, and that is ultimately more important.
- The paper is very well written and clearly structured, quite a pleasure to read.
- The figures (both schematics and experimental results) are well designed and very clear.
- Related work is discussed in detail and the relation of the presented method to other approaches is made quite clear.
- The authors have released the code to run the experiments.


## Weaknesses

- While TSNPE is a new algorithm, it is a modification of existing algorithms like SNPE and APT. Similarly, the proposed calibration tool builds strongly on existing work. This paper is thus somewhat iterative rather than revolutionary.
- There are a couple of aspects that could use a more in-depth discussion, see the Questions section below. In particular, I would like to understand better why rejection sampling in TSNPE can be more efficient than rejection sampling based on the APT posterior. Also, it would be great if the authors could explain a bit more why sampling from a truncated prior isn't quite sample-inefficient compared to sampling from the posterior.

## Bottom line

I would like to thank the authors for a solid contribution to the toolbox of simulation-based inference and to a high-quality paper. Still, I think it can be made stronger by extending the discussion and addressing some questions (see below). 
questions: The authors present the posterior leakage of APT as a major motivation behind their work. As they point out, this posterior leakage can be fixed in APT through rejection sampling, but that can be inefficient. However, the proposed algorithm (TSNPE) also involves rejection sampling (plain or in the form of SIR). I would appreciate if the authors discuss in more detail of why the rejection sampling in TSNPE is more efficient (has a lower rejection rate) than the rejection sampling in the APT posterior. Could SIR be applied equally well to the APT posterior?

Is it fair to say that in some sense, TSNPE is a compromise between vanilla NPE (no annoying importance factors in training) and SNPE (optimal use of past information in the posterior)? I'm trying to better understand how much information from past rounds is lost by sampling proportional to the prior rather than proportional to the current posterior (within some truncation region). Can the authors discuss this some more?

Phrasing the same question in a different way: Compared to standard SNPE, the proposed algorithm presents two changes. 1) The proposal is truncated and 2) within the truncated support, we sample proportional to the prior rather than the posterior. It would be interesting to disentangle these two factors and understand how they individually contribute to the sample efficiency. If I'm not mistaken, only doing 2 (and not 1) corresponds to vanilla, non-sequential NPE, is that correct? How would an algorithm do that does 1 but not 2, i.e. truncates the posterior and then samples proportional to the posterior within this truncation region? Would this for all practical purposes be identical to APT?

Some minor comments:
- The authors do a good job of listing the most closely related references. Perhaps SBI has grown to be too large a field by now, but there are still a number of SBI works not cited here.
- In the first sentence of the abstract, the authors describe SBI as "learning posterior distributions". As the authors themselves point out later, in many SBI methods other statistical objects like the likelihood or the likelihood ratio are learned, and other methods again do not have any explicit learning at all (like old-fashioned ABC). Maybe it is a good idea to rephrase this sentence. 
- The word "likelihood" is a bit overloaded and used both for the likelihood $p(x|\theta)$, but often also to the density of some trained surrogate of the posterior $q(\theta|x)$. This is to some extent unavoidable, as "maximum likelihood" is such a known term, but in other instance I found it slightly confusing (for instance in line 128). Perhaps the authors could refer to the "probability density" of the posterior rather than the "likelihood" of the posterior?
- In the experiments, how big is the error of SIR due to finite K?
- In the benchmark experiments, does the problem of APT posterior leakage occur? Do you fix it with rejection sampling (or in some other way)?
- In the experiments, do the methods that rely on rejection sampling or SIR have a substantial overhead in runtime from that, or is that negligible?
limitations: The authors have addressed the limitations of their methods and discussed them well. Some aspects could use a more in-depth discussion, see the Questions section above.

The potential societal impact is not discussed, but I do not see any obvious issues.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript presents a new sequential neural posterior estimation (SNPE) algorithm for simulation-based inference. Simulation-based inference refers to the task of inferring parameters of interest from a real observation, based on a simulator. Neural posterior estimation (NPE) refers to methods aiming to build a surrogate for the posterior (typically a normalizing flow) based on samples from the simulator. Sequential NPE refers to their sequential counterparts that alternate between training and sampling phases with the objective of sampling simulated observations close to the real observation of interest and hence typically sample parameters to use for simulation from the current posterior surrogate. However, current SNPE methods suffer from two issues limiting their applicability:
* As training samples are sampled from the current posterior surrogate, they are non-amortized, meaning that the algorithms must be fully rerun for each observation. In opposition, amortized methods reuse the same model to perform inference on multiple observations. Non-amortized methods hence have the drawback of being inefficient to diagnose. 
* As parameters used for simulation are not sampled from the prior, parameters with zero prior probability could be sampled. This phenomenon is called leakage.

Truncated SNPE (TSNPE) addresses both of those issues by using,  as proposal, a truncated version of the prior in place of the current surrogate posterior. The prior is truncated by removing parameters with surrogate posterior density below a given threshold, hence focusing on parameters of high posterior density. Consequently:
* Parameters sampled are always within the prior support, solving the leakage issue. 
* It makes the method locally amortized as the built surrogate is valid for observations resulting from parameters sampled from a region of the prior instead of a single observation and hence enables efficient diagnostics. The paper introduces the simulation-based coverage calibration (SBCC) diagnostic that leverages this property.
strengths_and_weaknesses: ### Major
* (+) The paper is very well written and easy to follow.
* (+) The method is sound
* (+) It addresses two important issues in SNPE (leakage and diagnostics)
* (+) Experiments are convincing.

### Minor

* (-) The SBCC diagnostic is quite known in the context of NPE and NRE (for example, see "Arbitrary Marginal Neural Ratio Estimation
for Simulation-based Inference", Rozet et al. 2021) and is a straightforward extension of SBC. Therefore, in my opinion, it should not be framed as a core contribution of the paper. The contribution rather lies in the development of an sbi algorithm that allows the use of this diagnostic (which is great).
questions: None
limitations: I do not see any potential negative societal impact of this work.
review_title: null

Meta-review: metareview: This paper received generally positive reviews, with one reviewer originally weakly backing rejection but ultimately backing acceptance after substantial discussions, and the other three confidently backing acceptance from the off.  Based on the reviewer's comments and my own assessments, I see the positives and negatives of the work as follows:

Positives
- Very strong writing and presentation
- Good range of experiments
- Key ideas seem relatively simple to use and deploy
- Important problem area and seems to make decent progress on two known issues in the area (leakage and coverage tests)

Negatives
- The novelty of the work is quite limited: the core approach is arguably a special case of an approach that already exists (APT) and the paper is mostly combining known ideas rather than proposing anything especially original. Thus, though the paper does certainly have some new ideas and is potentially useful to the community, I do think it is quite incremental.
- The use of SIR is worrisome and likely to cause theoretical and occasional practical failure cases, even if these have not especially manifested in the current experiments
- The method introduces biases (from SIR and the truncation itself) that might be difficult to quantify and there is a lack of any notable theoretical guarantees (though the objective is quite clearly sound with epsilon=0 and no iteration of the truncation, this provides no guarantees for real-world settings that will have epsilon>0 and multiple rounds of truncation).
- Some of the claims about efficiency and rejection rates could have been more clearly demonstrated and explained.

On balance, I agree with the reviewers that the positives outweigh the negatives; this is well-polished and potential useful work that will be of interest to the community.  As such, I recommend acceptance.

Minor comment
- I believe that the suggestion that TSNPE “can scale to problems that were previously inaccessible to neural posterior estimation” is over claiming given the actual experiment results, given the fact that only one approach (with a single “out of the box” proposal) is actually compared to and the fact that the results are generally qualitative rather than quantitative.  Moreover, given that TSNPE can arguably be seen as a special case of APT in the first place, the claim seems distinctly unreasonable.  I would thus like to see this claimed dropped or at least significantly toned down.
confidence: Certain
Decision: Accept