Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This study develops a new method for simulation-based inference (RNPE), capable of diagnosing model misspecification and reducing its impact on posterior inference. The empirical results on three tasks show that RNPE performs more reliable inference than a previous method (neural posterior estimation, NPE) according to a few metrics -- log probability of true parameters under the inferred posterior, and posterior coverage. In addition, RNPE provides a misspecification probability for each summary statistic on each task, information that NPE does not provide.

strengths_and_weaknesses: ### Originality

Simulation-based inference has the potential to accelerate the process of scientific discovery across domains of science. One of the main limitations of this set of tools is the problem of model misspecification, a problem which has rarely been tackled. The questions addressed by this study are thus timely and relevant.


### Quality

The paper is technically sound, claims are backed by empirical evidence, and the contributions are appropriately put in the context of previous work. However, I have one point that I believe would be crucial for the authors to tackle:

-RNPE was tested in tasks for which the misspecification impacted only one summary statistic. Although this could be a realistic scenario in some cases, for many cases, the misspecification is across several summary statistics. In such cases, and if the summary statistics are not correlated with each other, then the extension to misspecification across several summary statistics seems trivial. However, in many realistic cases, there could be some residual correlation between summary statistics, and there one might expect the feedback from RNPE regarding misspecification probabilities to be less precise: there could be some indeterminacy in terms of misspecification, e.g. with a trade-off between misspecifications of different summary statistics. I believe the study would benefit from discussing such cases and including at least one task tackling this.


### Clarity

The manuscript is clearly written, providing enough information to understand the technical contribution and empirical results. I just have one point that I would appreciate the authors to clarify:

-in Figure 3b, the density for the denoised samples is shifted towards the right, without (seemingly) any mass around the observation. I might be misunderstanding something, but wouldn't one expect some non-negligible mass around the observation? If so, is it hidden in the plot by the green lines, and could one make it more visible?



### Significance

The technical development and empirical results will be of interest to the ML community. In addition, the technical development could quite effortlessly be extended to other simulation-based inference algorithms, a possibility that will certainly entice other researchers on this topic.


questions: As written above, the manuscript would benefit if the authors would include a task with misspecification across several summary statistics.

limitations: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Simulation-based inference (SBI) enables Bayesian parameter inference in complex scientific simulators for which the evaluation of the likelihood is usually intractable. In most recent works on SBI this is achieved by conditional density estimation with neural networks trained on data simulated from the model. However, SBI generally assumes that the model is well-specified, and up to now it is not clear how SBI methods behave in the misspecified case, e.g., when the data distribution induced by the simulator (and the prior) does not contain the experimentally observed data. 
This paper proposes a new SBI approach, RNPE, to enable inference even in the misspecified scenario, and to additionally obtain insights into the nature of misspecification present in the model. The goal is to obtain the posterior over the model parameters given the experimentally observed data, $p(\theta | y)$, given only (potentially misspecified) simulations from the model $(\theta, x)$. RNPE (robust NPE) extends the established SBI method Neural Posterior Estimation (NPE) to a two-step procedure: First, one performs neural density estimation on simulated data to learn $p(x)$, and assumes a noise model of the mismatch between the observed data $y$ and the simulated data $x$ , $p(y | x)$, to then obtain “denoised” posterior samples from $p(x|y)$ via MCMC. Second, one performs NPE on simulated data to obtain an amortized posterior estimator $q(\theta | x)$ . Finally, one can obtain the posterior via a Monte Carlo approximation by repeatedly sampling denoised $\tilde{x}$ from the first step, and obtaining corresponding posterior samples from $q(\theta | \tilde{x})$. 
RNPE is compared to NPE on three misspecified SBI example problems and shows consistently better performance on the evaluated metrics. In addition, the authors demonstrate that the inferred noise model from the first step can by useful to diagnose whether specific summary statistics are misspecified, which in turn can serve as a tool for model criticisms. 
strengths_and_weaknesses: - *Originality:* 
The paper introduces a new SBI method, RNPE which extends the established SBI method neural posterior estimation (NPE) to handle scenarios where the assumed model is misspecified. This is a new direction of research in the field of SBI and RNPE clearly differs from previously proposed approaches in this direction. Most of the previous work is adequately cited, except for the work on neural ratio estimation for SBI (NRE, Hermans et al. 2019, Durkan et al. 2019, Thomas et al. 2020), which should be seen as the third arm of neural network-based SBI methods (next to NPE, and NLE). Additionally, I think it should be considered citing the original papers on HMC alongside the mixed HMC paper, e.g., Neal (1995), MacKay (2003), or Neal (2011).
- *Quality*: 
The idea of RNPE to assume a noise model and to learn $p(x)$ via density estimation to then obtain denoised data, followed by NPE, seems quite original, complete and overall technically sound to me. The required derivations are straight forward to presented clearly in the paper. I have two critique points regarding the evaluation:
    - first, I think it is essential to fully study and evaluate RNPE on a tractable example like the proposed Gaussian task (the Gaussian task is ideal because one can gradually increase the misspecification from well-specified (equal variance) to strongly misspecified (variance, say 25), and because one has easy access to the ground-truth analytical posterior). 
    In particular, it would good to first, empirically demonstrate that RNPE and NPE are equivalent in the well-specified case, and second, it would be good to show how the Monte Carlo (MC) approximation in the last step affects the results. For example, it would good to see (in the misspecified case) how as a function of the number of MC samples, the RNPE posterior samples approach the true posterior.
    - second, I think it is essential to use more evaluation metrics for all three tasks. On the Gaussian task with access to reference posteriors one could additionally show the error in posterior mean, in posterior variance, and the actual dispersion between true and approximated posteriors. For all three task one could show additional calibration metrics, as I believe the expected posterior coverage can be misleading in some scenarios (see questions below). One possibility would be simulation-based calibration (SBC, Talts et al. 2018), although I acknowledge that it might be computationally infeasible because the entire RNPE procedure would need to repeated ~100 for different $y$. Alternatively, local coverage tests around the observed $y$ could be performed using Zhao et al. 2021 ([https://proceedings.mlr.press/v161/zhao21b.html](https://proceedings.mlr.press/v161/zhao21b.html)). 
    Finally, I think it would be instructive to additionally compare posterior predictive samples of the different approaches, e.g., by sampling parameters from the posterior and then passing them through simulator and the noise model and compare “simulated” and “noised” predictive samples.
    
    The evaluation on three example tasks is informative. However, all three tasks are rather low-dimensional (or designed to be) in the data and the parameter domain. I am wondering how RNPE scales to higher-dimensional problems. NPE was shown to infer plausible posteriors in up to 30-dimensional parameters spaces (Goncalves et al. 2020, eLife) and large data spaces in the image domain using embedding nets (see NPE results in the GATSBI paper by Ramesh et al. 2021), but I fear that this will be challenging for RNPE for several reasons. RNPE requires a Monte Carlo approximation from NPE, which might become computational expensive in large parameter spaces. RNPE requires learning $p(x)$, which will quickly become challenging as $x$ is not reduced to low-dimensional summary statistics. 
    For the first problem I think it is feasible to design another evaluation task with more parameters, e.g., a high-dimensional Gaussian task, and to show Monte Carlo approximation behaves. For the second problem I see that it is challenging to extend RNPE to be used with embedding networks. But even more importantly, I think it is essential that the reader is informed about these limitations, which does not seem to be the case in the current version of the paper. 
    
- *Clarity*: 
The paper is clearly written and well organized. Limitations should be stated more clearly (see above). Nitpick: the numbers in 227 and 228 are formatted inconsistently.
- *Significance:* 
This is a significant contribution. RNPE appears conceptually solid and shows promising results. Once evaluated thoroughly, I believe it has the potential to be used both by SBI practitioners (depending on the availability of RNPE in an open-source package, e.g., $\texttt{sbi}$ Tejero-Cantero et al. 2020), and for future research on new SBI methods.
questions: 3. **Questions:** 
As explained in detail above, I think it is essential to evaluate RNPE performance on a fully tractable example like the Gaussian task, in order to answer questions like: 
    - Are RNPE and NPE empirically equivalent in the well-specified case?
    - How does slowly increasing the amount of misspecification change metrics like posterior mean error, variance error, dispersion, C2ST between the analytical and the NPE and RNPE posteriors?
    - How do the posterior predictive distributions of the true, NPE and RNPE posteriors look like in the misspecified case?
    - How does the Monte Carlo estimate in RNPE affect the RNPE vs the true posterior?
    
    Evaluation of additional metrics is especially important as in my impression, the expected coverage metric can be misleading. For example, in Figure 2b SIR it seems clear that NPE is massively overconfident, suggestion that it is too small posterior variance. But what is actually happening is that the NPE posterior is highly biased to other regions in the parameters space and actually has a very large variance. This behaviour would be immediately evident from a simulation-based calibration visualization of the ranks (Talts et al. 2018). I see that it might be challenging to perform SBC over $y$, but there might alternative metrics like the local coverage test (Zhao et al.). Alternatively, one could obtain reference posteriors for the SIR (Lueckmann et al. 2021) and then calculate metric between the posterior directly to evaluate RNPE vs NPE performance. 
    
    The results of the spike-and-slap error model are promising, however, I am wondering whether you investigated other error model, and if so, how did they compare?
    
    The denoised data distributions shown in Figure 4b and in the appendix tend to have quite funky shapes. Are there intuitive explanations why this is the case, e.g., in 4b, what is the reason for the bimodal shape, opposed to the unimodal shape of the simulated data? That makes me wonder, how was the training of the flow and successful approximation of $p(x)$ evaluated? 
    
    In the context of Figure 4c you point out that in contrast NPE, RNPE puts more posterior mass around the underlying ground-truth parameters. Here, I think it is important to note that in general posterior do not have be centered on ground-truth parameters, so in principle NPE could be correct here as well. One way to tell them apart would be checking the posterior predictive samples—how do they look like?
limitations: Limitations of RNPE to data and parameter space dimensionality of the SBI problem should be discussed in more detail (see above). Overall, I think this is an important and promising contribution that requires some additional evaluation and discussion of limitations.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors proposed an extension of a class of simulation-based inference using neural networks called "robust neural posterior estimation". They intorduced an additional inference layer to a simulation-based inference method "neural posterior estimation" by combining it with an "error model" that is a generative model of data point from a simulated sample. In this work, the error model was specified by a spike-and-slab distribution, which in turn provides an indicator "posterior misspecification probability" for degrees of misspecification of each specified data summary statistics. Their experiments compared "robust neural posterior estimation" and "neural posterior estimation".

strengths_and_weaknesses: - If I understand correctly, a core of the proposed idea is to introduce an error model $p(y \mid x)$, a model that generates $y$ from a simulated sample $x \sim \text{Simulator}_{\theta}$. Would this approach can be equivalent with defining a new simulator by plugging the original simulator into some condition probability distribution: $p(y \mid x = \operatorname{Simulator}_\theta )$? In that case, other general methods in SBI or ABC would become available to learn all the parameters of the new simulator?
- In this work, a spike-and-slab distribution is used for the error model $p(y \mid x)$. The choise of the distribution seems to implicitly specify what type of misspecification we assume to happen because the model $p(y \mid x)$ describe how datum $y$ is generated from a simulation $x$. In spike-and-slab case, it seems like at first glance that an implicitly assumed type of misspefication is less frequent outlier. The performance of robust neural posterior estimation seems to be dependent of the choice of the error model to a large degree. The spike-and-slab would be the only choice that posterior misspecification probability can be computed? Would it be possible to provide other  available choices and examine impacts by them?
- Given that above perspectives, if it is possible to apply other other general methods in SBI or ABC to learn the entire parameter of the simulator, there would be vairous other robust inference methods available such as one based on maximum mean descrepancy. If this is the case, it would be helpful to see or commnet on how efficient the proposed procedure is than others. In this respect, the reviewer was personally wondering if the broad name "robust" neural posterior estimation is throughly accurate/appropriate to represents the exact content of the proposal, given that there seems to exist many possible approaches to "robust" inference.
- The experiments seem to compare "robust neural posterior estimation" with "neural posterior estimation" only. It would be strongly helpful to compare it with other existing methodologies such as ones cited in Section 3.2 and to clarify when and how the robust neural posterior estimation has the advantage to others. The performances of robust neural posterior estimation and neural posterior estimation would also be dependent of the choice of neural networks to be used, where therefore it may also be helpful to justify that the used models were reasonable somewhere in the main paper.
questions: - To me personally, there seems several sentences whose meanings were not fully clear and accurate throughout the paper in general. For example, on the sentence '... probabilistically “denoising” the observation, thus producing samples consistent with both the error model and simulator' in the page 3, it wasn't clear what "denosing" means here and what "consistent with both the error model and simulator" means exactly. For other example, the sentence 'Despite this, the general approach in SBI is ... to query it with the observed data, q(θ | x = yo)' seemed too coarse description of  the procedure; what would you mean by "query with" for example? I understand some of intensions behind are to make explanation intuiteve but it would be helpful to refine inaccurate explanations so that they sounds more accurate and rigorous.
- It would be helpful to add an exact literature that coined the term "neural posterior estimation" if there is any.
- It would be helpful to add an exact literature that used the term “simulation-to-reality gap” in the page 2.
- The explanation of the algorithm procedure below (3) does not read well. For example, it may be helpful to write them as a certain type of well-organised lists.
limitations: There would be no negative societal impact. The clear limitation is that there was no comparison with other existing methodologies apart from neural posterior estimation. The significance of the proposal would have to be justified clearly by either or both extensive experimental evaluation or comprehensive theoretical analysis.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper addresses the issue of model misspecification in simulation-based inference (sbi). Simulation-based inference refers to the task of inferring parameters of interest $\theta$ based on an observation $y_o$ and a simulator. The simulator is based on scientific knowledge and allows to sample synthetic observations $x \sim p(x|\theta)$. Sbi assumes that those simulations match the real data generation process. However, in practice, this simulator is imperfect and there is a gap between simulation and reality. This work aims to resolve this gap by introducing an error model $y \sim p(y|x)$ explicitly modeling the gap and proposing a strategy to perform robust inference using both the simulator and the error model. There are two contributions presented in this work:
* Perform robust inference with a neural sbi method by leveraging the error model
* get explicit misspecification posterior probabilities from the error model.
strengths_and_weaknesses: * (+) The paper is overall well written
* (+) Model misspecification is a hot topic in simulation-based inference
* (+) Experiments are convincing

* (-) My main concerns about this work are about the problem statement. The authors proposed to use two models for the inference: the simulator $p(x|\theta)$ and the error model $p(y|x)$ which is assumed to be known. In my opinion, if the error model were to be known a priori, it could be included in the simulator, hence viewing $x$ as a latent variable and $y$ as the simulated observable. In this case, classical simulation-based inference algorithms could be used. The big challenge with model misspecification is that, in practice, part of those misspecifications are unknown. One can then leverage real observations or assume specific types of misspecification without having a proper distribution and leverage Generalized Bayesian inference to be robust to those types of misspecification. Note that the authors do say something about this at l212-l216:
> In contrast to more traditional SBI methods, few solutions have been proposed for performing robust inference with neural SBI methods.  Perhaps the most common approach is to augment the model with additional nuisance parameters (e.g. additive noise), to expand the range of values the simulator can reasonably produce (Cranmer et al., 2020). This again can generally be interpreted as a naïve method of incorporating an error model and does not facilitate model criticism.
>
However, it is not clear to me how combining the two components provides additional value regarding the two contributions presented in the paper.
   - Contribution 1: perform robust inference with a neural sbi method. What advantages does it have over the naive approach of including the error model in the simulator?
   - Contribution 2: get explicit posterior misspecification probability. Why do we need to combine the error model with the simulator? Couldn't the error model be used alone for this task?

questions: As explained in the strengths and weaknesses section, my main questions would be:
* How does combining posterior misspecification with the simulator helps to perform model criticism? 
* What advantages does it have over the naive approach of including the error model in the simulator for robust sbi?
limitations: I do not see any potential negative societal impact of this work.
review_title: null

Meta-review: metareview: The reviewers, based on their scores, are not in consensus about this paper. However, there has been an extensive amount of conversation between reviewers, amongst themselves and with the authors, who have made significant updates to the manuscript that have been appreciated by the reviewers. Two reviewers champion the paper, while another reviewer maintained his/her score based on the original version of the manuscript, upon reading the updated version, only voices a mild concern around significance. The final reviewer who voted negatively simply presents novelty concerns. 

After a careful read of the manuscript myself, I would like to break the vote towards acceptance. This is a topic that is often overlooked in this field and I think that the authors present an interesting approach and, perhaps more importantly, that this will lead to interesting discussion and further work in this field. 

I appreciate the authors' careful attention to revising their manuscript and thank the reviewers for their balanced approach to reviewing and discussing this paper. I hope that the authors address any remaining concerns in preparing the final version of their manuscript,
confidence: Less certain
Decision: Accept