Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper extends SNPE-C to the situation where multiple sets of parameters yield identical observations, but where the parameters can be split into local and global ones, which are specific to, or shared across all observations, respectively. It is shown how this additional information can be used to build a flow-based posterior estimator yielding sharper distributions.

main_review: The paper considers simulation-based inference in the presence of strongly coupled parameters leading to a non-injective likelihood function. This parameter estimation problem for such a system is ill-posed, but under additional assumptions this can be resolved with a hierarchical Bayesian model in which some parameters are shared across all observations.

The authors propose a practical realization of this solution in the form of a "h-flow" model, consisting of two normalizing flows modeling the posterior for the local and global parameters, and utilizing a deepset architecture for the global parameters to enforce permutation invariance. The network is optimized with KL divergence, and the authors show how to train it within the multi-round SNPE-C scheme.

h-flow is then experimentally tested on a simple model dependent on the product of two parameters, and on a 4d neural-mass model (NMM) of a cortical column with 3 local parameters and 1 global one. Performance is comparable to a naive normalizing flow in the simple case, but a significant improvement is seen for the NMM. To further show practical applicability, the authors then use h-flow to estimate NMM parameters on real EEG signals recorded under eyes open/closed conditions, and show that this results in separated distributions for the two conditions, in line with expectations.

The paper is clearly written, provides simple and illustrative motivating examples, and keeps the most relevant information in the main text. The proposed method is practically useful, and easy to implement, which should lead to adoption among practitioners already using SNPE-C or similar SBI methods. The experimental results could be made a bit more convincing with additional models or more baseline methods in the case of the NMM.

----
Score revised +1 based on the authors' responses and the promise to report improved baselines in the NMM test case.
limitations_and_societal_impact: No concerns seen here.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a new likelihood-free inference method for Bayesian hierarchical models, called h-Flow, that exploits additional information provided by a set of auxiliary observations with shared, global parameters. h-Flow works by approximating two relevant posterior distributions, i.e. that of the global parameter beta and the local parameters alpha given beta, with separate normalizing flows, which are then multiplied to yield the joint posterior distribution. The normalizing flows are trained by minimising the KL-divergence between the true and approximated posterior distribution and the training data is refined over several rounds by means of SNPE-C (also known as APT).
main_review: The authors provide an approach to likelihood-free inference, which is a known and important task, for Bayesian hierarchical models. While the method is new, to me it appears to be a slight variation of SNPE-C that includes a second normalizing flow for global parameters that is conditioned on a set of auxiliary variables. I am thus mildly worried about novelty and significance. I would encourage the authors to emphasise their contributions and the difference to related work further.

The derivations are generally clear and technically sound. While the examples in their experiments are very interesting, I don't think they highlight the method very well. It is still unclear to me whether or not any improvements are due to the proposed method, or a) an increased flexibility or b) simply having more auxiliary observations that are correlated with new observations. The authors attempt to clarify this by using a naive approach that ignores the hierarchical structure, but this only consists of one normalizing flow with a lower capacity than the two flows of h-Flow. Similarly, as the authors said, N>0 generally yields better posteriors but this is naturally not very surprising. In my opinion, a fair baseline for e.g. the EEG experiments would have been a comparison of N=0 vs N=9 with both naive architectures and h-Flow, but a setting where both approaches have the same capacity.

The submission is generally clearly-written, at least until Section 3. I believe the experimental section could have been more concise with some explanations about the models and approaches being moved to the appendix, leaving space for, perhaps, one more experiment or more elaborate baseline comparisons.
limitations_and_societal_impact: The authors did not comment on limitations of their method and the potential societal impact it may have.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an approach to perform statistical inference for hierarchical models with intractable likelihoods. It extends an existing approach for simulation-based inference (SNPE-C) in order to use models that have global and local random variables. 

The authors propose a scheme in which two conditional flows a trained: The first flow models a distribution over global parameters given an observation and $N$ permutation invariant auxiliary observations. Permutation invariance could e.g. be accounted for with a deepset architecture. The experiments do not make use of this flexibility, relying on an average operation instead. The second flow then models a distribution over local parameters given the observation and global parameters. They refer to the resulting inference scheme jointly as h-Flow, for hierarchical flow.

A motivating toy example is provided, along with a relevant, more realistic simulator from Neuroscience (a Neural Mass Model; NMM). h-Flow is compared against a naive approach on both of these examples. The comparison turns out about equal on the toy example and favorably for h-Flow on the NMM. On the toy example, h-Flow is additionally compared against a hierarchical ABC variant and LFVI, comparing favorably to both of them.
main_review: # Main Review

## Significance and Originality

Overall, the paper is well motivated and introduces a strategy which can potentially be useful for many applications of SBI. The proposed solution might come at no big surprise, but in fact this can be viewed as a strength: As long as the proposed strategy of training two flows (see summary above) performs robustly and demonstrably better than what one would naively do with SNPE-C, and given that it outperforms existing methods, I would view this paper as being a valuable addition to the conference that should be presented. 


## Clarity

The paper is generally well-written and easy to follow. I especially enjoyed that the authors introduced a motivating example that then finds its structure reflected in the applied setting.


## Quality

My reasons for not giving a higher score, and currently tending to rather reject the paper, are many open questions and some specific requests I have regarding the comparisons. I detail these concerns below and hope that the authors will address them, in which case I will revise my evaluation accordingly.

### Comparisons to Naive Approaches

My primary concern is that the paper does not yet clearly demonstrate that the proposed method is better than naive approaches using SNPE-C with a single flow (instead of using two flows, as h-Flow does). In the toy experiment, no difference between h-Flow and the naive approach is found (Figure 2). On the NMM, a difference is clearly visible in Figure 3, but as of now I am unsure how to interpret this result exactly and how it was obtained. Some specific questions and requests that can hopefully help clarify this issue follow.

**Q1**: In Figure 3, the difference between the naive approach and h-Flow _is biggest_ for $N=0$. I was surprised by this, since my understanding is that we expect improvements in resolving indeterminacies by additional observations, i.e. when $N>0$. In the toy example, which served as a motivation for the NMM, as $N \rightarrow 0$ the posterior distribution converges to $p(\alpha, \beta | x_0)$. In other words, for the toy example, we do not expect a difference (and empirically don't find) a difference in performance when $N=0$. Could the authors comment on the difference in performance for $N=0$ on the NMM? 

**Q2**: Regarding the results presented in Figure 3 the authors write: "We see that the factorization scheme yields uniformly better results than the naive architecture. This is mainly because the dimensionality of the context variables in the naive case gets too big when more observations become available, making the training of the normalizing flow in the non-factorized setting harder.“. Why exactly do the context variables in the naive case get too big with more observations? I would have thought that for the naive scheme we simply aggregate the additional observations before passing them as context to the single flow. If we do not aggregate them, we assume that the ordering matters, which is would put the naive implementation at a big disadvantage and is not how one would naively apply SNPE-C with a single flow in this context. It would be important to comment and clarify this.

**R1**: As a request for new experiments: The results in Figure 3 are reported in terms of Wasserstein distance between a dirac delta on ground truth parameters and the inferred posterior (median without error bars for 5 repeats). It would be important to extend this analysis for a much larger number of ground truth parameters and report results such that the variability can be evaluated (e.g. mean and standard errors). Ideally, a secondary metric to the Wasserstein distance was reported in the supplement, such as the more commonly used negative log probability of true parameters _averaged_ over as many ground truth parameters as feasible (this is crucial, otherwise this metric would be hard to interpret).

### Results for h-ABC and LFVI on NMM

**R2**: As a second request, it would be good to provide results for h-ABC and LFVI on the NMM model for completeness.


## Additional questions

**Q3**: In Figure 1 (right) we observe that the $N=10$ case is worse relative to $N=0$. To me this seemed counterintuitive at first and I think it would be good if the authors commented on it in the manuscript. My interpretation is that the density estimation problem gets more difficult when we have more observations (posterior gets more concentrated). So resolving indeterminacies leads to less accurate inference (as compared to the analytic solution). Would the authors agree with this interpretation?

**Q4**: "We did not carry out more experiments on this data-driven setting because of difficulties due to numerical instabilities in the training procedure when N increases and for certain choices of ground truth parameters.". It would be important to investigate this issue further, if possible for the current publication. Why would this only occur for certain ground truth parameters and not for others? 

**Q5**: For the toy example the authors set $\sigma=0$, so that $\epsilon = 0$. Usually SBI is only applied in the context of stochastic simulators. Would results change in a meaningful way if the experiments were repeated for a small value of $\sigma$?

**Q6**: The NSF in Figure 1 does not seem able to handle the sharp transitions that occur when $N=10$ or $N=100$. Do the authors have any suggestions to remedy this? 

**Q7**: On the NMM, MAFs instead of the more expressive NSFs are used (whereas the toy example does use NSFs). What are the reasons behind this choice and how do results change for h-Flow and the naive strategy when using them instead?

**Q8**: "Although the number of additional observations (N) was fixed in our analysis and experiments, this parameter could be randomized and amortized during learning and enable the posterior approximation to be fed with sets of auxiliary observations of varying sizes making it more flexible for applications.“ $\rightarrow$ My understanding is that the aggregation is simply done through the average in the experiments. What exactly would not work/require changes to have flexible N?

**Q9**: Have the authors done any internal experiments in which $\phi_3$ was learned as well? If so, were there any problems that they encountered that restricted the experiments to learning $\phi_1$ and $\phi_2$ only?


## Name of the algorithm

I see two potential disadvantages associated with with naming the algorithm h-Flow. They are:
1. One of the strengths of SNPE is that it can be used with any conditional density estimator. In the future, perhaps entirely different classes of CDEs will be invented. I might be misunderstanding something (please correct me if I'm wrong) but to me it seems there is nothing flow-specific about the proposed algorithm. In particular I assume that we can use it with all kinds of CDEs as long as they work with embeddings/can be designed to be permutation invariant and we can evaluate log probabilities and draw samples. For example, we could just as well use this with MDNs, which might be more suitable depending on the application/knowledge we have about a problem.
2. h-Flow in the paper title sounded to me as if a single architecture was trained, i.e. that this paper was about an entirely novel flow architecture. Instead, two flows are trained, and if the approach gets extended to multi-level architectures, training would involve more and more flows, not a single one. 

**Q10**: What are the authors thoughts on these remarks regarding their algorithm's name?

## Minor

- Fig. 3: There are display issues with this figure when opening the PDF using Preview on macOS. Most marginals are missing and the x-axis of the inset figure is cut off (3 instead of 30)
- L11: validate quantitatively our proposal $\rightarrow$ validate our proposal quantitatively
- L103: have to be approximated $\rightarrow$ we decide to approximate them
- L178: $5 \times 10^{−4}$ rather than $5.10^{−4}$?
- L227: captures well the global parameter $\rightarrow$ captures the global parameter well 
- L229: could well estimate local parameters $\rightarrow$ could estimate local parameters well

## Update

I thank the authors for their reply and have updated my score.
limitations_and_societal_impact: The authors state in their checklist that potential negative societal impacts of their work are not applicable. I agree that an extended discussion of negative impacts might not be strictly necessary, given the nature of this work is about statistical inference applied to a certain class of models (with hierarchical structure and intractable likelihoods). Thinking more broadly, the societal impact and potentially adverse effects will depend on the setting in which the algorithm gets deployed. However, I would suggest to at least briefly touch upon the general difficulties associated with diagnostics/criticism in the SBI context.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses the inference of illposed models, where the likelihood is intractable, and proposes to utilize hierarchical structures in combination with neural density estimators (normalizing flows) to ameliorate identifiability of the parameters. It motivates the approach with a simple analytical tractable example, and validates the approach on modelled and real EEG time-series.
main_review: The problem the authors consider is interesting and important for scenarios, where models are parametrized ambiguously. However, I consider the present work as too immature for publication, because to me, the proposed approach (making use of additional observations and separating the parameters in global and local ones) is in my opinion not sufficiently justified explored. Some questions I would like to see answered are
+ When is the separation off variables into local and global possible or desirable. E.g. when you remove the assumption that alpha and beta are upper bounded by 1 in the motivating example, the separation wouldn’t help any more in my opinion. I do not say, that the motivating example isn’t interesting, but I would like to have seen more discussion, on when it makes sense to have this separation. Hence, I would appreciate some general discussion, when such the proposed separation makes sense.
+ The problems, that are considered, are very low dimensional, which is making me wonder whether the neural density estimators are really necessary here. Just demonstrating that the separation ameliorates the inference by using Gaussians (or mixtures of them, or other simple estimators) for the density approximation in Eq. (5), might be more insightful, and easier to train.
+ Overall, I was missing details about how many parameters are used in the different models, how long it takes them to train, etc. So I couldn’t make myself a picture, whether comparisons are fair or not.
+ For the fact, that the paper doesn’t provide any theory (except for a specific toy example), how this separation is helping model inference, I think the empirical results are too little, to consider the proposed method as helpful in general.

### Update

Reading the comments of the other reviewers, and the response of the authors, I am reconsidered my score and raised it to 6 (though decreasing my confidence score). This is because,

+ Reading the reviewers' comments, it seems I underestimated the contribution of the paper, leveraging information of the hierarchical structure of models in the context of likelihood-free inference.
+ I mentioned originally, that I found the manuscript written too complicated and mentioned as examples invariance of the averaging operation to permutations, and that $\phi_3$ is introduced, but seems to be never learnt. I understand for the first point, that though trivial, it is a crucial point for the construction of the neural posterior, so it makes sense to give the fact more space than needed on first sight. The fact, that $\phi_3$ is not needed in the shown experiments (as I got from the response to reviewer fnEF), still poses the question, whether it is necessary to introduce it, or just to mention the possibility of having, but simplify notation.
+ Experiments seem to be computational heavy to provide new baselines for the NMM example, so it is maybe too much to ask.
limitations_and_societal_impact: + As mentioned above, I’d like to see some additional and rigour discussion on when such separation of variables helps the inference.
+ Please report number of parameters of models, what device was used to train them etc. to allow the reader to evaluate the showed results.
+ In general, more experiments, that demonstrate the improvement achieved by the proposed method 
+ I sometimes feel, the paper is written too complicated. For example, in l134 it is written that $f_{\phi^{(3)}}$ is “invariant to the ordering of the input”, which is obvious, since the function in question is a sum over inputs (which is commutative). I thought quite a bit about this comment, and I am not sure whether I am not seeing something, or why it is important to mention here. Another example is, that $f_{\phi^{(3)}}$ seems never to be used in the experiments, which makes me wonder, why is it introduced in the first place.
+ Some typos:
++ L129f: indices of phi are mixed
++ There are several approximation signs in the appendix that should be equal, and some equal signs that should be approximations.
++ L107: It should be \beta not \beta_0
review_title: null

Meta-review: Dear authors, 

congratulations on your paper being accepted at Neurips. As you know, the reviewers gave extensive comments and feedback on the submission, and we urge you to ensure that this feedback is incorporated into the final version of the article. In particular, reviewers felt that the name of the method, 'h-flow', is misleading, as the paper does not present a new flow, but rather a new SBI method (which will likely, but not necessarily, be applied to flows). We would ask you to take this advice seriously. 

With best regards and congratulations, your AC
Decision: Accept (Poster)