Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper develops an efficient feature selection method in simulation-based inference, where each evaluation of the likelihood is costly. Conventionally, to test the sensitivities of the posterior distribution to specific features, one would need to re-evaluate the posterior with different combinations of features many times. The key idea of the method is to estimate the likelihood as an intermediate quantity instead of directly estimating the posterior, and analytically marginalize the joint likelihood for each feature instead of re-evaluating from scratch each time. The marginalization is made possible by the use of mixed density networks with gaussian properties where different parameters are easily partitioned. Using a Hodgkin-Huxley model as an example, the paper demonstrates how this method can be used for selecting features at a much smaller computational cost.
strengths_and_weaknesses: This work seems to be a small but useful technical improvement to simulation-based inference (SBI). I am not an expert of the SBI method space in particular, so I cannot speak much on the originality or the novelty of the idea. But I have seen usages of SBI and surrogate likelihood/posterior approaches in neuroscience as well as in other domains (computational chemistry), and I have not seen an analytical marginalization approach like this before. Whenever the mixed gaussian model (with the convenient partitioning properties that allow marginalization) is a good fit to the system, this method will provide an immediate practical advantage. The paper is mostly clear and well written.

questions: i) It would be nice to have a better "chance level" curve for the random selection in Fig 4a, i.e., average over multiple random sequences of features.

ii) The speedup by the proposed method is evident, but 20 hours is still a long time if one wants to repeat. Can the authors comment on how/where the algorithm can use parallelization to reduce computation time?

iii) I had two questions, and they both boil down to understanding the computation time cost of considering more features:

- There is still a human curated step required in the procedure, namely the pre-selected feature list. Can the authors comment on how this new level of efficiency could change the constraints related to the choice of the pre-selected feature list? That is, can one now run through a very long list of candidate features, including all the silly things that one would not expect to be important a priori?
- Regarding the HH model experiment: I agree that the greedy features selection is a completely reasonable thing to do. But looking at Fig 4a made me wonder if This method also provides easy ways to explore combinations of features. (Specifically, from Fig 4a, I was wondering if the combination of APT, AHP, and APW were important, based on the drop in the blue curve at the addition of APT, and the drop in the black curve at the addition of APW.) How much additional cost would it be to investigate combinations of features?
limitations: The authors have adequately addressed the limitations of their methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposed a method to determine the "informativeness" of data features by comparing the width of posteriors with some data features marginalized out. The authors compare a naive solution where many models are trained to their proposed technique which takes advantage of the properties of Mixture Density Networks (DMN) as the architecture for Neural Likelihood Estimation (also known as synthetic likelihood) to trivially marginalize over data features without retraining.

Their method Feature Selection Through Likelihood Marginalization (FSLM):
- allows for closed-form marginalization of data features, thereby enabling access to posteriors which are conditioned on arbitrarily marginalized data random variables without retraining.
- comes with a greedy feature selection strategy to select useful features from a predefined set.
- is compared to ground truth posteriors from simple linear Gaussian model and against a non-linear neuronal model. 
strengths_and_weaknesses: ## Originality

### Strengths
- The closed-form marginalization is simple and easily seen as a result of using a MDN. I've not seen it used before and I find it interesting for their purposes.

### Weaknesses
- The paper does not reference the work of Benjamin Wandelt and coauthors which aims to transform simulation data into a compressed format which maximizes the information shared by the data and its compressed representation for Neural Likelihood Estimation. This is not strictly the same subject as Wandelt's work does not quantify the value of specific dimensions of input data; however, it is a highly relevant dimension reduction technique. Please consider citing all of the papers on the [PyDelfi](https://pydelfi.readthedocs.io/en/latest/intro.html) website (there are three).
- There is other tangentially related work regarding marginalizing over parameters. In particular, [Truncated Marginal Neural Ratio Estimation](https://arxiv.org/abs/2107.01214), [Arbitrary Marginal Neural Ratio Estimation for Simulation-based Inference](https://arxiv.org/abs/2110.00449), and (from pydelfi) [Nuisance hardened data compression for fast likelihood-free inference](https://arxiv.org/abs/1903.01473). Perhaps it would be interesting to include for the purposes of dimensionality reduction.

## Quality
### Strengths
- The Gaussian linear model example is chosen well to show the relevance of independence between parameters and data and the corresponding effect on the estimated posterior.
- The posterior visualization, tables which indicate computational cost and estimated KL divergence, and search result visualizations are all quite clear.
- The limitations of using MDNs are made clear and the benefit for feature search is as well.

### Weaknesses
- Although the method seems nice, I still don't understand why the practitioner would not instead want to compute the mutual information. We do not know what sort of observations we will have in the future and the mutual information is likely more robust to future data (rather than overfitting feature selection to x_o.)

## Clarity
- Clarity was satisfactory in my opinion.


## Significance
### Strengths
- They're certainly a nice feature of MDNs and NLE.

### Weaknesses
- MDNs seem to be of limited value in the times of normalizing flows, etc.
- I also don't see why setting the feature importance based off a single x_o is really so interesting. Selected features should have some sort of predictive power for future data, right? If so, shouldn't we then integrate over all possible data, i.e. estimate the mutual information between different feature sets and the parameters?

questions: - Why is this method superior to a method which estimates the mutual information between parameters and data with some data features marginalized out?

limitations: - I think this was also made clear.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new method for determining informative features in simulation based inference. Recently proposed simulation based inference methods proceed by (1) generating parameter samples from the prior, (2) simulating to generate conditional samples of the data given the parameter, (3) fitting an approximation to the likelihood of data features under the simulator model consisting of a mixture model parameterized by neural networks and (4) sampling from the posterior over parameters given the data, using the conditional mixture model as an approximation to the true likelihood. The authors observe that to obtain a posterior conditional on a subset of features, one can use the property of multivariate normals that their marginals are analytically tractable, and replace the full approximate likelihood with the marginal approximate likelihood.  
strengths_and_weaknesses: The method itself is very straightforward, enough so that it seems like a trick any researcher in the field would be able to figure out, as it relies on well-known properties of multivariate normal distributions. However, the method is presented very clearly. Identifying informative features in simulation based inference is an important and poorly studied problem (to my knowledge); this paper offers smart and clear problem statement and setup. Moreover, writing a paper about the method helps motivate the particular advantages of neural likelihood estimation in contrast to competing techniques (such as NPE). 

The evaluations are clear and compelling, especially the biological application. Hodgkin Huxley models are notoriously hard to fit, and the connection between the features that electrophysiologists focus on and actual underlying ion channel mechanisms are notoriously difficult to understand. This paper thus offers a compelling advance for the computational neuroscience/electrophysiology field in particular.


questions: There are some serious typos and lack of clarity in Eqn. 6 - there's min_j, but j does not appear; lowercase m and n are undefined; and it's unclear what min^N_j means (min raised to the Nth power??).

More generally I'm confused by the motivation for the KL estimator. KL estimation from paired samples is notoriously difficult and unreliable in high dimensions. I think it would make more sense to use maximum mean discrepancy or related techniques, which are well-established for moderately high dimensional data, and  which offer the further advantage of providing a witness function that can help determine where the two distributions differ. (see https://jmlr.csail.mit.edu/papers/v13/gretton12a.html and https://papers.nips.cc/paper/2015/hash/0fcbc61acd0479dc77e3cccc0f5ffca7-Abstract.html)




limitations: The authors discuss some limitations. One additional limitation that I think is important is that the success of NLE in general depends crucially on the choice of prior; if the true parameters are far out in the tails of the prior, the likelihood approximation is likely to be very poor, particularly as it restricts the tails of the likelihood to be Gaussian (and will thus fail if the true likelihood has, for instance, very fat tails). I worry that the proposed method does not take into account uncertainty or systematic errors in the approximation quality
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a method to efficiently estimate the importance of several features / outcome variables of a simulation model in a Bayesian setting. The method is based on mixture density networks and the fact that marginalization of Gaussian distributions is analytically tractable and relatively simple.
The authors use their method to demonstrate in 2 practical examples that the method reliable identifies feature importance and demonstrate the method's superior speed. 
strengths_and_weaknesses: __Strengths__

- Reasoning about the importance of _interpretable_ summary statistics is commonly (and unjustly) ignored in the era of neural networks. The authors therefore tackle an issue that is highly relevant for reliable and trustworthy inference. 
- Marginalisation using the MDN is indeed an efficient approach and using the so computed marginal distributions to compute the feature important makes sense to me.
- The method is easy to understand and implement
- The writing is coherent and what is said is easy to follow (but the text does lack crucial information, see below)

__Weaknesses__

- The paper seems rushed: there is very little information on many relevant details. For example, what kind of density estimator was used for NLE? A flow? If so, what kind of flow? How many layers? I have many questions, see below. If any of those are answered in the text and I missed them, could the authors please let me know?
- The paper contains only 2 examples, one of which is just a simple Gaussian. I understand the value of the Gaussian example, but the experiments section then only has one "real" model, which puts this section on the lighter side. (Perhaps the relatively long description of the HH model could have gone to the appendix in favour of another example.)
- The literature review is a bit biased towards more recent developments and perhaps a bit unfair towards earlier contributions. For example:

> likelihood-based Bayesian methods such as variational inference [8] or Markov Chain Monte Carlo [9] cannot be used.

That's a bit imprecise. Of course, MCMC can be used, in the form of ABC-MCMC [1] or pseudo-marginal methods [2]. The argument against those methods is more the computational cost and potentially slow convergence, not however, that it's impossible.

- Equation 11 in the appendix is confusing. The RHS has a variable $valid$ which the LHS does not have. The reference for proofs is to imprecise, I could not find the relevant part. The reference should point to the theorem that establishes the result. 

[1] Marjoram, P., Molitor, J., Plagnol, V., & Tavaré, S. (2003). Markov chain Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences, 100(26), 15324-15328.

[2] Andrieu, C., & Roberts, G. O. (2009). The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics, 37(2), 697-725.

questions: - What kind of density estimator was used for NLE? Just an MDN? Why not a flow? Have alternatives different from the default be tried? This should be mentioned. (Could any of those choices reasonable lead to NLE working better than your method, e.g. a flow instead of an MDN.)
- How is time for training compared? Is there a predefined number of iterations or are you using a stopping criterion? Or is this just to say that you train one MDN and marginalize instead of retraining an MDN on the subset? (I.e. if you used a flow for NLE it's not clear how to compare time.)
- Are you modelling the MDN with a full covariance matrix? If so, what would you suggest for higher dimensional data spaces?

limitations: The authors discuss some limitations in the discussion section. While there are some elements on the limitations of MDNs, I'd like to know more about the covariance matrix that has to be estimated and hence the scalability. (A reasonable argument to be made would be that scalability isn't that important since handcrafted summary statistics are usually of limited dimensionality, but I think this should be discussed.)
review_title: null

Meta-review: metareview: The paper presents a method for feature selection in simulation-based inference, that is, for quantifying to what extent each feature (or summary statistic) contributes to reducing posterior uncertainty. As this can be accomplished naively by re-estimating the posterior after systematically omitting features, the focus is on developing a fast method instead. The proposed method is evaluated on parameter estimation of a Hodgkin-Huxley neuron model from neuroscience.

The reviewers found the paper to be clearly written and did not voice major concerns regarding its technical quality. The proposed method is simple, efficient and clearly presented, and the evaluation on the Hodgkin-Huxley model is convincing. A potential drawback of the method is that it requires a model that can be analytically marginalized over, and thus may not benefit from current and future advances in generative modelling.

Seeing as the paper is of good quality without major problems, I'm happy to recommend acceptance.
confidence: Certain
Decision: Accept