Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose two new synthetic likelihood methods for simulation based inference using energy based models, UNLE and AUNLE.

They introduces a tilting trick and an amortized sequential model to perform posterior inference with intractable likelihoods and improve both modeling quality and inference budget.
strength_and_weaknesses: Stregths:
- the authors propose an interesting "tilted" inference procedure which performs well and is a neat trick
- the utilization of conditional EBMs in SBI makes sense and is well executed
- the authors propose a useful sequential version of their system

Weaknesses:
- like in many papers on this topic, the experiments are quite toy and not as ambitious as they could be, but they are clean and well-executed.
clarity,_quality,_novelty_and_reproducibility: The paper is very well written, has clarity on the details of the techniques and makes sense.
It also nicely ties into prior work on SBI and positions the paper fairly.
The main contribution is also fairly interesting as it reduces the computational budget for SBI, which is a core goal in the field.
summary_of_the_review: The authors propose a rigorous and interetsting spin on SBI using EBMs.

The proposed extensions make sense and should be useful in the field, and the experiments appear well justified -if comparatively simple.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper try to solve the Simulation-Based Inference (SBI) task using the energy-based model (EBM). In SBI task, one assume a known simulator that can generate samples x from interesting distribution parameterized by $\theta$. The task is then estimate $p(\theta | x)$. The authors using EBM to parameterize $q_\phi(x|\theta) $ or $q_\phi(x, \theta)$ and then inference can be done using bayes rule. In the paper, the authors propose two algorithms. AUNLE use a parameterization with tilting trick to make the normalizing constant of EBM independent of $\theta$ and thus reduce the computational cost in the previous SMNLE method. And AUNLE replaces the normalizing flow model used in previous method with EBM. Experimental results seem to support that the proposed models outperform its baselines.
strength_and_weaknesses: I mainly work on EBM instead of SBI. So my opinion my focus more on EBM part.

For the shining point of this paper, I think I like idea of this paper of using EBM to model the joint or conditional distribution of the paper. On one hand, EBM can be more expressive than normalizing flow given similar number of parameters and on the other hand, it makes sense to me that by cleverly design the EBM term, one can avoid the approximate doubly intractable estimation needed in previous method.

However, I also have some questions.
1. As pointed out by [2] and [4], EBM trained with MCMC as a sampler and MLE as objective may have the "short-run" effect. That is, instead of learning a correct energy function, the model converges to a point that the Langevin sampler becomes a noise injection generator. Since in the task of SBI, one actually needs a correct energy term to infer the likelihood. I'm wondering whether this problem exists in the current training process, especially for those complex tasks.  

2. The authors mentioned that their model can be more effective comparing with previous methods. Thus, I think it is necessary to add a complexity analysis part for both AUNLE and SMNLE, e.g. list different models' (both the new models and baselines) accuracies together with their training/testing/inference time and make a comparison.

3. I'm not very sure whether the SBI experiments results listed here are strong enough to make conlusion that the new proposed method outperforms the previous ones. Would like to listen to other reviewer's opinions.

4. Some important EBM literatures are missing. Recommend the authors to add the following [1] [2] [4] papers.

[1]  A Theory of Generative ConvNet. ICML 2016

[2] Learning non-convergent nonpersistent short-run mcmc toward energy-based model.  NeurIPS, 2019

[3] Cooperative Training of Descriptor and Generator Networks. PAMI, 2020

[4] A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model. ICLR 2022
clarity,_quality,_novelty_and_reproducibility: The paper is in general well written and easy to follow but since in the paper there are many algorithms with abbreviation. I do recommend the authors to try to make these more clear.
summary_of_the_review: In general, I like the idea of this paper of using EBM to solve the Simulation Based Inference (SBI) problem. And I think the theory part is OK for me.  But since I'm not an expert in SBI, I would like to listen other reviewers' opinion on how solid the experiments are.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces (adapts) ML training of EBMs for SBI problems.
The main problem in SBI is that one cares about sampling from posterior $q(\theta|x)$, and in the case of using EBMs, sampling from posterior results in doubly intractable inference since the partition function depends on $\theta$ as well. 
The paper suggest parameterizing the joint model $q_\psi(\theta, x)$ as $\pi(\theta)\exp(-E\psi(\theta, x))/Z_\psi$ and show that the optimal posterior also have similar form:  $q(\theta | x) = p(\theta)\exp(-E\psi(\theta, x))/Z_\psi$. This is important since they now can directly sample from the posterior without worrying about the partition function. The main limitation of this approach is that they need to have an analytical form p(\theta) as prior, which is not always available, especially for the sequential setup, in which the prior is defined using posteriors on the previous rounds. Therefore, the sequential version of their approach (SUNLE) just trains an EBM using contrastive divergence and uses doubly-intractable MCMC for inference.

The authors compare AUNLE and SUNLE with NLE and SNLE, which use normalizing flow, on four toy datasets and one real-world example.

strength_and_weaknesses: AUNLE is well-motivated, the approach is interesting and novel, and its correctness has been backed up by Proposition 1.
However, SUNLE is simply training EBMs with CD, I am not seeing any novelty there.

I don't understand why the authors didn't use SMNLE as a baseline. They ruled out SMNLE for being double-intractable, but they are already using double-intractable MCMC for their real-world setup. The comparison in Figure 1 is not enough to rule out SMNLE from the baselines. Moreover, SMNLE shows a better likelihood ratio than NLE in Figure 1.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written easy to follow and partially novel (AUNLE)
summary_of_the_review: The paper is well-motivated and discusses an important application. However, the main novelty of the paper is the introduction to AUNLE, however, the amortized version is not very useful in practice since using sequential modeling reduces the number of required samples. 
I am not seeing any novelty in the sequential version of their work except only training the EBM using CD in their setup.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method for learning synthetic likelihoods in the likelihood-free setting of simulation-based-inference (SBI).  The method fits a particular tilted version of the joint distribution of the parameters and observations.  Under certain assumptions, the solution of this optimization problem is proportional to the conditional density of the data given the parameters.  As such, the authors then use this unnormalized conditional density to estimate posteriors using standard approaches (e.g., MCMC).  The authors also present a sequential version of this approach.  The authors apply these methods to 4 toy datasets and compare to other SBI posterior inference methods, where they obtain better performance on one of the toy problems.  Finally, they apply their method to a neuroscience dataset, where they can obtain a good approximate posterior using fewer simulations than a previous method.
strength_and_weaknesses: Strengths
* The proposed approach is straightforward, conceptually neat, and very interesting.
* The paper is well-written and the method and results and presented exceptionally clearly.
* I appreciate that in the toy models the strengths _and_ weaknesses of the present approach relative to existing methods (e.g., normalizing flows) are presented.  I think that the gains of the present approach on complex posteriors are sufficiently interesting (and well-explained and well-motivated) to more than outweigh the comparable (or slightly worse) performance to existing methods on simpler toy problems.

Weaknesses:
* The key assumption of AUNLE is $(Z, \theta)$-uniformization.  This allows one to interpret the learned energy model as a likelihood up to an unknown normalizing constant that is independent of $\theta$.  If this uniformization does not hold, then the normalizing constant will depend on $\theta$, resulting a double-intractable problem.  Given that uniformization only holds if 1) the energy model is sufficiently flexible such that the true joint distribution lies within the space of learnable models; and 2) the optimization problem can be exactly, globally solved, it seems like in practice uniformization will always be violated.  Such assumptions are frequently made in SBI (e.g., it is common to plug approximate posteriors into various formulas involving the true posterior) so I don't think that this is deeply problematic (and indeed such assumptions are often justified by good empirical performance), but I would be interested to see some empirical results about how much uniformization is violated in practice and how much that affects downstream inference.  One possibility would be to run double-intractable methods on the learned energy model (without assuming that the normalization is independent of $\theta$) for an extremely long time to obtain a "ground truth" and compare to the output of AUNLE.  Another possibility would be to estimate different normalizing constants for the implied likelihoods for different values of $\theta$, and see the extent to which they differ.  In any case, I think it should be made more explicit in the main text that this uniformization is unlikely to _exactly_ hold in practice, and the rest of the AUNLE methodology hinges on it holding  _approximately_.
* For all results it would be good to show some degree of replicability across initializations and random seeds.  How different are the different methods relative to differences in neural network initialization and randomness in the simulations?  Are the exact same simulations used to train the different methods, or are different random sets used?  E.g., the difference between using MALA and SMC within AUNLE results in comparable differences on SLCP as compared to the difference between AUNLE and NLE.

Typos:
* "making naive Bayesian inference impossible" --> "making standard Bayesian inference impossible"
* "each of which endowed with" --> "each of which is endowed with"
* Species names are never capitalized: "_Cancer Borealis_" --> "_Cancer borealis_"
* "given an observed an neuronal recording" --> "given an observed neuronal recording"
* "We show that using this new methods can" --> "We show that using these new methods can"
* "does not suffer from the bias of incurred by the" --> "does not suffer from the bias incurred by the"
* "the number of steps (or intermediate distributions) is SMC is beneficial" --> " the number of steps (or intermediate distributions) in SMC is beneficial"
* "When applying SMC to within AUNLE’s training" --> "When applying SMC within AUNLE’s training"
* "using more SMC samplers steps usually increase the quality" --> "using more SMC samplers steps usually increasea the quality"
* "comparing multirond sequential methods" --> "comparing multiround sequential methods"
* "posterior estimate on the _C. Borealis_ simulator" --> "posterior estimate on the _C. borealis_ simulator"
clarity,_quality,_novelty_and_reproducibility: The approach proposed in this paper is, to my knowledge, novel.  It is interesting, and clearly presented.  The authors have code for reproducing results available at an anonymous github repo.  I did not run their code, but it looks thoroughly documented.
summary_of_the_review: Overall, I think this is a well-written interesting paper providing a promising approach for simulation-based-inference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes new methods for Simulation Based Inference. The goal is to perform Bayesian inference on model parameters given data for situations where the likelihood function is not known in closed form and can only be accessed through a simulation mechanism. The central challenge of SBI learning is "doubly intractibility" because the normalizer of the posterior distribution depends on the model parameter. This work proposes two energy-based methods for SBI. The first bypasses the doubly intractibility by defining a joint EBM over $X$ and $\theta$ so that the normalizer does not depend on $\theta$. This allows amortized posterior sampling for any observed data. To tailor the posterior inference to a specific observed datapoint, the work further proposes to use an EBM to learn the conditional distribution and performs doubly intractible inference. Results show competitive performance with score based SBI models for toy datasets and a crab biology dataset.
strength_and_weaknesses: STRENGTHS

1. The AUNLE formulation provides a nice way of dealing with a major limitation of double intractibility by tilting the normalizer. There are clear methodological and computational benefits to this approach, but it still remains limited in its ability to fine-tune the learned posterior to a specific observed datapoint.
2. Experimental results show that AUNLE and SUNLE can outperform score-based SBI for multimodal datasets, and that SUNLE can outperform score-based SBI for complex biological data.

WEAKNESSES

1. The presentation of equations and algorithms is not always clear. In Algorithm 2, should the dataset be $(X_i, \theta_i)$ instead of just $(X_i)$? In Equation 5 and Algorithm 4 is the data variable in the intractible term meant to have the index on the $X_i$, or should it be $X$ like in Algorithm 2, since it is a generated sample and not a data sample?
2. The novelty of the SUNLE method is somewhat limited because it simply accepts the doubly intractible limitation and proceeds with standard EBM learning. Standard methods for flow and score models have already been used for a nearly identical purpose. 
3. It would be interesting to see other real-world applications similar to the crab synapses, although benchmarking and evaluation are likely difficult.
clarity,_quality,_novelty_and_reproducibility: The AUNLE method deals with a major theoretical limitation of existing SBI methods and is the most technically original part of the work. The SUNLE method is a straightforward application of EBM learning to a conditional density estimation problem that was previously tackle by flows and score models. More thorough experimental results would improve the paper, although I am not familiar with the area of SBI and I cannot suggest specific examples.
summary_of_the_review: The most interesting part of the paper is the tilting of the normalizer in the AUNLE method, although unfortunately this method likely will not be able to achieve the performance of models tailored towards inference for specific data. The lack of novelty of the main SUNLE method and sparse experimental results lead me to recommend not accepting this paper at this time.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies a novel application of energy-based models for simulation-based inference, and experiments are conducted to verify the effectiveness of the proposed framework. The paper is clear, well-written and organized. The rebuttal only addressed some of the concerns raised by 5 reviewers and the paper finally received 5 mixed scores (i.e., 3 marginal reject, 1 marginal accept, and 1 accept). After the rebuttal, the average score falls into the range of a borderline paper.  Even though Reviewer m5HY champions the paper due to the novelty, there still remains unaddressed concerns raised by other reviewers. For example, Reviewers mHnK, kgMn and jDWH think that the novelty of the paper is marginal because the training of the EBM is standard and only the application is new. Besides, Reviewers mHnK, kgMn and VwrC think that the experiments are too limited to demonstrate the effectiveness of the proposed model. After reading the rebuttal and having an internal discussion with the reviewers, the AC thinks that even though some of the reviewers are satisfied with the rebuttal or increase their rating due to the feedback from the authors, the current paper still has critical concerns about limited novelty and insufficient empirical results. Therefore, the AC has to recommend rejecting the paper at the current stage. However, due to the promising outputs and the interesting applications of EBM shown in the paper, the AC urges the authors to improve and revise their work by considering the reviewers’ comments and re-submit it to the next venue in the future.

justification_for_why_not_higher_score: the current paper has a major issue about marginal novelty, which directly leads to a rejection of the paper at the current stage.
Decision: Reject