Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces GATSBI, an algorithm for simulation-based inference based on adversarial training. This algorithm paves a promising avenue for applying SBI in high-dimensional parameter spaces or with implicit priors -- a setup currently out-of-reach for simulation-based inference algorithms. Experiments deliver a convincing proof-of-concept on a 784-dimensional parameter space, which none of the current SBI algorithms can solve. Experiments on lower-dimensional parameter spaces are less convincing.
main_review: Strengths:
- This work contributes a new algorithm for SBI for the setup of high-dimensional parameter spaces or implicit priors. This is a significant contribution in itself since this setup is currently out-of-reach for most (if not all) simulation-based inference algorithms.
- The paper is generally very well documented. The position of GATSBI within the context of previous methods for SBI is discussed in detail, much beyond what is usually found in other SBI papers.
- The connection to LFVI is made explicit and discussed in detail in the supplementary materials. Experimental results indicate that both methods perform similarly on the low-dimensional problems.
- The experiments demonstrate the usefulness of GATSBI on two high-dimensional problems. The Noisy Camera model experiment is the most convincing, as it shows its real potential in comparison to other methods.
- The quality of the approximate posteriors is diagnosed with SBC. It reveals that GATBSI provides well-calibrated posteriors.

Weaknesses:
- GATSBI is technically very close to LFVI, proposed initially for likelihood-free in hierarchical implicit models. For this reason, the novelty is limited.
- The experimental validation is limited to two common benchmark problems and two high-dimensional problems. GATSBI does not perform as well as NPE and NLE on the low-dimensional benchmarks.
- GATSBI comes with all the challenges and issues of training GANs.
summary_of_the_review: In this work, the authors try to address a major obstacle in simulation-based inference. Very few SBI approaches can operate in high-dimensional parameter spaces or with implicit priors, but GATSBI constitutes a convincing proof-of-concept that ought to be investigated further. Experiments are carried out properly, although the experimental validation could have been more thorough. The technical novelty is limited.

Despite some acknowledged weaknesses, I am overall positive about GATSBI. I am willing to recommend it for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors propose to use a GAN as an implicit posterior model for performing likelihood-free Bayesian inference, where the generator is trained to sample from the posterior, while the discriminator is trained to distinguish between parameter samples from the prior and the posterior *conditioned on the data sample*. Authors connect the method to prior work in adversarial (variational) inference, and also consider a *sequential* variant of the method, where the posterior approximation is refined iteratively for a particular datapoint. Authors show that the method matches the results of baselines on low-dimensional inference tasks, while also scaling to high-dimensional problems where classic likelihood-free inference methods don't work well.
main_review: I've enjoyed reading the paper: it is written with great attention to detail. The problem is introduced well, the method is clearly motivated and presented in great detail. The highlighted connections to existing methods are nice.

The biggest question mark for me is in novelty. I commend the authors for discussing the connection to LFVI, but this does highlight the fact that the differences between the proposed method and LFVI are arguably minor. Apart from the additional term in the loss (the role of which is unclear), the difference in the used divergence is the main one. As also pointed out by authors in the appendix, the best choice for the divergence is not clear, and also likely problem-specific. It would be interesting to better understand the role of the divergence in such methods, at least empirically.

In addition, work by Adler & Öctem (2018) seems to cover similar ground: a comparable method (but, again, with a difference divergence measure) is proposed, while Adler & Öctem also highlight that their method works with an implicit prior, and evaluate the method on a high-dimensional inverse problem.

Finally, the experimental results are somewhat underwhelming. It's not surprising that we don't win much on the low-dimensional problems, but I wish authors explored more high-dimensional inverse problems, perhaps also more realistic ones. 

References:
- Adler & Öctem, Deep Bayesian Inversion, 2018. https://arxiv.org/abs/1811.05910
summary_of_the_review: Overall, while the paper is written excellently, my doubts about novelty and the extent of evaluation are significant enough to put the paper slightly below the bar for acceptance. More extensive evaluation and/or discussion in the context of the prior work would help.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes to exploit GANs to do simulation-based inference (SBI) for black-box simulators. Generally speaking, the proposed GATSBI is identical to do the original GAN in a joint space, with the true distribution $p(x,\theta)=p(\theta)p(x|\theta)$ ($p(x|\theta)$ denotes the simulator) and the fake distribution $q(x,\theta)=p(x)q_{\phi}(\theta|x)$, where $p(x)$ is the marginal of $p(x,\theta)$. The key is that for a black-box simulator, one can query a lot of data pairs $(x,\theta)$. 

main_review: I think the novelties of this work are highly related to the importance of the considered simulation-based inference (SBI). Accordingly, it is worth discussing the wide applications of the SBI.

In Section 2.4, it's stated that ``GATSBI, ..., performs an optimization that also finds a minimum of the reverse $D_{KL}$". I think this is an obvious fact, because GATSBI ideally minimizes the JSD whose minimum is identical to that of the reverse $D_{KL}$. I don't understand why using a whole subsection to state that fact? 
summary_of_the_review: Technically, the novelties are limited, as both the original GAN and the reverse-KL GAN are well-known in GAN research fields.
However, if the simulation-based inference (SBI) for black-box simulators is quite important in some research fields, then there are certainly contributions from this work.

Empirically, the performance of the proposed GATSBI is only comparable to the baselines.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new method to do inference in simulation based systems. It performs "compiled inference" in a sense using a neural network to infer the latent inputs to the simulation. It accomplishes this using a clever GAN-based training objective.
main_review: This paper has the potential for broad impact. The number of applications for simulation based inference in science and engineering are far broader than recognized in the ML community.

Much of the prior art, as referenced in the paper, uses ABC, which is really a method of last resort and should be avoided. By presenting a GAN-style alternative to ABC it allows simulation based inference to be applied in much higher dimensional settings than prior methods. The authors show in the paper that the GAN equilibrium distribution is the one that performs exact Bayesian inference through the simulator. It is a clever approach.

Weaknesses:
- The VAE comparison doesn't seem as useful to the reader as more details on how to do actually get training to work. Perhaps there should be a swap of what goes in the appendix.
- The sequential GATSBI setup in eq (7) doesn't make much sense. The reweighting factor requires the computation of the marginal p(x), which is not tractable in general. This setup needs to be clarified.

Questions:
- The paper assumes we do not have access to the likelihood or gradients of the simulator output. In general, the likelihood will be hard to compute for complex simulations. However, with modern autodiff, it is not unreasonable to assume that the gradients may be available even in a complex simulation. Have the authors considered an extension where gradients are available?

Nitpicks:
- Subfigure captions typically go below the figure
- The figures are not BW friendly
- It is typically i.e., not i.e.
summary_of_the_review: The paper has big potential impact. However, some of the extensions in the paper don't make much sense as presented.
review_title: null

Meta-review: The paper explores the application of generative adversarial networks as posterior models in simulation-based inference. A new method is proposed, and its connections with related work are studied. The proposed method is empirically evaluated on joint inference of up to 784 parameters.

The reviews are borderline, with one weak reject, two weak accepts, and one strong accept. Overall, the paper is well-written and well-executed. Its main strength is the promising performance of the proposed method in high-dimensional parameter spaces, which are out-of-reach for many existing approaches. The main weakness of the paper is its lack of novelty: the proposed method is only marginally different from already existing ones, while the paper could have explored the differences to a greater extent.

On balance, I'm leaning towards recommending the paper for acceptance. Despite the lack of novelty, the paper is well executed with potential impact in high-dimensional simulation-based inference.
Decision: Accept (Poster)