Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigate sampling problems in HEP simulations and neural importance sampling (NIS) based methods. In addition, an NIS algorithm developed based on the ZÜNIS, a PyTorch library, is introduced. The performance of the proposed algorithm is demonstrated by toy examples and some numerical experiments of HEP simulations.
main_review: It is unclear what is the major contribution of this paper. The library ZÜNIS seems an important tool in the field of HEP, but is discussed very briefly. The loss function (10) seems new, but it is unclear for readers why such an auxiliary probability distribution q is required and q can be simply selected as a uniform distribution.
summary_of_the_review: The innovation is insufficient.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a new importance sampling scheme based on a transformation and relying on Normalizing Flows.
main_review: The work contains a new importance sampling scheme. The state-of-the-art discussion  is quite poor, ignoring all the recent literature on importance sampling and adaptive importance sampling schemes (except the VEGAS algorithm). See my comments below, please.

- The differential is missed in Equations (7) and (8).

- The state-of-the-art discussion that the authors do not take into account is, for instance,

M. F. Bugallo et al, "Adaptive Importance Sampling: The Past, the Present, and the Future", IEEE Signal Processing Magazine, Volume 34, Issue 4, Pages: 60-79, 2017.

M. Bugallo et al, "Adaptive Importance Sampling in Signal Processing", Digital Signal Processing, Volume 47, Pages: 36-49, 2015.

J. M. Cornuet, et al, “Adaptive multiple importance sampling,” Scandinavian Journal of Statistics, vol. 39, no. 4, pp. 798–812, December 2012.

O. Cappe et al, “Population Monte Carlo,” Journal of Computational and Graphical Statistics, vol. 13, no. 4, pp. 907–929, 2004.

V. Elvira et al, "Improving Population Monte Carlo: Alternative Weighting and Resampling Schemes", Signal Processing Volume 131, Pages: 77-91, 2017

G.R. Douc et al, “Minimum variance importance sampling via population Monte Carlo,” ESAIM: Probability and Statistics, vol. 11, pp. 427–447, 2007.

L. Martino et al, "Layered Adaptive Importance Sampling", Statistics and Computing, Volume 27, Issue 3, Pages: 599-623, 2017.

O. Cappe et al, “Adaptive importance sampling in general mixture classes,” Statistics and Computing, vol. 18, pp. 447–459, 2008.

L. Martino et al, "An Adaptive Population Importance Sampler: Learning from the Uncertanity", IEEE Transactions on Signal Processing, Volume 63, Issue 16, Pages 4422-4437, 2015.



summary_of_the_review: The paper could contain interesting material but requires additional work on order to be published.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This manuscript proposes a framework to compute integrals of arbitrary functions over a finite volume (uniform distribution). The toolbox is a modular implementation of Neural Importance Sampling (NIS, Muller et al. 2018). The experiments demonstrate the toolbox and its usefulness compared to a popular software (VEGAS) and uniform sampling.

main_review: # Strengths

* The background on IS and NIS is very didactic and has tutorial value.
* The toolbox is modular and straightforward use.

# Weaknesses
* There is no discussion regarding activation functions and their role in the behavior of the resulting distribution (IS proposal). 
* The technical novelty is very limited. The only innovation is a gradient trick in Section 3.2 to make training more stable.


# Observations
* In Section 2.3 onwards, the authors use the notation $p(x, \theta)$ — and $q(x, \theta)$. Since $\theta$ parameterizes $p$ and $x$ is the only random variable here, the usual notation is something like $p(x;\theta)$ or $p_{\theta}(x)$.
summary_of_the_review: This is work proposes an organized toolbox and has very limited novelty. Therefore, I’m suggesting a rejection. However, since I’m not a connoisseur of high energy physics, I am not able to exactly gauge the utility of this toolbox to the specific demographic.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper describes ZüNIS, a neural importance sampling (NIS) library. The main application area is high-energy physics (HEP) simulation which is used to, for example, interpret experiments at the LHC. Traditionally, integrals arising in HEP simulations are solved numerically with the VEGAS algorithm. VEGAS is an adaptive importance sampler that neglects correlations between variables and optimizes a coordinate-wise discrete transformation of the integration domain. VEGAS potentially suffers from large variances in the Monte Carlo approximation and violations of the underlying assumptions. NIS offers a remedy in that it can adapt itself more flexibly to the integrand and thereby achieves smaller Monte Carlo errors. The authors propose a simple extension of NIS that allows them to reuse samples when the evaluation of the integrand is very complex and expensive (as is the case in HEP simulations). They show that ZüNIS outperforms the VEGAS algorithm on various toy examples and simple HEP processes. 

main_review: # Pros

1. The paper introduces ZüNIS, a fully automated software library for easy NIS. 

2. ZüNIS offers flexible and data-efficient training of normalizing flows in the context of importance sampling. 

3. The paper establishes a rigorous link between the true loss function and its Monte Carlo approximation (which can be used for training flows). 

# Cons

1. The paper is in most parts a description of the ZüNIS package but does not contain much original research. 

2. The approach is focusing on a single target $f$, whereas other sampling algorithms such as MCMC typically aim to generate samples that can be used for multiple targets. 

3. Comparison with other sampling approaches (MCMC or quasi Monte Carlo) is missing. 

# Comments and Questions

- In case of expensive integrands $f(x)$, the authors propose to sample from an auxiliary pdf $q(x)$ to approximate the intractable variance of the importance sampling estimator (eq. 10). This separates sampling (to estimate the intractable integral) from the trained model. However, how to choose the auxiliary pdf $q$? Hasn't the problem of solving an intractable integral just been cast into another intractable integral? Why should it be easier to choose a suitable $q$ than to choose a suitable proposal for the main task (i.e. solving $\int_\Omega f(x) dx$)?

- Also the related remark about deep $Q$-learning on page 4 remains quite obscure to me. Are you using a $Q$-learning strategy? 

- Against which implementation of VEGAS do you compare ZüNIS? There seems to be an enhanced version, VEGAS+, which performs superior to the original formulation (see Lepage, 2021). 

- I could not reproduce your results by following the instructions in "Reproducibility statement" due to import errors. After fixing some import errors, new import errors occurred such that I finally gave up. 

- It seems that figures 3, 4 and 6 are not discussed in the main text. 

# Typos

- page 1: "predictins" 


summary_of_the_review: The paper mainly describes the ZüNIS package but does not contain much original research.
review_title: null

Meta-review: The paper considers neural importance sampling (that is, importance sampling with a trained flow proposal) and its application to high-energy physics. The two contributions of the paper are: (a) a methodological improvement in the training of the proposal; (b) a description of a software library that implements the framework.

All reviewers were critical of the paper and recommended rejection. The main issue raised was that the methodological contribution was not novel or significant enough, and not sufficiently evaluated. The authors disagreed with the reviewers that the methodological contribution was not significant enough, but they acknowledged that the first version of the paper did not present the contribution clearly; consequently, they submitted a heavily revised second version following the reviewers' feedback.

Although it seems that the second version is an improvement over the first one, it's clear that the paper requires a second round of reviewing to ascertain whether it satisfies the requirements for acceptance. At this stage, the consensus among reviewers remains that the paper should be rejected. For that reason, I cannot recommend acceptance to ICLR. I sincerely hope the reviewers' feedback will be useful to the authors for a future submission to a different venue.
Decision: Reject