Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces contrastive neural ratio estimation (NRE-C) to develop a better estimator for the likelihood-to-evidence ratio p(x | \theta)/p(x) for simulation-based inference (SBI). They first cast the parameter estimation problem as a classification problem, then augment the K-class classification problem (where the goal is to select the best parameter \theta_k among K options) in Durkan et al. with another (K+1)th class which contains samples that were generated independently from p(\theta)p(x). They demonstrate that NRE-C is consistent and outperforms existing baselines on simulated benchmarks.
strengths_and_weaknesses: Strengths:
- NRE-C is a clean and straight-forward generalization of Durkan et al. and the binary classification “trick” for estimating likelihood ratios.
- The authors also performed extensive experiments and empirical evaluations on the SBI benchmarks suite.

Weaknesses:
- It’d be helpful if the paper was a bit more self-contained. There is a lot of information in the supplementary that would benefit from being moved to the main text (e.g. related works, further details on the SBI benchmark). As another example, l. 35 (equivalence of the 2 ratios) should be better explained. Why do we want these two quantities to be equivalent? The text should explicitly mention that it is to try to make posterior inference easier in SBI.
- The presentation was also unclear. For example, the figures are pretty confusing – I’m not sure what we are supposed to be taking away from this image/caption. Figure 1 was unclear, and Figure 2 was also not clear with the different notations (e.g. b, b’). 
- There is notation (e.g. f_w, x, theta, theta_0, p, etc.) that is not explained. It’d be clearer if this notation was collected in the following section (or introduced properly before its usage) to make it easier to follow. This causes conduction, such as \sigma \circ f_w(\theta, x) on l. 41.


questions: - When constructing this “hybrid” objective function (that blends NRE-A and NRE-B), what exactly are the limitations? For example, is it computationally expensive to generate samples x^{(b)} \sim p(x)p(\theta) to sZet up the classification problem? Given that the simulator takes a prior p(\theta) and a way to sample from the likelihood p(x|\theta), it seems like sampling from p(x) would be very hard). The paper also mentioned that sampling from the prior is slow. If we take into consideration this samping procedure for constructing the (K+1)th class, is NRE-C still more efficient than NRE-B?
limitations: Broader Impact statement is too broad, should be more specific about what “decisions about matters that affect living creatures” are, etc.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Density-ratio estimation can be performed via classification tasks (binary, multisample) and learn a potentially unnormalized model for the posterior distribution. 

This submission highlights a simple yet important flaw: when the posterior is parameterized as a softmax (this is the case for NRE-B a.k.a. InfoNCE), the scaling indeterminacy translates to a bias in the density-ratio estimate which trickles down to the rest of the analysis. The submission proposes a new loss that removes this bias. It carefully examines how that impacts posterior estimation in practice, through a series of controlled experiments. 

strengths_and_weaknesses: **Strengths**

This work is interesting, as it highlights a simple yet important flaw in a popular loss (InfoNCE) and shows how a simple fix can lead enable better posterior learning.

The connections with NRE-A (essentially NCE, depending on how the discriminator is parameterized) and NRE-B (InfoNCE) are appreciated. Given that the authors' contribution, NRE-C, can be seen as a generalization of the previous losses, it is interesting to see in Figure 1 what the best setup is: while it has been noted in previous work that (i) using many negative samples are advantageous, the insight that (ii) imbalanced classification is favorable when the tuples of entirely noise samples are over-represented, seems novel. 

The diagnostics section is appreciated as it is an important and rather open question when estimating conditional density models, such as the posterior distribution. 

**Weaknesses**

As I understand it, the simulations evaluate the quality of estimation by averaging across datasets (Figure 4). Is it common to report the variance *across* datasets? Could you report the variance *per* dataset across many (data-generation) *seeds*?

The effect of the hyperparameters on the convergence speed (Figure 5) is interesting, however it is not clear to me from the plot "that increasing K leads to earlier convergence at fixed gamma". Could the authors explain further?




questions: It is known from [1] that the NRE-B loss recovers the density ratio + a bias term. However, in [1] the bias is learnt as part of the estimation procedure. This way, the *un-normalized* density ratio (and by extension, its numerator the conditional density model, in your case the posterior) can be recovered without bias. 

Could you clarify why learning the bias does not solve the problem? Is it because you would only recover the *unnormalized* posterior whereas you need the *normalized* posterior in the diagnostics?
limitations: I see no particular negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors present a generalising framework of (what they call) NRE-A and NRE-B. They argue that is has clear advantages over especially NRE-B. In the experiment section they show some good values of hyperparameters K and gamma, found on a benchmark dataset.
strengths_and_weaknesses: I think the major strength is the generalising framework the authors present. This type of work usually helps fields move forward. 
The weakness, in my eyes, is the introduction of hyperparameters, especially gamma. I am unsure how easy it is to find good values of this if data is limited.
questions: I really enjoyed your introduction and motivation; it helped me a lot. However, I could understand that equation 2 is not influenced by k? Can you elaborate here?

In general, I think the motivation to have multiple k could be more elaborate. You mention it improves simulation-efficiency; is there more to it?

Line 89: "In contrast with NRE-A, one of these parameters θ_k is always drawn jointly with x.". As far as I could tell one of the two classes in NRE-A is exactly drawn jointly with x? So why is it in contrast?

In Equation 14: is Z_w computable? Or should it be approximated, and how good can such an approximation be? And, in data-limit (only) how quick is converge to 1? Vague answer is accepted here.



limitations: The authors include a section on societal limitations. I agree completely with the authors here.
review_title: null

Meta-review: metareview: The three reviewers agreed that the work is a valuable contribution to its field, and presents extensive experiments.

For the readers' benefit, I kindly ask the authors to take into account reviewers comments while preparing the camera-ready version.
In particular, the revised version should include:
   - the updated results table (across seeds, per dataset);
   - a clearer formatting of Figure 5;
   - an expanded discussion points on (i) how their method compares against learning the bias of NRE-B (Ma and Collins, 2018) (ii) clarifying the part on sequential vs. amortized methods.
confidence: Certain
Decision: Accept