Venue: UAI 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: Computing an expectation can be more efficient if we can reformulate the task to one of estimating the normalization constants of multiple densities as shown in an earlier TABI paper.

The paper shows how the TABI idea can be easily implemented in a PPL by rewriting the program automatically. A new annealed importance sampling algorithm is mentioned but not described in the main text.

There is some theoretical justification and experiments which prove the point that TABI is a good idea.

Q3 Main strengths: The paper demonstrates one of the key benefits of focusing research energy on Probabilistic Programming Languages. Namely, that a current research idea in probabilistic inference such as TABI can be applied quite easily to any PPL and it can benefit all users of the PPL without them having to reimplement the idea.

Although the implementation of TABI in the target PPL of this paper is quite simple that fact should not detract from the paper. It is still a somewhat novel idea and other PPL implementors would benefit from reading this paper and learning the importance of including TABI in their own PPL,

Q4 Main weakness: The authors focused exclusively on implementing TABI and annealed importance sampling exactly as in the prior TABI paper with very similar experiments. I see this as a lost opportunity. For example, a natural question that arises to me as a PPL researcher is how does TABI + annealed IS compared to TABI + MCMC. In other words, there are MCMC algorithms that can estimate the normalization constant of unnormalized densities. Why not try any one of them and compare? Such an analysis can help inform the reader whether TABI has to be used with annealed IS or are other methods equally good with TABI.

The experiment on the SIR model shows very poor results for MCMC. This is somewhat hard to believe given that MCMC is often used in SIR models quite successfully (just google search "SIR model MCMC"), there is even an R package for this. Perhaps the authors should use a better MCMC implementation for their baseline comparison.


Q5 Detailed comments to the authors: Beyond what is mentioned above:

Please include the ESS numbers for MCMC in all the ESS figures.

The RSE curves in Figures 3 and 4 appear to be trending downward. In this case I would be curious to know how these samplers perform with further samples, at least 10^6.

Normally most PPLs come with a library of multiple inference algorithms for estimating the normalization constant, please consider using one or more of them to help the reader get more insights on using TABI.

Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: While the work is moderately innovative and would provide valuable insights to PPL researchers, it could benefit by comparing to more competitive baselines and other similar approaches to make this valuable.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper with, with novel ideas, excellent impact on at least one area of AI or high to excellent impact on multiple areas of AI, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: This paper introduces the concept of “expectation programming” and develops program-transformations for Turing probabilistic programming language that turn an expectation program into several standard Turing programs, which can be used to perform inference for the expectation programming problems. The authors prove correctness of their transformation (under natural mild conditions on the program).
Q3 Main strengths: Isolating an interesting sub-problem, expectation programming, for probabilistic programming is in itself an interesting contribution.

The transformation is simple, based on target-aware Bayesian inference (Rainforth et al, 2020) and I think the simplicity is really a plus here. Basically, the transformation works by pattern-matching return statements in the program and using a @addlogprob! command from Turing, which allows adding arbitrary terms to the unnormalized density function. This is nice because it allows exploiting marginal inference routines from Turing for expectation programming (however, the authors in the end also implemented a new marginal inference method into Turing: annealed importance sampling).
Q4 Main weakness: I am not an expert on PP, but I did not really see any weaknesses of this paper. Yes, it is rather straightforward, mostly exploiting target-aware Bayesian inference, but I do not think that should be a problem,
Q5 Detailed comments to the authors: I enjoyed reading the paper and do not really have any detailed comments on how to improve it.
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: This is a strong paper. It identifies an overlooked problem and designs an elegant solution for it.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: This work provides a strategy to directly compute the expected values from a Turing program, by rewriting the program and providing the AnIS importance sampling inference engine. This has as advantage that the task at hand can be achieved more accurately by utilizing properties such as those presented in the TABI framework.
Q3 Main strengths: - The focus on the task of expectations is interesting and relevant. 
- Achieving this through a source transformation makes the method modular and compatible with many existing systems and approaches.
Q4 Main weakness: - The work is mainly an implementation of the TABI framework in Turing.
- The authors explain that this task, while inefficient, is available in Stan, Pyro, PyMC and plain Turing (sec 4). Why then not compare with these implementations and show the added value of the proposed method. This seems trivial to do given the programs used in the experiments and makes me wonder why this is not done. I assume that the authors consider the MCMC baseline in the experiments to be this comparison?
- There is also no comparison with just using the TABI framework. Since the paper is heavily based on that work (and implements it), this seems like a mandatory baseline.
Q5 Detailed comments to the authors: 
Minor:
- 2.2.: "sometimes referred to as the marginal likelihood", wouldn't "partition function" be more common?
- Some of the sections (e.g. def 4, proof of theorem 1) seem quite straightforward.  This slows down the pace of the paper. For example, the argumentation follows directly from the definition or the assumptions part of the theorem. 
- Another point where the paper build up is slow: while it is often noted that the proposed method is fundamentally different it takes a long time to show how different it is and whether approximations will be used (i.e. not fully clear whether the proposed will be an analytical method on top of the traces or an approximation when the engines are introduced in 3.2/3.3).
- "All our experiments correspond to target functions which are always positive", why this choice?
Q2(4) Quality of experiments (Optional): 1: Poor: The experimental evaluation is flawed or the results fail to adequately support the main claims.
Q7 Justification for your score: I like the overall direction of this paper but feel the comparison with existing systems is too limited. The focus is too much on the own implementation.
review_title: null

Meta-review: Meta Review: Reviewers agree that the ideas in this paper are novel, interesting, highly relevant to UAI, and likely to have practical impact on the probabilistic programming community.  Reviewers appreciate clarifications in the rebuttal which leads to a unanimous decision to accept.
Decision: Accept (Oral)