Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an algorithm for computing an approximation of the posterior and marginal likelihood by analysing the sequence of programs using neural networks, as well as a meta-algorithm for learning the network parameters over a training set of probabilistic programs.  Experiments demonstrate the feasibility of the meta-algorithm for learning inference algorithms that generalise well to new but similar programs; these learnt algorithms were sometimes found to outperform alternatives in terms of time-efficiency. 
main_review: There were quite a few terms that I was not familiar with - for example, what is the "state" of an algorithm?  I did not find a formal definition of this term in the paper.  In the definition of the INFER function, what does the keyword "in" mean?  It did not seem clear how well the white-box inference algorithm performs when the number of commands in the program is very large. 
summary_of_the_review: I am not at all familiar with probabilistic programming; this does look like a serious piece of work, though I do not know how novel it is or whether the claims in the paper are correct.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents an algorithm for composing inference algorithms out of simpler neural net building blocks, one per unique statement type in the probabilistic programming language presented. The language is simple without recursion or loops, reducing issues due to feedback problems from the approximation. The networks are trained using HMC or importance sampling samples from many programs in a similar space. There is an empirical study of several classes of small Gaussian probabilistic programs.
main_review: - The paper claims that the learned inference algorithm works well for tasks which are "similar" to the training problems, but the notion of similarity is not fully defined, nor is there an example of how the system fails when applied to a dissimilar program. Are there diagnostics for checking the output when the model is applied to a program which is too dissimilar?
- The experimental study tests performance within a family of programs, each using small neural networks to infer each program statement. Does training on all families of programs allow the system to make accurate inferences on any of those families?  Does it allow it to generalise across families? Without some notion of how the system generalises I'm not sure when I would choose to use this rather than running HMC on my program, given a single HMC run will be faster than training the neural networks on multiple HMC runs for different programs "close" to the program of interest.
- The experiment in section 5.3 shows that the system is around 2x faster than importance sampling from the prior, but this doesn't take into account the time necessary to train the neural nets nor the time taken for the importance sampling runs used to generate the training data.
- How are losses propagated through the program? If each neural net is 3 layers, then programs with 10 statements have at most 30 layers, which is usually past the point where some amount of regularization or normalization is necessary to stabilize training or prevent vanishing gradients. Could the authors comment on the stability of training?
- What's the failure mode when the test loss diverges? Is it detectable without having HMC or other high quality samples?
- How robust is this approach to differing choices of neural net architecture?  The paper uses a 10 dimensional state space when parsing the program, but it's not clear how this value should be modified as the number of latent variables or the program complexity changes.
- Overall the paper is well written and explained, and the experimental study is detailed for the areas it covers.
summary_of_the_review: The paper presents a learned inference algorithm, but it's not clear how it generalises either across program types (which is necessary to amortize the training cost wrt HMC), or across neural network architectures (e.g., changing the internal state space in response to increased program complexity).  Additionally it's noted that occasionally the test error diverges, but there's no discussion of how to detect this in practice if the system was used for inference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This introduces a meta-learning algorithm for learning
inference algorithms applicable to any probabilistic
program. This is accomplished by associating a neural network
with every grammar rule of a probabilistic programming
language and outputting posterior draws. This generalisation is
possible because each neural network component is feed marginal
likelihood information for each PPL instruction.
	
main_review: This work is very interesting and novel. It's a unique attempt
to learn a general inference algorithm. I think meta learning
is something that is a good fit for many Bayesian approaches
and I want to see more work like this.

I'm curious about the expressivity of the language. The
grammar suggestions a modelling language that consists of
sequence of commands. I don't see how this language would be
able to express recursive programs. It seems one would need
something like a label and jump commands to accomplish
that. This is admitted in the appendix but not really
acknowledged in the main text. I think the main paper should
reflect the present limitations and not over promise the
existing contribution.

I have some concerns about the experiments. Some examples are
fairly simple and the results for the more substantial ones
are not shown (like hierd and rb). The test losses look fairly
bad for the experiments that are shown, so I'm not fully sure
generalisation has been demonstrated. If the issue is a few bad
 generated programs maybe show median loss? The results in Figure 4
seem to be for during training, but feasibility presumably
requires similar results on unseen programs at test time?

 I also worry about correctness. As the neural networks are
used as is and not as a proposal, how much can we trust
the posteriors that come out of this method? Is there anything
that can be said about the learnt posterior? It seems right now
that the learned distribution can recover the mean of the true
parameters and maybe the variance?

Some of the writing is slightly sloppy. For example the phrase
"so-called static single assignment assumption" is used. I don't
know what that means, but I do know there is a static single
assignment intermediate representation that exists within many
compilers. I think that's what the paper meant to refer to.

Related work should cover how this approach differs from Stites et al.
(https://arxiv.org/abs/2103.00668). 
summary_of_the_review: I recommend this paper for rejection. While I think the approach has the potential to work, right now it's very hard to get a sense of what was learned by the meta learning algorithm, how well any of this generalises when model structure or even observed data changes significantly, or even if the language is too restricted to make this is a significant enough contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new restrictive class of probabilistic programs with fixed number of random variables and without loops. The authors then propose an inference technique that learns the parameters of a neural network for sampling from the programs posterior distribution by composing it from individual neural networks for each atomic command in the language. This technique is shown to perform well during inference once the neural network has been trained on training programs. At the very least the inference speed is shown to be very high. 


main_review: The main drawback in this work is a lack of novelty. The use of neural networks to train a proposer for a model is not new. While the authors attempt to cast their work as something different than IC this doesn’t quite come out. The claim that a neural network for one program can be used for a different program even though the neural network takes a one hot encoded representation of the variables in the model is hard to see. A clear technical statement of what kind of cross model generalization is possible is needed. 

The paper shows results across model structures where the dependency graph and the position of a function changes but the number of variables is the same. It is not clear why IC can’t deal with this minor variation in the same probabilistic program. 

These models are so simple, how hard would it be to train ic on these models and then do inference? I would like to have seen ic results in this paper to believe that this work is different. 

The language chosen is not a universal ppl. I can’t follow how this can be used as an intermediate language by a compiler for a universal ppl as claimed on page 3. Please show an example of how a program with unbounded random variables can be compiled into this language. 

The inference in this paper looks like mean field variational inference. Which makes me wonder whether hmc is really such a good comparison. Please show some comparisons to vi. In Stan this would be trivial to run since you are already running hmc. 

The models are very simplistic with no discrete variables and no multimodal posteriors. It is not a meaningful claim to make (footnote on page 9) that the inference algorithm provides a good coverage of the posterior by covering all the modes. The posteriors shown for multimodal models should at least look multimodal.  Variational distance or symmetric kL divergence results would be needed to make claims about correctness of the posterior. 

Regarding ess per second results these can be misleading. The algorithm might have a cap of ess for example. It would be better to run your algorithm for the same duration as hmc and show higher ess numbers.

summary_of_the_review: The claim that the paper provides generalization of compiled inference across models is not supported by the description or the simple examples. These appear to be covered by existing work on inference compilation. 

The focus on a very restricted class of ppls makes this work very limited. I don’t believe I learned anything from this paper.
review_title: null

Meta-review: The paper presents a meta-algorithm for learning a posterior-inference algorithm for restricted probabilistic programs. While the reviews agree that this is a very interesting research direction, they also reveal that there are several questions still open. One reviewer points out that there learning to infer should take both the time for learning+inference and the generalization to other programs into account, i.e., what happens if the program is too different from the training set? Is benefit than vanishing? Moreover, as pointed out by another review, recursion as well as while loops are not yet supported. Also, the relation to IC needs some further clarification. These issues show that the paper is not yet ready for publication at ICLR. However we would like to encourage the authors to improve the work and submit it to one of the next AI venues.
Decision: Reject