Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper extends the previous memoized wake-sleep method to train complex generative models with discrete/structural and continuous latent variables. The method incorporates an assumption on the conditional dependence between the discrete and continuous latent variables and essentially employs a clever importance sampling procedure to capture the dependence. The experiments show applications to spatial and temporal compositional models where the proposed method outperforms VIMCO and reweighted wake-sleep.
main_review: Strengths:
1. The paper is clearly written with good motivation and methods sections. The rigorous and concise notations convey a lot of information. The diagram in Fig 5 is less clear but many details are given in the appendix. 
2. The method is technically sound and involves a non-trivial application of importance sampling, which is of value.
3. The experiments are reasonably complicated (but see below) and serve a good purpose demonstrating the potential of this method.

Weaknesses:
1. My main concern is the significance of this paper. The applications are relevant to comp. cogsci community, but the authors did not discuss much of the cognitive link or compare it with any cognitive/behavioural data. As a machine learning paper, I do not find the comparison with VIMCO and RWS to be sufficient. Further, although the two problems in the experiments involve reasonably complicated generative models, these models are not the main contributions of the paper, and they are arguably low-dimensional in latent space. For example, in the second experiment, there are only 4 discrete locations among other latent variables. Again, as a machine learning paper, the question of scalability is not well addressed. As a simple test, is it possible to learn a flexible parametric likelihood (Gaussian with NN mean) for experiment 2, rather than using a differentiable renderer?
2. The fact that the problems are relatively low-D raises two further questions: how does this method scale to larger latent dimensions? How does this method compare with ablations of this method? As far as I understand, the gap between RWS and the proposed method include a) memory; b) importance sampling to maintain extra dependency between $z_d$ and $z_c$. Either of these can be augmented to RWS for improvement. How do these two contribute to performance? I am not asking the authors to "just compare more baselines", but to conduct a more careful comparison between contributions of various components in their model.
3. Importance sampling has a proveably large variance. How does this method not suffer from this problem? Is it because of the sleep-phase updates? I hope the authors could address this problem. 

Other suggestions
1. On the algorithmic side, I'd like the authors to expand a bit more on the paragraph before experiment 4, which I think touches on important upside (mentioned dependence between $z_c$ and $z_d$) and downside: for model where the induced true posterior dependency factories the other way as $p(z_d|z_c,x)p(z_c|x)$, the proposed method may suffer.
2. In the explanation before equation (10), the authors mentioned that "We want to minimize the average of KL". Why does this make sense? The authors could include a derivataion starting from the joint KL
$$
KL[p(z_c,z_d|x) || q(z_c, z_d|x)] = \int  p(z_c,z_d|x) \log \frac{p(z_c,z_d|x)}{q(z_c,z_d|x)} = \int p(z_c|z_d,x)  p(z_d|x) \log \frac{p(z_d|x)}{q(z_d|x)}\frac{p(z_c|z_d,x)}{q(z_c|z_d,x)}
$$
This will then expose another assumption that the aforementioned expectation is in fact taken over the approximate $q(z_d|x)$. This is also related to the expectation propagation method. 
3. For the runtime comparison, I'd like the authors to also show the memory cost comparison and be explicit about the tradeoff.
summary_of_the_review: In general, I think the method is an interesting and valuable contribution, I also enjoyed the precise notations the authors adopted. But either the impact/scalability of the method or the machine learning power needs to be more clearly demonstrated for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an extension of memoised wake-sleep which allows applications to hybrid discrete-continuous graphical models. The approach is directed towards Bayesian program synthesis.

Specifically, the memoised wake-sleep algorithm requires computing the joint probabiltiy of a discrete latent variable and an observation, which would require integrating out any continuous latents. This paper proposes to solve that problem using importance sampling.
main_review: **Strengths**

* The paper deals with a relevant issue - how to extend an existing approach to hybrid discrete-continuous graphical models.
* The exposition is clear. Related work and background are explained well. It is made easy to see where the authors' work fits in to previous work.
* The proposed method is principled and based on a well-established method from approximate inference (importance sampling). There are no ad-hoc terms or deviations from theory.
* Experiments are favorable towards the approach and show that it is better suited to program synthesis than other applicable methods.

**Weaknesses**

* Somewhat weak in novelty.

**Other questions/comments**

* In 2.1 you say:

> If we try to use the same approach for hybrid discrete-continuous latent variable models, all proposed continuous values are will be unique and the posterior approximation will collapse onto the MAP estimate.

I don't immediately see why this is the case. Is this an empirical observation or does it follow from theory? Also there is a typo in that sentence.

* In theory we could approximately integrate out the continuous variables using other methods. Did you consider comparing to a simple MC estimate instead of importance sampling (maybe even with a single sample, as done in VAEs)? Or is there a reason why such a comparison is not possible?


summary_of_the_review: I think the paper is good and recommend accepting.

The main weakness I see is that the contribution seems to be limited to using importance sampling for marginalizing out continuous variables. The actual presence of continuous variables can be seen in the Gaussian mixture models experiment of the memoised wake-sleep paper [1], where it was possible to treat them analytically. Of course, the general case doesn't allow for that and therefore the paper is still making a valuable contribution by investigating what to do in the general setting.

Further, the paper is specifically aimed at settings where discrete variables arise in a programmatic context and impact control flow. The experiments show that the proposed approach has an edge over other general methods in this setting.

[1] Hewitt et al., "Learning to learn generative programs with Memoised Wake-Sleep", https://arxiv.org/pdf/2007.03132.pdf
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Memoized wake-sleep is a past method that builds a variational approximate posterior over discrete latent variables by memorising previously drawn samples.  However, memorised wake-sleep can currently be used only on models that are purely discrete.  This paper extends memoized wake-sleep by providing a mechanism for using VI to integrate over continuous latent variables.
main_review: Good paper, giving an interesting new method.

My main concerns are around the empirical evaluations and wallclock times.  In particular, I can easily see that memoization in MWS is going to significantly help runtime.  This isn't so clear in the present algorithm, because there's an awful lot more going on (all the steps involving continuous latent variables don't seem to benefit much from memoization).  This concern is further reinforced by comparing algorithms by iterations (it would also be helpful to look at algorithms by runtime).  Could the authors clarify why their method should help runtime (or whether it won't, and its primarily intended to help performance), and include wallclock time comparisons?

I am also surprised by how much better the algorithm performs relative to RWS.  Intuitively, it would seem that HMWS is doing something very similar to RWS, but memoizing the best samples from the approximate posterior over latent variables.  That would seem to help, but it isn't clear to me that it would help _that_ much, as I wouldn't expect the best samples from the approximate posterior to be that different from typical samples.  Can the authors comment? This is especially concerning, given that:
* There is no hyper parameter optimisation (Adam with default hyper parameters).
* There is no discussion of "matching" the RWS and HMWS architectures to ensure a fair comparison (that I could see).
summary_of_the_review: Good paper, presenting an interesting new method.  A few concerns as regards evaluations + performance tradeoffs.
review_title: null

Meta-review: The authors propose a method called Hybrid Memoised Wake-Sleep (HMWS) for training models with both discrete and continuous latent variables efficiently using amortized inference. They extend Memoised Wake-Sleep (MWS), which can only handle discrete latent variables, to discrete-continuous systems by using importance sampling to approximately marginalize out the continuous variables and then applying MWS to the discrete variables.

This is well motivated and well written paper. The method is novel, clearly described, and evaluated in two fairly different interesting settings. However, while the empirical evaluation was considerably strengthened by the ablation studies and other experiments included in response to the reviewers, it is still on the weak side. The main issues are the relatively low-dimensional latent spaces and insufficiently tuned baselines. As one reviewer pointed out, importance sampling is unlikely to scale well to high dimensions, so exploring this aspect experimentally would strengthen the paper. The use of default Adam parameters for all methods and models substantially undermines the results and might at least in part explain the underwhelming baseline performance. For example, VIMCO has been successfully used to train a discrete/continuous model of seemingly comparable complexity in [1] and yet here it seems to fail completely. To shed some light on this, the authors might want to explain how they used VIMCO and whether their approach was substantially different from the one from [1].

Finally, it at least somewhat misleading to claim, as is done e.g. in Sec. 4, that HMWS is more memory efficient than the baselines, when unlike them, it needs to store M discrete latent configurations per training case. Please make this claim more precise to avoid potential confusion.

[1] Variational Memory Addressing in Generative Models, Bornschein et al., NIPS 2017
Decision: Accept (Poster)