Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a simple method for the arbitrary conditioning problem, which trains an amortized inference network for the distribution $q(z | x_o)$. Compared with previous work, the method does not require reparameterizable inference networks, and thus enables the use of more flexible models.

**Post-rebuttal update**: I am satisfied by the authors' response, and believe this work is suitable for publication.
strengths_and_weaknesses: **Strengths**:

The method is simple, intuitive and appears to have competitive performance.

**Weaknesses**:

I do not have any major concerns.  My only gripe is that Theorem 3.1 appears to add little value -- the distributions to be optimized depend on $\theta$ in a somewhat complicated way, and the only thing I can read out of Eq. (4) is that when both $p_\phi$ and $q_\psi$ are optimized, the imputation distribution determined by $q_\theta$ will be correct; but this already follows from Eq. (3).  More discussions on the general case would be helpful.
questions: I do not have any major questions.  A few minor questions / comments:

- You should cite the early works on data imputation with VAEs, e.g. Rezende, Mohamed & Wierstra (2014, Appendix F).

- In Theorem 3.1 the definitions of $p_{\theta,\phi}, q_\theta(z | x_o, x_u)$ should be moved to the text.  The latter distribution should also depend on $\phi$ (the definition of $q_\theta$ does not involve $x_u$).  As both terms depend on $\theta$, ideally there should be more discussion about the optima, as mentioned in the above section.
limitations: Limitations and potential negative societal impact are adequately addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This manuscript proposes a method "Posterior Matching" that trains Variational Autoencoders (VAEs) to be robust to missing data when they are trained with fully observed data.  The idea is straightforward to implement and include with a wide variety of VAEs, and leads to relatively straightforward algorithms.  Theory demonstrates that the method attempts to approximate arbitrary conditional distributions.  Limiting to a subset of VAE approaches, the posterior matching method demonstrates improved performance on a number of benchmarks.
strengths_and_weaknesses: The biggest strength of this paper is that it is a theoretically motivated algorithm that is simple to include in many VAE models, and can especially work with more complex variants such as VADE.  While there are two glaring limitations (see next section), this is a big advantage and the empirical results suggest that the method is effective.  The simplicity of the active learning approach is quite nice as well.

A weakness is the relative lack of comparisons.  While Posterior Matching is compared to multiple algorithms within the VAE space, there are many competing approaches for applications such as inpainting.  It seems clear that the proposed approach can improve in this cases, but it is unclear if I should consider their approach for an inpainting task.

There are several minor issues.  For example, the notation is a bit inconsistent.  As an example, in equation (1) the partially observed posterior is written in terms of the variational posterior; that is explicitly true as the variational distribution is an approximation and they are not exactly equal.  This seems like the authors meant that this term should be similar to $q_\theta(z|x_0)$ instead.  Either way, this should be clarified and explained more.

As a comment, given the motivating case in Figure 1, it would have been nice to see the proposed approach maintain the multimodality in the generated examples of Figure 5(b).  I would appreciate getting stronger evidence that the multi-modality posterior is in fact maintained.


questions: How feasible is it to extend this approach to a case where the data is not fully observed?

Can the in-painting examples in fact capture the uncertainty in digits as motivating in Figure 1?

How well can this method handle missingness that isn't missing completely at random?

What is the optimal strategy for picking the missingness patterns during training?  How would this be determined for a new dataset?
limitations: The authors clearly state that they assume that they have access to fully observed data for training, and they correctly identify that as a limitation; however, they state that "this is generally not a relevant limitation for our purposes." This limitation is quite large and is downplayed too much in their manuscript.  For example, in the motivating medical case given in the introduction, this would render the method infeasible.  I work with medical data, and I am unaware of any problem within that domain where I could naturally apply their methods, whereas I can and do apply competing missing data approaches for VAEs.  I would suggest that the authors clarify this limitation upfront in the introduction, at least, and discuss it at greater length rather than making it an small comment in a technical section.

The other major limitation, which was not addressed, is that this method assumes a "Missing Completely at Random" model of missingness.  This model is incorrect for many medical problems, and would be an issue on the motivating example in the introduction.  I do not see how to adapt the Posterior Matching approach to handle this case, but it would be helpful for the authors to clearly state this limitation to give the readers a better sense of when this method will and will not work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a general approach for adding arbitrary conditioning to models based on the VAE architecture. An additional encoder which takes partially observed inputs is used to define a partially observed posterior which is trained to match the fully observed posterior by adding an additional term to the loss function. The simplicity of this approach allows one to train a partially observed posterior for any VAE-based model and even to learn a partially observed posterior for an existing trained VAE. 

The authors demonstrate the effectiveness of their approach on a number of tasks, using MNIST to illustrate the model matches their motivating intuitions and then using image inpainting, imputation of tabular data, and partially observed clustering to demonstrate the effectiveness of their method on tasks involving arbitrary conditioning. The method presented in this paper does not achieve top performance on all tasks, but is competitive with the SOTA and improves on other VAE-based methods. Other methods are less flexible, in the sense that they cannot be immediately integrated with a new VAE architecture, one clear appeal of the method presented here.

Additionally, the authors apply their method can be used for greedy active feature acquisition, where a conditional distribution based on partially observed features is used to greedily select the most informative feature to add to the model. Here, the method is shown to be more accurate and faster than existing methods.
strengths_and_weaknesses: The paper is based on a simple but novel idea, which is clearly explained and validated using well thought-out experiments. 
questions: No questions to the authors
limitations: One limitation, which the authors remark on, is that this method requires fully observed training data. While there may be cases where it is advantageous to be able to train a model on partially observed data, there are many cases where some fully observed data will be available at training time.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors propose a model and learning method, called posterior matching, to perform arbitrary conditioning with VAE models. The proposed method can be applied to a variety of VAE models without modifications to the VAE model itself, and can be used with a pre-trained VAE or can be trained alongside with the actual VAE model. 
strengths_and_weaknesses: Originality:
Previous work has proposed deep generative models for arbitrary conditioning using e.g. autoregressive and flow based models.  VAE based have also been proposed for arbitrary conditioning.  Regarding originality and novelty, this work seems to extend previous VAE methods by allowing more expressive posteriors as well as by allowing applying the proposed method to a pre-trained VAE model. Authors do not contrast their method to recent methods that allow learning VAEs from partially observed data (see detailed comments below). 

Quality & Clarity:
Quality and clarity appears generally good. I have comments regarding the clarity of the experiments (see below). 

Significance: 
Significance of the work is very high. Arbitrary conditioning with (more classical) statistical models is of central importance, and the same applies to arbitrary conditioning with deep generative models. Consequently, this work is well motivated and has important applications. 
questions: List below contains both minor and major comments. 

1. In Background section where you introduce the notation x_u and x_o, it would be very helpful to specifically mention that you assume fully observed datasets. A reader may easily get disoriented that you are dealing with partially observed data. 

2. Equation. 1 in Section 3.2: From the point of view of the generative model, Eq. 1 is not exact but rather already an approximation of p(z|x_o) = p(x_o|z)p(z)/p(x_o) because your expression involves the variational approbation q(z|x). 

3. Equation 2: it would be useful to more clearly define q_{\theta}, e.g. by referring to Fig. 2 already at this stage. 

4. Equation 3: write already here that what are the parameters that you are optimizing. 

5. At a general level, as summarised in Fig. 2, the proposed model contains two branches: the "original" VAE and the "proposed model part" that includes the partially observed encoder and provides the posterior matching loss. The amortized variational approximation of the "proposed model part" takes as an input the "observed" variables x_o (used for conditioning) as well as a binary vector b that indicates which variables are missing/non-missing. This proposed model part seems very similar with the models proposed to learn VAEs from partially observed data in (Collier et al, 2020; Ipsen et al, 2021). These methods can learn approximations of the posteriors p(z|x_o,b) and thereby used to infer p(x_u|x_o). Please contrast your model with these two e.g. in Section 4, and include them in ablation studies (see below). 

REFS: 
Mark Collier, Alfredo Nazabal, Christopher K.I. Williams. VAEs in the Presence of Missing Data pdf  Published on arXiv 13 July 2020. Presented at the first Workshop on the Art of Learning with Missing Values (Artemiss) hosted by the 37th International Conference on Machine Learning (ICML 2020).

NB Ipsen, PA Mattei, J Frellsen, not-MIWAE: Deep Generative Modelling with Missing not at Random Data, International Conference on Learning Representations, 2021. 

6. Experiments and test metrics:
The general goal of the proposed method is to perform arbitrary conditioning p(x_u|x_o). I would have liked to see much more extensive experiments with variety of datasets that quantify performance directly using that metric, instead of devoting a large number of experiments to various other downstream tasks, where the contribution of the proposed model is less directly quantifiable. Current manuscript evaluate conditional log-likelihoods only in one experiment (Table 2). 

Related to the comment on conditional log-likelihood evaluations: authors evaluate their method on various downstream tasks using a variety of evaluation metrics (peak SNR, precision, recall, clustering accuracy). While the metrics themselves are well-known, the way they are computed and applied here is not explained in detail, which leaves it unclear that what the numbers exactly tell about the proposed method itself. 

7. Ablation studies:
I would have liked to see ablation studies. For example, it common to use the standard VAE or VAE with the Gaussian mixture prior and impute unobserved (missing) feature with zeros x_u = [0 0 ... 0], and then approximate the posterior directly using q(z|x_o,x_u), and then estimate p(x_u|x_o).  More recent methods explicitly model the missing features, such as the method by (Collier et al. 2020) and (Ipsen et al, 2021) (see comment above).  A comprehensive ablation study including e.g. these methods would provide more insight into the proposed model. 

Overall, this is an interesting study that addresses an important problem. The proposed method is intuitive and statistically well-motivated, and results appear promising. Regarding the novelty, I would like to see the proposed method to be compared by recent works that propose closely related methods.  An ablation study and more straightforward, direct comparisons on the arbitrary conditioning would strengthen the manuscript. 
limitations: Limitations are adequately addressed.
review_title: null

Meta-review: metareview: The paper presents a method for conditional generation of part of the data variable given the rest part, where the joint data distribution is defined by a trained VAE. The partition can be made arbitrary. A model standing alone from the VAE is trained. Extension for faster active feature acquisition is also presented. All reviewers acknowledged the significance of the task and the simplicity/flexibility and empirical effectiveness of the method. Notably multi-modality generation results are seen. Reviewers also expressed concerns on the notation, clarity (e.g., insufficient emphasis on requiring fully observed data), and conceptual and empirical comparison with similar methods. The authors have addressed most of them. In all, this paper makes an interesting contribution to the community.

The authors are encouraged to include the discussions with reviewers into the paper (particularly, relation and comparison with similar methods mentioned). It would make Theorem 3.1 more insightful if the authors could explain how the second term in Eq. (4) makes the method better than directly optimizing the arbitrary conditioning likelihood. Moreover, it seems the method requires $x_u$ and $x_o$ are conditionally independent given $z$ (so that $p(x_u | x_o, z) = p(x_u | z)$). Though Line 163 mentioned factorization, it is not for addressing this point.
confidence: Less certain
Decision: Accept