Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes variational selective autoencoders (VSAE) to learn the joint distribution model of full data (both observed and unobserved modalities) and the mask information from arbitrary partial-observation data. To infer latent variables from partial-observation data, they introduce the selective proposal distribution that switches encoders depending on whether each input modality is observed.

This paper is well written, and the method proposed in this paper is nice. In particular, the idea of the selective proposal distribution is interesting and provides an effective solution to deal with the problem of missing modality in conventional multimodal learning. The experiment is also well structured and shows higher performance than the existing models.  However, I have some questions and comments, so I’d like you to answer them.

Comments:
- The authors state that x_j is sampled from the "prior network" to calculate E_x_j in Equation 10, but I didn’t understand how this network is set up. Could you explain it in detail?
- The authors claim that adding p(m|z) to the objective function (i.e., generating m from the decoder) allows the latent variable to have mask information. However, I don’t know how effective this is in practice. Specifically, how performance differs compared to when p (m | z) is not used and the decoder p (x | z, m) is conditioned by the mask included in the training set instead of the generated mask?
- Why did you not do image inpainting in higher-dimensional experiments like Ivanov et al. (2019), i.e., considering each pixel as a different modality? Of course, I know that Ivanov et al. require the full data as input during training, but I’m interested in whether VSAE can perform inpainting properly even if trained given imperfect images.
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: The paper proposed variational selective autoencoders (VSAE) to learn from partially-observed multimodal data.  Overall, the proposed method is elegant; however, the presentation, the claim, and the experiments suffer from significant flaws. See below for detailed comments.

[Pros]
1. The main idea of the paper is to propose a generative model that can handle partially-observed multimodal data during training. Specifically, prior work considered non-missing data during training, while we can't always guarantee that all the modalities are available. Especially in the field of multimodal learning, we often face the issue of imperfect sensors. This line of work should be encouraged. 

2. In my opinion, the idea is elegant. The way the author handles the missingness is by introducing an auxiliary binary random variable (the mask) for it. Nevertheless, its presentation and Figure 1 makes this elegant idea seems over-complicated.
[Cons]

1. [The claim] One of my concerns for this paper is the assumption of the factorized latent variables from multimodal data. Specifically, the author mentioned Tsai et al. assumed factorized latent variables from the multimodal data, while Tsai et al. actually assumed the generation of multimodal data consists of disentangled modality-specific and multimodal factors. It seems to me; the author assumed data from one modality is generated by all the latent factors (see Eq. (11)), then what is the point for assuming the prior of the latent factor is factorized (see Eq. (4) and (5))? One possible explanation is because we want to handle the partially-observable issues from multimodal data, and it would be easier to make the latent factors factorized (see Eq. (6)). The author should comment on this. 

2. [Phrasing.] There are too many unconcise or informal phrases in the paper. For example, I don't understand what does it mean in "However, if training data is complete, ..... handle during missing data during test." Another example would be the last few paragraphs on page 4; they are very unclear. Also, the author should avoid using the word "simply" too often (see the last few paragraphs on page 5). 

3. [Presentation.] The presentation is undesirable. It may make the readers hard to follow the paper. I list some instances here. 
		a. In Eq. (3), it surprises me to see the symbol \epsilon without any explanation. 
		b. In Eq. (6), it also surprises me to see no description of \phi and \psi. The author should also add more explanation here, since Eq. (6)  stands a crucial role in the author's method.
		c. Figure 1 is over-complicated.
		d. What is the metric in Table 1 and 2?  The author never explains. E.g., link to NRMSE and PFC to the Table. 
		e. What are the two modalities in Table 2? The author should explain.
		f. The author completely moved the results of MNIST-SVHN to Supplementary. It is fine, but it seems weird that the author still mentioned the setup of MNIST+SVHN in the main text. 
		g. The author mentioned, in Table , the last two rows serve the upper bound for other methods. While some results are even better than the last two rows. The author should explain this.
		h. Generally speaking, the paper does require a significant effort to polish Section 3 and 4. 

4. [Experiments.] The author presented a multimodal representation learning framework for partially-observable multimodal data, while the experiments cannot corraborrate the claim. First, I consider the tabular features as multi-feature data and less to be the multimodal data. Second, the synthetic image pairs are not multimodal in nature. These synthetic setting can be used for sanity check, but cannot be the main part of the experiments. The author can perhaps consider the datasets used by Tsai et al. There are seven datasets, and they can all be modified to the setting of partially-observable multimodal data. Also, since the synthetic image pairs are not multimodal in nature, it is unclear to me for what the messages are conveyed in Figure 3 and 4. 


I do expect the paper be a strong submission after a significant effort in presentation and experimental designs. Therefore, I vote for weak rejection at this moment.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper proposes a novel training method for variational autoencoders that allows using partially-observed data with multiple modalities. A modality can be a whole block of features (e.g., a MNIST image) or just a single scalar feature. The probabilistic model contains a latent vector per modality. The key idea is to use two types of encoder networks: a unimodal encoder for every modality which is used when the modality is observed, and a shared multimodal encoder that is provided all the observed modalities and produces the latent vectors for the unobserved modalities. The whole latent vector is passed through a decoder that predicts the mask of observed modalities, and another decoder that predicts the actual values of all modalities. The “ground truth” values for the unobserved modalities are provided by sampling from the corresponding latent variables from the prior distribution once at some point of training.

While I like the premise of the paper, I feel that it needs more work. My main concern is that sampling the target values for the unobserved modalities from the prior would almost necessarily lead to blurry synthetic “ground truth” for these modalities, which in turn means that the model would produce underconfident predictions for them. The samples from MNIST in Figure 3 are indeed very blurry, supporting this. Furthermore, the claims of the model working for non-MCAR missingness are not substantiated by the experiments. I believe that the paper should currently be rejected, but I encourage the authors to revise the paper.

Pros:
* Generative modelling of partially observed data is a very important topic that would benefit from fresh ideas and new approaches
* I really like the idea of explicitly modelling the mask/missingness vector. I agree with the authors that this should help a lot with non completely random missingness.

Cons:
* The text is quite hard to read. There are many typos (see below). The text is over the 8 page limit, but I don’t think this is justified. For example, the paragraph around Eqn. (11) just says that the decoder takes in a concatenated latent vector. The MNIST+SVHN dataset setup is described in detail, yet there is no summary of the experimental results, which are presented in the appendix.
* The approach taken to train on partially-observed data is described in three sentences after the Eqn. (10). The non-observed dimensions are imputed by reconstructions from the prior from a partially trained model. I think that this is the crux of the paper that should be significantly expanded and experimentally validated. It is possible that due to this design choice the method would not produce sharper reconstructions than the original samples from the prior. Figures 3, 5 and 6 indeed show very blurry samples from the model. Furthermore, it is not obvious to me why these prior samples would be sensible at all, given that all modalities have independent latents by construction.
* The paper states multiple times that VAEAC [Ivanov et al., 2019] cannot handle partially missing data, but I don’t think this is true, since their missing features imputation experiment uses the setup of 50% truly missing features. The trick they use is adding “synthetic” missing features in addition to the real ones and only train on those. See Section 4.3.3 of that paper for more details.
* The paper states that “it can model the joint distribution of the data and the mask together and avoid limiting assumptions such as MCAR”. However, all experiments only show results in the MCAR setting, so the claim is not experimentally validated.
* The baselines in the experiments could be improved. First of all, the setup for the AE and VAE is not specified. Secondly, it would be good to include a GAN-based baseline such as GAIN, as well as some more classic feature imputation method, e.g. MICE or MissForest.
* The experiments do not demonstrate that the model learns a meaningful *conditional* distribution for the missing modalities, since the provided figures show just one sample per conditioning image.

Questions to the authors:
1. Could you comment on the differences in your setup in Section 4.1 compared to the VAEAC paper? I’ve noticed that the results you report for this method significantly differ from the original paper, e.g. for VAEAC on Phishing dataset you report PFC of 0.24, whereas the original paper reports 0.394; for Mushroom it’s 0.403 vs. 0.244. I’ve compared the experimental details yet couldn’t find any differences, for example the missing rate is 0.5 in both papers.
2. How do you explain that all methods have NRMSE > 1 on the Glass dataset (Table 1), meaning that they all most likely perform worse than a constant baseline?

Typos and minor comments:
* Contributions (1) and (2) should be merged together.
* Page 2: to literature -> to the literature
* Page 2: “This algorithm needs complete data during training cannot learn from partially-observed data only.”
* Equations (1, 2): z and \phi are not consistently boldfaced
* Equations (4, 5): you can save some space by only specifying the factorization (left column) and merging the two equations on one row
* Page 4, bottom: use Bernoulli distribution -> use factorized/independent Bernoulli distribution
* Page 5, bottom: the word “simply” is used twice
* Page 9: learn to useful -> learn useful
* Page 9: term is included -> term included
* Page 9: variable follows Bernoulli -> variable following Bernoulli
* Page 9: conditions on -> conditioning on
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes to impute multimodal data when certain modalities are present. The authors present a variational selective autoencoder model that learns only from partially-observed data. VSAE is capable of learning the joint
distribution of observed and unobserved modalities as well as the imputation mask, resulting in a model that is suitable for various down-stream tasks including data generation and imputation. The authors evaluate on both synthetic high-dimensional and challenging low-dimensional multimodal datasets and show improvement over the state-of-the-art data imputation models.

Strengths:
- This is an interesting paper that is well written and motivated.
- The authors show good results on several multimodal datasets, improving upon several recent works in learning from missing multimodal data.

Weaknesses:
- How multimodal are the datasets provided by UCI? It seems like they consist of different tabular datasets with numerical or categorical variables, but it was not clear what the modalities are (each variable is a modality?) and how correlated the modalities are. If they are not correlated at all and share no joint information I'm not sure how these experiments can represent multimodal data. 
- Some of the datasets the authors currently test on are quite toy, especially for the image-based MNIST and SVHN datasets. They should consider larger-scale datasets including image and text-based like VQA/VCR, or video-based like the datasets in (Tsai et al., ICLR 2019).
- In terms of prediction performance, the authors should also compare to [1] and [2] which either predict the other modalities completely during training or use tensor-based methods to learn from noisy or missing time-series data.
- One drawback is that this method requires the mask during training. How can it be adapted for scenarios where the mask is not present? In other words, we only see multiple modalities as input, but we are not sure which are noisy and which are not?

[1] Pham et al. Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities, AAAI 2019 
[2] Liang et al. Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization, ACL 2019

### Post rebuttal ###
Thank you for your detailed answers to my questions.
review_title: Official Blind Review #2

Meta-review: This submission proposes a VAE-based method for jointly inferring latent variables and data generation. The method learns from partially-observed multimodal data.

Strengths:
-Learning to generate from partially-observed data is an important and challenging problem.
-The proposed idea is novel and promising.

Weaknesses:
-Some experimental protocols are not fully explained.
-The experiments are not sufficiently comprehensive (comparisons to key baselines are missing).
-More analysis of some surprising results is needed.
-The presentation has much to improve.

The method is promising but the mentioned weaknesses were not sufficiently addressed during discussion. AC agrees with the majority recommendation to reject.

Decision: Reject