Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper motivates a variational objective (eq. (7)) for multi-view representation learning. The proposed objective is an alternative to the standard multi-view VAE objective, and should be better suited for extracting the correct representation when views are missing at random and when a subset of the views is overwhelmingly informative. They then propose optimising a convex combination of their proposed objective and the standard multi-view VAE objective.
main_review: The problem tackled by the author seems well motivated, since multi-view representation learning in a partial setting (with missing views) is an important problem.
The paper is generally well structured and easy to read.

The definition of “complete representation” plays a crucial role in the paper, but is never stated formally. Adding a dedicated “Definition” paragraph would be useful for the reader. For example, as done in sec. 2.1 of:

Zhang, Changqing, et al. "CPM-Nets: Cross partial multi-view networks." (2019).

Lines 128-129 are very puzzling. After motivating and deriving objective (7), the authors say they “sometimes observed that the model is prone to overfitting”, and suggest instead optimising a convex combination of their proposed objective and the usual multi-view variational ELBO, favouring a minimally sufficient encoding. 
The authors allude to a tradeoff between minimal sufficiency and completeness, but this is not extensively commented upon in the paper, despite apparently constituting a fundamental point for how it informs their final objective. The feeling is that something is missing here, both in theory (characterising this tradeoff, relating it to properties of the ground truth data generating process, etc.) and in experiments (e.g. the authors say “we observed that…”, which seems to refer to some unreported experiments).

I need to say, I am not an expert on the related literature (e.g. references [34] and [35] cited in the paper), so my confidence level in assessing its quality and novelty within that context is not extremely high. Overall I am under the impression that the paper, while being well motivated and written, is not very convincing in some parts, and that some key aspects would require more elaboration and possibly a more rigorous treatment, as detailed above.

Typos and minor comments:

References [12] and [13] are the same?

Line 191: “variable.Deep” — space missing after dot

The captions for figures 1, 2 and 3 could be made more informative.

**Update** I will increase my evaluation to 6 based on the authors' answer.
limitations_and_societal_impact: The authors address this in the supplementary material.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes the variational-based generative approach to learn the multi-view (multi-modal) representation. The whole framework is motivated and derived from maximizing the total correlation  reduced by the representation, it focuses on learning a shared latent representation that is informative as well as succinct to capture the correlations among the given views (domains). Specifically, the framework is based on VAE framework with product-of-expert posterior encoder, and is trained via extended ELBO with additional regularization KL between joint encoder and view-specific encoder. The authors verify their proposed model on multi-view classification and translation. They also conduct experiments on partial multi-view representation learning where several views are missing in the training, and need to be inferred during testing.  
main_review: For originality: 

 (+) the idea presented makes sense and interesting, though I feel the whole model is actually built on the MVAE with extra conditional VIB term as in eqn. 7. The paper also describes the difference with the existing relevant works. 

For quality: 

 (+) the authors consider various tasks (e.g., complete-views/partial views, translation/classification) for verification of their model. 

 (-) however, it's better to also include some "standard" dataset for direct comparisons. For example, the existing baseline MMVAE use pair data from SVHN-MNIST for multi-domain translation/prediction (they also include MVAE performance), their models are well-tuned on such data for better performance and could be more illustrative.  Since the main difference is the training objective, if authors could use similar net structures to obtain results on such benchmark, that could easily give us direct and convincing comparisons. 

 (-) perhaps the word "multi-view" is slightly misleading.  There are tons of multi-view representation learning papers in the literature that are mainly focused on face images under multiple viewpoints . This paper seems instead focused on multi-modal(domain) learning. Does the proposed model capable of handling face images under different view-points? Does the paper flexible enough to deal with raw pixels instead of extracted features on Multi-PIE data (for example)? It would be interesting to show the multi-view translation results on face (such as given one view point, use the learned network/representation to generate the face under novel view point). 

For clarity: 

 (-) the paper can be not so easy to follow. Such difficulty is not coming from the math derivation, but from the presentation of the paper. Quite a few symbols/notations are used without properly definition and descriptions beforehand. For instance: line 83, $r(z) \approx p_\theta(z)$, what is this $p_\theta(z)$? is it empirically marginal distribution based on observation or is it simply prior distribution of your decoder? If the latter, use $\theta$ indicates you have learnable prior? Another example would be $r_\phi$ in eqn. 7, when author write down eqn. 7, it didn't clearly show us what is the relationship between $p_\theta(z|o)$ and $r_\phi$. So have to go back and forth to check some notations in the previous sections. Changing the presentation would greatly improve the paper and make it more accessible. 

For significance: 

 (+) the new conditional VIB term might have some interesting properties and may be applicable for follow-up development. 



limitations_and_societal_impact: The limitation of the work could be described more clearly.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to measure the quality of a multi-view generative model by the amount of total correlation among observations that is explained by the latent complete representation. While directly maximizing the reduction in total correlation is intractable, the authors derived a variational lower bound as a surrogate objective which encourages the model to learn a minimal sufficient representation and also regularizes the latent representation to be inferable from any view. The proposed formulation works well when some views are missing during training or testing. Hence it's suitable for partial multi-view representation learning. It also naturally calibrates the view-specific encoders to prevent the model heavily relying on only the most informative views. Empirical evaluation shows that the proposed method performs noticeably better than its baselines on most multi-view classification problems and multi-view translation tasks while achieves competitive results on a few others.
main_review: Originality: the idea of using the reduction of total correlation as the objective of learning auto-encoder-like models has been studied in the literature. The authors have also cited related work in the submission. The originality of this paper comes from applying total correlation to multi-view representation learning and deriving a novel variational lower bound that is suitable for learning with partial observations.

Quality: the derivation of the proposed objective looks solid. But i haven't carefully checked the details in the supplementary material. Empirical study on the proposed method is pretty thorough. The paper can be strengthened if the authors can discuss about the computational complexity of their new method and the baselines. The choice of dimension of latent space is not covered in the main text either. It is not clear to me how much tuning effort the proposed method requires.

Clarity: this paper is well written in general but can be certainly polished further. On L136-137, the definition of encoder and decoder is confusing to me. The encoder, $r_{\psi}^v (z | o_v) = N(\mu_v, \sigma_v^2 I)$, is a Gaussian distribution that does not involve $o_v$, which feels unintuitive. The same for the decoder $q_{\psi}^v(o_v | z)$.

Significance: the novel variational lower bound and its application to partial multi-view representation learning appear to be a solid contribution to this domain. I am not very familiar with the state-of-the-art results on the related tasks. But, based on the experimental results, the proposed method performs well in practice and advances the best results further, though the amount of improvement varies by task.
limitations_and_societal_impact: The authors have mentioned potential societal impact of this work in the supplementary material.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a novel Multi-View Representation Learning method to learn a shared representation of multiple observations. The objective is to learn a complete representation through maximizing the reduction of total correlation. The method is based on the VAE framework and is capable of dealing with missing views. 
main_review: *originality*

The paper proposes a novel objective function for Multi-View Representation Learning (MVRL). The objective optimizes a lower bound of the total correlation between the views and the latent representation. 

*quality*
- In Eq. (6), you mention "the latent representation is regularized to be inferrable from any view and thus cannot be dominated by a few one". But the MI terms could still be high for some views and low for some other views. You are taking the average in the end anyway. I don't think this solves the imbalance issue.
- In Eq. (7), it seems that both $p(z|o)$ and $r(z|o_v)$ are learnable posteriors. Would this cause any training instability?
- PoE seems to be an aggregation of all the individual posteriors and definitely has a larger support than individual ones. This could also bring some numerical concerns. How did you address this issue?
- When you say "VIB in Eq. (5) favors the minimal sufficient encoding", I'm ok with the minimal due to the minus. But I don't think this term really encourages "sufficient". This term could potentially cause "insufficiency".
- For the KL terms, did you use analytical form or sample estimation?
- In Fig. 1, MMVAE's performance cannot be improved with more views. What might cause this?
- In Fig. 3, there seems to be no significant gap between MVTCVAE and other unsupervised methods (mmJSD, MoPoE). Besides, as the missing rate increases, the performance drop trends are quite similar. Then I wonder if carefully tuning other methods could achieve the same goal.

*clarity*
- Authors should really put Fig. 4 to the main text rather than the appendix. 
- When you bring up the limitations "Missing views", as a reader, I thought you would immediately explain how you address this. But I have to wait till the end of section 2.4. I would suggest you explaining section 2.4 first. 

*significance*
It is novel to propose such an idea for MVRL. But maybe authors can demonstrate more scenarios besides the accuracies.

limitations_and_societal_impact: I think the writeup could be improved, especially the order of different modules. Besides, Fig. 4 should be moved to the main text with more descriptions. The whole pipeline would be more clear.
review_title: null

Meta-review: This paper proposes a total correlation objective within VAE framework for multi-view representation learning, and the proposed method can handle situations with missing views. 

The main concern of the reviewers is with the presentation and clarity. 

The overall assessment is that this is an incrementally novel contribution with incrementally improved experimental results. 
Decision: Accept (Poster)