Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work aims to achieve disentangled representations for multi-view data. It explicitly summarizes the conditions for disentangled representations and proposed example solutions to achieve each of these criteria.

In my aspect, the three conditions for disentanglement is intuitional reasonable and can present insight for other works. However, I have the following concerns on this work:

1. the author presented the three conditions to be necessary conditions for disentanglement and argued them to be 'strict conditions'. However, I am afraid sufficient proof is needed to support these conditions to be necessary. In my opinion,  Information theory can be a candidate perspective to perform your proof, e.g. [1].

2. The proposed model achieves disentanglement on s and e with the product of the expert. So, will the design results in high computational cost at this phase? 

3.  The compared baseline methods are not up-to-date.  This work specifically targets disentanglement, it is not fair to compare vanilla CCA methods which simply considers the shard information. VCCA-private [2] can be a good candidate to compare here. 

4.  the work in [3] also specifically studies the multi-view disentanglement problem. Can you please discuss this method? It is also good to compare this method in the reconstruction visualization part.


[1] Gao, Shuyang, et al. "Auto-encoding total correlation explanation." The 22nd International Conference on Artificial Intelligence and Statistics. 2019.

[2].Wang, Weiran, et al. "Deep variational canonical correlation analysis." arXiv preprint arXiv:1610.03454 (2016).

[3]. Gonzalez-Garcia, Abel, Joost Van De Weijer, and Yoshua Bengio. "Image-to-image translation for cross-domain disentanglement." Advances in neural information processing systems. 2018.
review_title: Interesting motivation, but the major claim needs solid proof. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes definition and conditions for unsupervised multi-view disentanglement providing general instructions for disentangling representations between different views. The authors also provide a novel objective function to explicitly disentangle the multi-view data into a shared part across different views and a (private) exclusive part within each view. I have the following comments on the paper.

major comments
1. In literature, there are several ways to estimate mutual information, such as lower bound of JS divergence (Hjelm 2019, Federici 2020), InfoNCE etc. Also there are other types of mutual information estimators that maximize similarities between two views. Could you provide any indication which of them work better than the other in you model?

2. I think the proposed model presented in the paper is inspired by the paper by Gonzalez-Garcia in 2018, where the authors didn't use the maximization and minimization of mutual information. Why a similar criteria based on mutual information performs better than the distance metric?

3. Maximizing mutual information is often considered as a difficult task and needs complicated sampling strategy (negative, hard negative etc). How do you handle those issues in your setting? Some details will be helpful for the community.

4. Without a bottleneck, i.e. assuming unbounded capacity, maximizing mutual information can be trivially solved by setting the underlying function to identity (Xu Ji et al. ICCV 2019). Have you tried any bottlenecking in your case? If not, how do you ensure that the mutual information maximization does not results in a degenerated solution. Do you think including a bootlenecking would improve the results?

5. The experimental evaluation only shows experimental comparison with some baseline approaches. I think it is also worth to compare the proposed methods with the prior works (for example, Gonzalez-Garcia et al. NeurIPS 2018).

Based on my current understanding and the above comments, I currently recommend the paper as "marginally below acceptance threshold". I would like to hear clarification on the proposed models and if satisfied would be happy to increase my recommendation.

minor comments
1. In ICLR 2020, there were few works that proposed to learn mutual information from diverse domains. I think it is worth to provide to have a discussion on them.
(i) M. Federici et al., Learning Robust Representations via Multi-View Information Bottleneck, ICLR, 2020.
(ii) M. Tschannen et al., On Mutual Information Maximization for Representation Learning, ICLR, 2020.
2. I think it worth providing some details on the implementation and architectures in the paper. I would also recommend to share the code.
review_title: Clarifications on mutual information is needed

final_score: rating: 5: Marginally below acceptance threshold
review_content: For multi-view feature learning, this paper presents the definition as well as implementation for unsupervised multi-view disentanglement. Both consistency and complementarity are well investigated. Experiments on clustering and classification tasks validate the effectiveness.

Pros:
(1) The definition in 2.1 is quite straightforward, and the objective function in Eq. (1) is easy to understand.
(2) To measure the consistency and complementarity, this paper gives the solution based on mutual information.
(3) The presented experimental results look good.

Cons:
(1) I'm afraid that the consistency and complementarity have been investigated in multi-view clustering. [1,2] focus on affinity matrix based multi-view learning, and they already divided the affinity matrix of each view into the shared and the view-specific representations in an unsupervised manner. The difference lies in the format of input data and how to measure these properties. In view of this, the novelty of this paper is reduced.
(2) Please give more details about the optimization process.
(3) This paper focuses on multi-view learning. But the experiments only involve two views. How about the extension to more views. For example, you can conduct experiments on datasets with the number of views larger than 2 to validate the performance.
(4) In experiments, only s^{1], s^{2} and the concatenated representation are used for clustering or classification. Do you mean that the view-specific features are useless for the downstream tasks? Please give more explanation.

[1] Robust Multi-View Spectral Clustering via Low-Rank and Sparse Decomposition. AAAI 2014
[2] Exclusivity-Consistency Regularized Multi-view Subspace Clustering. CVPR 2017
review_title: Good definition and implementation for multi-view disentangled representation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The goal of this paper is to define multi-view disentanglement in an unsupervised manner. The authors list four principal rules for representation disentanglement, including completeness of combining shared and specific representations, the exclusivity of two specific representations and between specific and shared representation, and commonality of two shared representation. The authors follow the above rules to design a VAE-based model and demonstrate favorable results on image clustering and classification tasks.

*Strengths*
1. Clear definition of representation disentanglement: as the prior works do not clearly define all constraints, which are ought to be valid for representation disentanglement, the prior models might not able to disentangle the representations into disjoint representations. This paper clarifies the four conditions and formally define the problem with four mutual-information terms. This sets the standard objective function for future works.

2. While the proposed model is based on VAE with modifications, the proposed model can perform favorably against existing works as presented in Table 1-4.


*Questions*
1. Extension to supervised representation disentanglement: I am wondering how to extend the proposed model to the supervised setting so that the semantic of the disentangled feature can be learned for further manipulation? 

2. As beta-VAE is also designed for unsupervised disentanglement (although it is single-view), how beta-VAE performs in the tasks demonstrated in Table 1-4?

3. Generalization to more complicated data: I am wondering if the model is able to perform consistent improvement on a larger scale and more complicated dataset, as modeling mutual information in a higher dimension might be more difficult.
review_title: Clear definition of representation disentanglement

Meta-review: This paper focuses on disentangled representation learning from multi-view data, which is an interesting and hot topic. However, there are several papers published in the last couple of years (especially in NeurIPS2020 and ECCV2020) solving very similar problems with closely related contributions to this paper. The contributions of this paper compared to all recent works in this space is unclear. Contributions and benefits of individual components in the method are not investigated. Although the method is designed for multi-view settings, the authors run experiments on simple settings with only two views. The experiments seem quite limited and do not show the method's capabilities. The rebuttal does not properly address the reviewers' concerns either. 

The paper received four reviews with three recommending below acceptance threshold (rejection) and one above the acceptance threshold (although this one was the least confident scoring). Given all the above shortcomings and reviewer recommendations I do not recommend acceptance of the paper.



Decision: Reject