Contrast with Aggregation: A Scalable Framework for Multi-View Representation Learning

Authors: HyeongJoo Hwang, Ahn Young Jin, Heesan Kong, Seokin Seo, Seunghoon Hong, Kee-Eung Kim
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multi-View Representation Learning, Multimodal Representation Learning, Contrastive Learning

Abstract:
Multi-View Representation Learning (MVRL) aims to learn the joint representation from diverse data sources by discovering complex relationships among them.
In MVRL, since the downstream task information and the view availability are often unknown a-priori, it is essential for the joint representation to be robust to the partial availability of views.
However, existing methods exhibit various limitations, such as discarding potentially valuable view-specific information, lacking
the ability to extract representation from an arbitrary subset of views, or requiring considerable computational resources that increase exponentially with the number of views.
To address these challenges, we present a scalable MVRL framework based on contrastive learning.
Our approach employs a set of encoders that is able to extract representations from arbitrary subset of views, and jointly trains them with a computation cost that scales linearly with the number of views.
We conducted comprehensive evaluations across 7 MVRL benchmark datasets ranging from 2 to 8 views, demonstrating that our method robustly handles diverse input view combinations and outperforms strong baseline methods.

Decision: reject
Reviewer scores: 5, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730465812280

thread_type:
official_review

<thread_object>
creator: reviewer_48cg

<content>
summary:
This paper studies an innovative multi-view representation learning framework：Contrast with Aggregation: A scalable Framework for Multi-view Representation Learning. Existing multi-view representation learning methods have three main limitations: (1) discarding potentially valuable view-specific information; (2) inability to extract representations from arbitrary subsets of views; (3) computational costs exponentially increasing with the number of views. To address these issues, this paper proposes a scalable subset view representation learning framework based on contrastive learning. This framework reduces the number of encoder parameters by simplifying the existing encoder structure, thus lowering computational costs. It is capable of learning each subset view representation through a single objective function, making the computational cost grow linearly with the number of views. Finally, this paper introduces an operational lower bound to effectively calibrate the subset view representations based on contrastive learning. Experimental results demonstrate that this method can effectively handle various combinations of views across multiple benchmark datasets and outperforms other strong baseline methods.

strengths:
1.This paper is well-structured, with clear explanations of the problems with existing methods, the proposed solutions, and the experimental results. 

2.The motivation  to extract representation from an arbitrary subset of views  is commendable.

3.CwA proposes a scalable subset view representation learning framework based on contrastive learning. This framework serves as a new paradigm for multi-view representation learning, offering new insights into how different data views can enhance the model's learning capabilities.

weaknesses:
1.Please provide a detailed explanation of the relationship between the analysis method presented in this paper and the one in [1], and if possible, offer a comparison with [1].

2.This paper mainly addresses two issues: scalability and calibration. However, the experimental section does not perform calibration verification.

[1] Variational Distillation for Multi-View Learning, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, p 4551 - 4566.

questions:
See the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730514256755

thread_type:
official_review

<thread_object>
creator: reviewer_hony

<content>
summary:
This paper introduces Contrast with Aggregation (CwA), a scalable Multi-View Representation Learning (MVRL) framework designed to efficiently learn joint representations from diverse views. By leveraging contrastive learning with the InfoNCE objective and encoding subsets of views through Weighted Mixture of Experts (WMoE), CwA enables linear computational scaling relative to the number of views, allowing it to extract robust representations from any subset of views.

strengths:
1. Compared to traditional methods, this approach demonstrates greater adaptability and scalability across different combinations of views.
2. By simplifying multiple terms that maximize mutual information (MI) into a single term, this strategy reduces computational costs, making it more effective in multi-view scenarios.
3. The paper includes open-source code, which enhances reproducibility.

weaknesses:
1. The paper lacks sufficient theoretical justification for the choice of mutual information estimators and the use of specific distance metrics (Mahalanobis distance). More detailed theoretical explanation for this selection should be provided.
2. In the experiments, the paper notes that the method performs slightly worse than other baseline methods in single-view scenarios. While this issue improves with multiple view combinations, a thorough exploration of why CwA fails to surpass other methods in single-view cases and the implications for practical applications is necessary.
3. Although the paper conducts extensive experiments across various tasks, the selected datasets and task types are relatively narrow, primarily focusing on sarcasm and emotion recognition. Validation on more diverse datasets (e.g., ImageNet and NYU-Depth-V2) and tasks (e.g., action recognition on videos) should be explored to verify the method's effectiveness.
4. For the used datasets, the current advanced pretrained deep backbones should be employed to perform feature extraction for different data views instead of the somewhat handcrafted preprocessed features. More thorough studies are necessary to show its superiority.

questions:
Please refer to the weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730563447818

thread_type:
official_review

<thread_object>
creator: reviewer_vxpo

<content>
summary:
This paper presents Contrast with Aggregation (CwA) framework for multi-view representation learning that efficiently aggregates information from any subset of views using contrastive learning. CwA addresses computational and robustness challenges in multi-view learning by leveraging an information-theoretic objective and Mahalanobis distance to jointly train subset-view representations with a computation cost that scales linearly with the number of views. Extensive experiments demonstrate CwA's effectiveness and scalability, showing it outperforms baseline methods across multiple benchmark datasets source.

strengths:
(1) The paper tackles computational challenges in multi-view learning by using an information-theoretic objective and Mahalanobis distance to train subset-view representations jointly, achieving a computation cost that scales linearly with the number of views.

(2) The paper has presented extensive experiments showing the effectiveness and scalability of CwA. The paper is clearly written and different details are well understood.

weaknesses:
(1) The paper lacks an explicit statement of its contributions, leaving its main claims unclear. This should definitely be clarified in the introduction. Furthermore, the related works section does not sufficiently situate the proposed approach within the existing body of research. Without these clarifications, it is challenging to fully assess the paper.

(2) The paper could be seen as an extensive application of the inverse variance weighted (IVW) average for information fusion. However, it does not investigate whether this fusion method is optimal for various downstream tasks. Additionally, it would be valuable to explore whether this fusion approach is effective when applied with other metrics as well.

(3) Table 1 indicates that, for some datasets, performance achieved with a single modality surpasses that with combined views. For instance, on the MOSI dataset, CwA attains a classification accuracy of 67.23% using only the text modality, whereas combining all three views results in a lower accuracy of 65.61%, suggesting that information fusion may actually reduce model performance. This observation and a reasoned explanation is not discussed in the paper. So, I am looking for some insightful discussion to understand the reasons behind this observation.

questions:
Please see the weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612085782

thread_type:
official_review

<thread_object>
creator: reviewer_zrh5

<content>
summary:
This paper proposes a scalable multi-view representation learning (MVRL) framework named CwA based on contrastive learning to address the challenges in MVRL. It can effectively aggregate information from any subset of views. It optimizes each subset-view representation with a computational cost that increases linearly with the number of views, thus optimizing the computational cost. In addition, by integrating the Mahalanobis distance into the InfoNCE objective, this article reformulates their method as a contrastive learning (CL) approach for calibrating each subset-view representation. It has achieved good results on multiple datasets.

strengths:
They introduced CwA, a scalable Multi View Representation Learning (MVRL) framework that can effectively aggregate information from any subset of views. Not only did it reduce the complexity of some models, decrease the complexity of parameter count and training time, but it also achieved better results than most related works.

weaknesses:
1.This paper describes the reduction in the number of encoder parameters, but lacks a comparison of the number of encoder parameters in the paper. The effectiveness and necessity of the paper's method cannot be verified through poor experiments.
2.The explanation of related work in the paper is relatively simple, and it is relatively difficult for non professionals to understand the paper.  Besides, the role of the encoder and the significance of the operations performed in this article are not clear enough, and the description of the methods section is not concise enough. A large amount of space is provided to supplement the explanation, which I think is unreasonable.
3. This article proposes that the model can be modal extended, but I did not see any difference in this scalability compared to previous models in the paper. Please clarify with the submitter.

questions:
1.Have you ever considered why the model performs relatively poorly in a single modality? Is it due to hyperparameters?
2.Are the results obtained by other methods tested by the author themselves on the server or published in other papers? Does the author think it has any reference value?
3.Has the author considered the performance of the model in subsequent tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732196101142

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal by Authors

comment:
# General Response

We sincerely appreciate the reviewers for their constructive and thoughtful feedback, which have greatly contributed to improving the quality of this work.

In our responses to the reviews, we have carefully addressed all raised concerns. These can be summarized as follows:

- **Additional experiments** (hoNY): We conducted experiments on ImageNet-100 to further validate our method and provided results demonstrating its effectiveness in realistic data.
- **Theoretical justification** (hoNY): We elaborated on the rationale for selecting the InfoNCE objective as the MI estimator and the Mahalanobis distance as the critic function, emphasizing their suitability and advantages in our context.
- **Optimality of IVW fusion** (Vxpo): We provided a detailed explanation on the inverse variance-weighted (IVW) average, showing its statistical grounding and addressing concerns regarding its optimality.
- **Discussion on single-view performance** (zRh5, hoNY): We discussed why our method performs relatively weakly with single views, while highlighting its robustness with multi-view combinations.

- **Relationship to prior work** (zRh5, Vxpo, 48cg): We clarified how our method builds upon and extends concepts introduced by Multi-View VAEs and Contrastive MVRL approaches, situating our work within the context of related research.

Due to space limitations, we are actively revising our manuscript to carefully incorporate all the valuable feedback from the reviewers. This ongoing process includes:
- Adding further explanations and clarifications to Sections 1, 2, and 3.
- Expanding experimental details, including descriptions of the four datasets from MultiBench and additional results from ImageNet-100.
- Enhancing the overall presentation to improve clarity and accessibility.

We are fully committed to completing these revisions as soon as possible and will promptly notify reviewers once the updated manuscript is available. If there are any remaining concerns or suggestions, we welcome the opportunity to address them during the ongoing discussion period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732463575279

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated Manuscript

comment:
Dear Reviewers,

We sincerely appreciate your valuable feedback and have carefully incorporated your suggestions into the revised manuscript.\
The updates are highlighted in blue text for clarity.

Below is a summary of the key changes:

- Explicit statement of **contributions** in Section 1 (Vxpo).
- **Positioning of our work** within the MVRL context in Section 2 (Vxpo).
- Justification for the **InfoNCE objective and Mahalanobis distance** in Section 3.3 (hoNY).
- Detailed **explanations on the 4 datasets** from MultiBench in Section 4.2 (zRh5).
- Discussion on the **lower performance of combined views** in Section 4.2 (Vxpo).
- New experiment results on **ImageNet-100** in Section E.6 (hoNY).
- Discussion on the **optimality of IVW** within MVRL in Section G (Vxpo).

We welcome further discussion on any remaining concerns and are happy to incorporate additional suggestions to further improve our work.

Thank you for your thoughtful feedback and support throughout the review process.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733202668373

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Rebuttal and Revision Status (D-7h)

comment:
Dear Reviewers,

We believe that we have thoroughly addressed all comments from reviewers through our responses and the revised manuscript.\
Below, we summarize the updates made to address the issues raised:

- Conducted an analysis of what is learned and ignored in representations using the controlled synthetic dataset (Vxpo), while clarifying the significance of synthetic datasets in evaluating multi-view representation learning (zRh5).
- Performed additional experiments on ImageNet-100 as requested, demonstrating the effectiveness of our method on realistic datasets (hoNY).
- Provided a detailed explanation of the theoretical and algorithmic contributions of the method, emphasizing the novelty of its information-theoretic objective (Vxpo).
- Justified the use of IVW-based encoders, the InfoNCE objective, and Mahalanobis distance as the critic function (Vxpo, hoNY).
- Discussed challenges in defining modality informativeness and selecting the optimal method-dataset combination due to the unsupervised nature of multi-view representation learning (Vxpo).
- Addressed concerns about performance inconsistencies in MultiBench and explained how these reflect common challenges in multi-view representation learning (zRh5).

In addition, we carefully revised the manuscript with the following changes:
- Explicitly stated contributions in the introduction for clarity.
- Included new experimental results on ImageNet-100.
- Added an analysis of what is learned and ignored in representations using the synthetic dataset, with revised explanations to provide further insights.
- Included justification for using IVW-based encoders, the InfoNCE objective, and Mahalanobis distance as the critic function.

With the deadline now less than 7 hours away, we kindly remind reviewers that we remain available to promptly address any remaining concerns.\
If you feel that your concerns have been adequately resolved, we would greatly appreciate it if you could consider revisiting your review score.

Thank you,\
The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a contrastive learning-based multi-view representation learning (MVRL) method designed to perform effectively even when some data sources (or "views") are missing. Experiments were conducted on seven datasets containing between 2 and 8 views. The reviewers' ratings were highly diverse, though slightly positive on average. 

However, there are notable limitations. The multi-view constraint in Eq. (1) relies solely on a pairwise solution, which increases complexity as the number of views grows. Additionally, the sub-sampling mechanism lacks novelty, as multi-view data often vary significantly in perspective, with each view contributing differently. This approach may fail to capture intuitive relationships across views. Reviewer zRh5 highlighted inconsistencies in the results, such as the 3-view configuration in Table 1 performing worse than the 2-view configuration for the MOSI dataset.