An Information Criterion for Controlled Disentanglement of Multimodal Data

Authors: Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, Caroline Uhler
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multimodal Representation Learning, Disentanglement, Self-Supervised Learning, Information Theory

Abstract:
Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose $\textbf{Disentangled}$ $\textbf{S}$elf-$\textbf{S}$upervised $\textbf{L}$earning (DisentangledSSL), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called $\textit{Minimum Necessary Information}$ (MNI) point is not attainable. We demonstrate that \algo successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730528174557

thread_type:
official_review

<thread_object>
creator: reviewer_bzkx

<content>
summary:
This paper addresses the disentanglement of modality-shared and modality-specific information in multi-modal self-supervised learning and proposes DisentangledSSL. From an information-theoretic perspective, the modality-shared information is optimized using a conditional entropy bottleneck (CEB). Correspondingly, the authors formulate an optimization problem to isolate modality-specific information. Theoretical analysis of the optimality of each disentangled representation, particularly when Minimum Necessary Information is unattainable, along with experimental results, demonstrates the superiority of DisentangledSSL.

strengths:
1.The idea of applying conditional entropy bottleneck (CEB) in multi-modal self-supervised learning is novel.

2.Comprehensive theoretical analysis is conducted to prove the optimality of both modality-specific and modality-shared information.

3.DisentangledSSL demonstrates superior performance across tasks, including vision-language prediction and molecule-phenotype retrieval.

weaknesses:
1.How does the optimization objective for the modality-shared representation change from Eq. (1) to Eq. (3)? Why is $I(X^1;X^2)$ omitted?

2.Related studies, such as CoCoNet [1] and SimMMDG [2], which also address the separation of modality-shared and modality-specific information, are recommended for inclusion and comparison. Note that SimMMDG [2] can be easily adapted to a self-supervised setting by substituting supervised contrastive learning with the original self-supervised counterpart.

3.Ablation studies on the impact of $\beta$ and $\lambda$ in the MultiBench datasets should be provided to demonstrate the importance of separating modality-specific and modality-shared information in real-world applications. 

4.Some typos.
1)$Z^2$ should be $X^2$ in Eq.(6) and in the equation below Eq.(6).
2)$I(X^1;Z^1)$ should be $I(X^1;X^2)$ in Figure 3.
 
[1] Li J, Qiang W, Zheng C, et al. Modeling multiple views via implicitly preserving global consistency and local complementarity. TKDE, 2022.
[2] Dong H, Nejjar I, Sun H, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization. NIPS, 2023.

questions:
Are there any alternative approaches for implementing the $I(Z^1;\hat{Z}_{c}^{1*})$ in $L_s^1$? If so, how does their performance compare to that of the orthogonal loss implementation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612030143

thread_type:
official_review

<thread_object>
creator: reviewer_dhvl

<content>
summary:
The authors believe that "shared information between modalities is precisely what is relevant for downstream tasks" and "the modality gap ... results in misalignment between modalities, restricting the application of these methods in numerous real-world multimodal scenarios." Hence they motivate "the need for a disentangled representation space that captures both shared and modality-specific information". To address this need, they propose DISENTANGLEDSSL, an information-theoretic framework designed to learn disentangled shared and modality-specific representations of multimodal data. The authors have conducted a theoretical analysis to evaluate the quality of disentanglement, which can be applied even in cases where Minimum Necessary Information (MNI) is unattainable. The efficacy of DISENTANGLEDSSL is demonstrated across a range of synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.

strengths:
It makes sense to learn disentangled shared and modality-specific representations of multimodal data. 

The theoretical analysis to evaluate the quality of disentanglement, which can be applied even in cases where Minimum Necessary Information (MNI) is unattainable, is provided.

The efficacy of DISENTANGLEDSSL is demonstrated across a range of synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.

weaknesses:
Some of the key assertions motivating this work are inaccurate, and the proposed graphical model is flawed, which together raise questions about the validity of the resulting framework.

For instance, the statement "shared information between modalities is precisely what is relevant for downstream tasks" is not universally accurate. What is relevant depends heavily on the specific downstream tasks. Furthermore, "shared information" is a vague term, especially when applied across modalities, making it problematic in this context.

The assertion that "the modality gap ... results in misalignment between modalities, restricting the application of these methods in numerous real-world multimodal scenarios" is also oversimplified. The modality gap is not the sole or primary cause of misalignment; other factors such as alignment methods, distributional shifts, domain shifts, sampling biases, and more can contribute significantly.

I suggest the authors to refine their motivation to more accurately reflect the complexities of multimodal relationships, while still maintaining the core idea of disentangling shared and modality-specific information.

The graphical model in Figure 2, intended to represent the generative process (as claimed in line 128), has two major issues. First, the authors have conflated the generative and inference processes. Only the pathway from Z to X represents the generative model, while the path from X to \hat{Z} corresponds to inference, which should not be included in the data-generating graphical model. This issue is straightforward to resolve by removing the inference process from the model, which would not affect the algorithm or results, or to create separate diagrams, one for the generative process and the other for inference to clarify.

The second issue is more severe: the assumption of a shared latent variable Z_c existing between two modalities may not hold. This assumption lacks foundation, as it is more likely that the relationship takes the form of two variables Z_c^1 and Z_c^2, with potential connections between them:  (1)  Z_c^1 -> Z_c^2, (2)  Z_c^1 <- Z_c^2, or (3) or Z_c^1 <-C-> Z_c^2, depending on the modalities involved. This discrepancy significantly undermines the justification for the proposed algorithm.  I encourage the authors to discuss the implications of using separate but potentially connected latent variables (Z_c^1 and Z_c^2) instead of a single shared Z_c, and how might this change affect the proposed algorithm and results. This could lead to a more nuanced and realistic model of multimodal relationships.

questions:
Please respond to the weaknesses listed above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672238664

thread_type:
official_review

<thread_object>
creator: reviewer_h7jq

<content>
summary:
In this paper, the authors proposed a novel self-supervised representation learning method for multimodal representation learning. The proposed method disentangles the representation of multimodal information into shared information and modality-specific information, and uses a separate neural network to learn each representation. The authors supported their proposed method through extensive theoretical analysis and proofs. The proposed method was evaluated on both synthetic datasets and real-world multimodal datasets, where the proposed method achieved top performance over baselines in most tasks.

strengths:
1. The paper introduces a new perspective in how to perform effective multimodal representation learning. Specifically, the paper demonstrated that disentanglement of modality-specific and shared information into 2 separate representations can be effective in improving downstream task performance, which seems to be original and novel.

2. The authors provided extensive theoretical justifications and proofs for their proposed approach.

3. The experiments are quite comprehensive. They include both synthetic and real-world datasets, and the proposed method is evaluated against several multimodal SSL baselines on different tasks from drastically different domains. The proposed method achieved top performance in almost all tasks.

4. All experiment details are presented in the Appendix (high reproducibility).

weaknesses:
While the extensive use of information-theory notations allowed rigorous proof of the proposed approach, it isn't really easy for readers who isn't interested in the proofs and just want to learn about the concrete algorithm/methodology. Perhaps the authors should consider including a Pseudocode/Alg block either in the main text or in Appendix to clearly demonstrate the training process.

questions:
Although in most cases DisentangledSSL-both achieves highest performance, there are also several tasks where either  DisentangledSSL-shared or  DisentangledSSL-specific alone achieves top performance (and sometimes by quite a large margin over  DisentangledSSL-both, such as Mustard). Do you observe any trend / suggest any guidelines for when it is best to use each configuration?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730741721556

thread_type:
official_review

<thread_object>
creator: reviewer_njhk

<content>
summary:
The paper introduces a self-supervised learning approach to disentangle shared and modality-specific information in multimodal data. The authors also explain that the optimal case in prior work is not doable based on Minimum Necessary Information (MNI), as the comprehensive analysis of the optimality. The experiments on vision-language and molecule-phenotype retrieval tasks demonstrate its effectiveness.

strengths:
1. The paper presents a rigorous, information-theoretic framework for disentangling data for different domains and modalities under conditions where MNI is unattainable.

2. Based on the theoretical analysis, the authors designed a two-step training algorithm, and provided an optimality guarantee for this proposed method theoretically.

weaknesses:
Weakness:
For the experiments on synthetic data, they only contain the data for unattainable MNI. The synthetic experiments for attainable MNI are also essential here to demonstrate the reliability of the theoretical bound. My suggestion is to add Gaussian noise produced by different variances as the degrees of MNI attainable or unattainable.


1. In Figure 4 (b), the superiority of DISENTANGLEDSSL is not significant compared to other baselines, such as JointOPT. The hyperparameter front of JointOPT is closer to the front of DISENTANGLEDSSL.

2. Table 1 shows the three versions of DISENTANGLEDSSL are unstable. For example, DISENTANGLEDSSL (shared) can achieve promising performance on MOSI while failing on MUSTARD. It would be better to explain why such a phenomenon exists, probably focusing on the main difference between these three versions and properties of multi-modal data. 

3. The performance gain shown in Table 2 is quite limited. Could the author explain if small improvements might be important in this domain? In addition, it would be better if the authors could provide the reason for this tiny improvement. 

4. The paper does not satisfy the page limitation.

questions:
Please check the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731999562300

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Consolidated response to all reviewers and the AC

comment:
We thank all the reviewers for their constructive feedback that helped us improve our paper. Based on these comments, we have revised our manuscript, with the changes highlighted in magenta. Below we provide a brief summary of the key revisions:

* We revised the introduction and method section to further demonstrate the motivations and problem settings. We included a pseudocode for our proposed algorithm in Appendix I and a table summarizing the definitions of all variables used in our paper in Appendix J.
* We updated the results of MultiBench in Table 1 and Figure 5 to better clarify the results and communicate the strength of our approach.
* We included additional baselines and related works (CoCoNet and SimMMDG) and conducted ablation studies of varied data noise levels on the simulation setting and different $\beta$ and $\lambda$ on MultiBench.

In response to **reviewer nJHK and H7Jq** about questions around Table 1 of the three versions of DisentangledSSL representations, we provide the following clarification and explanation:

> Clarifications around Table 1 showing three versions of DisentangledSSL

We acknowledge that in the initial manuscript, there was some confusion surrounding Table 1, which may not have effectively communicated the strengths of our approach. To address this, we have added clarifications, improved Table 1, and introduced Figure 5 in the revised manuscript. 

In the updated Table 1, we highlight the best results among the three representations—shared, modality-specific, and the concatenation of shared and modality-specific (i.e., both)—for all methods. Further, Figure 5 illustrates the performance of each of these three types of representations learned through DisentangledSSL on MultiBench. MultiBench is a collection of real-world datasets where the importance of shared and unique information varies by task. The performance of each representation (shared/specific/both) depends on the relevance of the shared or modality-specific information to the particular task. We provide detailed descriptions of data modalities and task labels of each dataset in Appendix H.2.

Our key observations are:
* MIMIC, MOSEI, and UR-FUNNY: Combining shared and specific representations leads to superior performance.
* MOSI: Shared representations alone achieve the highest performance.
* MUSTARD: Specific representations capture nuances of sarcasm, such as sardonic expressions or ironic tone, which are crucial for sarcasm-level prediction. This explains the high performance of specific representations in Figure 5.

These result trends align with those reported in Table 4 of FactorCL [1]. Further, we ablate the performance of the shared and specific representations for the baselines in Table 4 in Appendix H.2. 

[1] Factorized Contrastive Learning: Going Beyond Multi-view Redundancy. NeurIPS 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission received the ratings of four reviewers, which recommended 5, 6, 6 and 8, averaging 6.25. Given the plenty of competitive submissions in ICLR, this stands at a score above the borderline. The reviewers' concerns focus on the unclear motivation, assumption and some insufficient demonstration in performance. After the substantial rebuttal by authors, most of reviewers considered the concerns well addressed, and one reviewer still maintain some concerns about the performance gain ~2%. After carefully checking the reviewer comments and the authors' feedback, the AC considered that although in real-world case, there are some confounding factors that affect the performance, however, for scientific research, we should allow some tolerance in ideal settings for clear understanding and overcoming problem one by one. Therefore, I tend to recommend acceptance towards the current submission, and hope all advice are well incorporated into the final submission.