An Information Criterion for Controlled Disentanglement of Multimodal Data
Authors: Chenyu Wang
Sharut Gupta
Xinyi Zhang
Sana Tonekaboni
Stefanie Jegelka
Tommi Jaakkola
Caroline Uhler
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Representation Learning
Disentanglement
Self-Supervised Learning
Information Theory
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose DisentangledSSL to separate modality-specific information from shared information in multimodal data, providing theoretical guarantees and strong empirical performance, especially when Minimum Necessary Information is unattainable.

Abstract: Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose $\textbf{Disentangled}$ $\textbf{S}$elf-$\textbf{S}$upervised $\textbf{L}$earning (DisentangledSSL), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called $\textit{Minimum Necessary Information}$ (MNI) point is not attainable. We demonstrate that \algo successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data.

Published as a conference paper at ICLR 2025
AN INFORMATION CRITERION FOR CONTROLLED DIS- ENTANGLEMENT OF MULTIMODAL DATA
Chenyu Wang" 1,2, Sharut Gupta",, Xinyi Zhang!? Sana Tonekaboni",
Stefanie Jegelka 1,3, Tommi Jaakkola', Caroline Uhler
MIT BBoodd Institute of MIT and Harvard 'TU Munich
ABSTRACT
Multimodal representation learning seeks to relate and decompose information in- herent in multiple modalities. By disentangling modality-ppecific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. propose Disentan- gled Self-Supervised Learning (DISENTANGIEEDSS) approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly fo- cusing on the scenario not covered in prior work where the so-called Minimum Necessary Information (MNI) point is not attainable. We demonstrate that DIS- features on
a novel self-supervised
ENTANGLEDSSL successfully learns shared and modality-speen multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, well as molecule-phee molecule-phenotyp retrieval tasks for biological data. The code is available at https: //t/t/gttttttett com/ wahherlab/Dee Dtt.laa///eeeeetglesss....
INTRODUCTION
Humans understand and interact with the world using multiple senses, each providing unique and complementary information essential for forming comprehensive mental representation of the environment. To emulate human-like perception, multimodal representation learning (Bengio et al., 2013; Liang et al., 2022a) seeks to decipher complex systems by combining information from multiple modalities into holistic representations, showcasing significant applications across fields from vision- language (Yuan et al., 2021; Lu et al., 2019; Radford et al., 2021) to biology (Yang et al., 2021; Zhang et al., 2022; Wang et al., 2023a). Large multimodal representation learning models such CLIP (Radford et al., 2021), trained through self-supervised lcarning, maximally capture the mutual information shared across multiple modalities. These models exploit the assumption of multi-view redundancy (Tosh al., 2021; Sridharan & Kakade, 2008), which indicates that shared information between modalities is exactly what is relevant for downstream tasks.
However, the misalignment between modalities restricts the application of these methods in real-world multimodal scenarios. Among various contributing factors, the modality gap, rooted in the inherent differences representational nature and information content across modalities (Liang et al., 2022b; Ramasinghe et al.. 2024; Huh et al., 2024), plays significant role. This highlights the need for disentangled representation space that captures both shared and modality-speciff information:
Coverage: Distinct modalities often contribute unique, complementary information crucial for specific tasks (Liang et al., 2024; Liu et al., 2024b), making it essential capture modality-specine features effectively. For instance, in multimodal sentiment analysis, text conveys explicit sentiment, while vocal tone and facial expressions provide nuanced emotional cues.
Disentanglement: As emphasized by Zhang et al. (2024) in biological contexts, separating shared from modality-specific information is vital for interpretability and decision-making, However,
"Equal contribution

---

Published as a conference paper at ICLR 2025
Dreg tegeting clll
Hoalhy
Cancer cell viabilit at certain time point
Drug mechonism
Modulity 1: X
Modality 2: X²
X1
X2
proteei-tyronie
Post- Perturbation
Canoer cell
Motecular Structure
Phenotype
Cell state healthy cells
Functional inclevant moleznlar sericture
cell death
Reality
Imatinib
Stared
Medulty spocita
Erzungled due to unottiinable MNI
cost-perturbation phenotype (X1) (i.c., cellular images gene expression after the application a drug to cells) and molecular structure (X2) of an underlying drug perturbation system where cancer cells are targeted and killed while healthy cells remain unaffected. The Venn diagram illustrates shared and specific information between modalities X and X2: shared content is shown in red, modality-specifet content in green, and entangled content due unattainable MNI in orange. For example, for the drug mechanism, the molecular structure conveys full information, while the phenotype provides partial information (i.e. mechanisms causing cell death). Similarly, for the states of healthy cells, the phenotype specifies their cell states, whereas the molecular structure only indicates that the cells are unaffected without detailing their specific states.
Figure 1:
maximizing mutual information across modalities can blur this separation (Fischer, 2020), reducing stability and robustness for tasks like cross- moosl translation and counterfactual generation.
Disentangled representation learning in multimodal data traces back to seminal works in Variational Autoencoders and Generative Adversarial Networks (Lee Pavlovic, 2021; Daunhawer et al., 2021; Denton et al., 2017; Gonzalez-Garchn et al., 2018), aiming to isolate the underlying factors of variation within the data. Recent studies have increasingly explored self-supervised learning methods to capture both shared and modality-specific information (Liu et al., 2024b; Li et al., 2024; Zhang et al., 2024). In particular, information- -heoreeic approaches model modality-specifie information as the complement of shared information, learning these representations either sequentially (Sanchez et al., 2020) or jointly (Liang et al., 2024; Pan et al., 2021). However, none of these methods provide rigorous definitions theoretical guarantees the optimality of disentangled representations, particularly under the multimodal scenario where the Minimum Necessary Information (MNI) point between the modalities, as introduced in Fischer (2020), is often unattainable. In simple terms, the MNI criterion characterizes Z as representation of X that perfectly captures all the important information needed understand X without containing any unnecessary details. Essentially, knowing Z gives you the same understanding of X as directly knowing X and vice versa'. However, the shared and modality-speciffr information is deeply entangled wide range of real-world applications, i.e., the shared and modality-specinns components are intertwined and result in similar observations. leading to unattainable MNI. Figure illustrates an example in the context of high-content drug screens. Specifically, the two modalities capture distinct but related aspects of drug mechanism and cancer cell viability, and extracting precise shared features from each modality is not feasible. We provide additional examples of unattainable MNI in Appendix A.
In this work, we propose Disentangled Self Supervised Learning (DISENTANGLEEDSSL) a self- supervised representation learning approach for multimodal data that effectively separates shared and modality-speciffc information. Building on principles of information theory, we devise a step- by-step optimization strategy to learn these representations and maximize the variational lower bound on our proposed objectives during model training. Unlike existing works, we specifically address the challenging and frequently encountered scenarios where MNI is unattainable. We further articulate a formal analysis of the optimality of each disentangled representation and offer theoretical guarantees for our algorithm's ability to achieve optimal disentanglement regardless of the attainability of the MNI point. To the best of our knowledge, this establishes the first set of analyses on the disentangled representations under such settings. Empirically, we demonstrate that DISENTANGLEDSS! successfully achieves both distinct coverage and disentanglement for representations on a suite of synthetic datasets and multiple real-world multimodal datasets. consistently outperforms baselines prediction tasks in the multimodal benchmarks proposed in Liang et al. (2021), as well as molecule-phenotyy retrieval tasks in high-content drug screens
Formally, ((2: 1(7; x²) T(X); x²) (Fischer, 2020), which further explain in Section 2.2.1.
2

---

Published as a conference paper at ICLR 2025
datasets: LINCS gene expression profiles (Subramanian al., 2017) and RXRX19a cell imaging profiles (Cuccarese et al., 2020). To summarize, the main contributions of our work are:
We propose DISENTANGLEDSSLL an cetprmmationtthhoreee framework for learning the disentangled shared and poddalitysppecinff representations of multimodal data.
We present a comprehensive theoretical framework to study the quality of disentanglement that generalizes to settings when the Minimum Necessary Information (MNT) is unattainable. We prove that DISENTANGLEDSSL is guaranteed to learn the optimal disentangled representations.
Empirically. we demonstrate the efficacy of DISENTANGLEDSSL across diverse synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and moheoule-phenotyee retrieval tasks for biological data.
METHOD
In this section, we detail our proposed method, DISENTANGLEDSSL, for learning disentangled representations in multimodal data. We begin by outlining the graphical model that formalizes the problem in Section 2.1 and defining the key properties of "desirable" representations Section 2.2. Subsequently, describe the DISENTANGLEDSSS, framework in Section 2.3 and provide theoretical guarantees on the optimality of the learned representations. Finally, in Section 2.4, we present the tractable training objectives that facilitate efficient learning for each term of our method.
2.1 MULTIMODAL REPRESENTATION LEARNING WITH DISENTANGLED LATENT SPACE
DISENTANGLEDSSL learns disentangled representations in latent space, separating moddaltty-seccifee information from shared factors across paired observations (X-,, X²). This generative process is modeled in Figure Each observation is generated from two distinct latent representations: the modality-specifff representations (Z. and Z?) that contain information exclusive to their respective modalities, and a shared representation (Z₂) that contains information com- mon to both modalities. We refer to these the true latents.
Tre Leteuts
Zi
Generntive Process
X!
X²
X
X²
Inference
21
23
DISENTANGLEDSSL infers the shared representation from both modalities independently, i.e. 2! ~ p(-1x²) and 2: ~ 1P-p(-1X''' p())x)). modality is encoded by variables 2! and 2?. Note that for the true latents, Z3 and Ze are condition- ally dependent on X due to the V-structure in the graphical model. To preserve such dependencies the inferred latents, 2 and 2 are conditioned on both the respective observations and the inferred shared representations, with 2! ~ p(-|x 2.) and 23 ~ x(-|X2, 2?).
Inferred Latents
The modality-specifii information for each
Figure 2: Graphical model.
2.2
INFORMATION CRITERIA FOR THE OPTIMAL INFERRED REPRESENTATIONS
We establish information-theoreti criteria to ensure the shared and modality-specife representations are informative and disentangled, capturing key features while minimizing redundancy.
2.2. INFORMATION BOTTLENECK PRINCIPLE AND MINIMUM NECESSARY INFORMATION
The shared representations Z! and 22 should effectively balance compactness and expressivity, as studied by the information bottleneck (IB) principle in both supervised and self-supervised settings (Tishby et al., 2000; Shwartz-Ziv & Tishby, 2017: Shwartz-Ziv & LeCun, 2023). The IB objective seeks to optimize the representation Z1 of an observation X in relation to a target variable X following the Markov chain z &- X1 + X2. It balances the trade-off between preserving relevant information about X², i.e. I( 22), and compressing the representation, i.e. I(Z';') (see more details in Appendix B). The optimal representation should both sufficient, i.e. [(Zt; X²) = [(X1; X2) (Achille Soatto, 2018; Shwartz-Ziv LeCun, 2023), and minimal (Achille & Soatto, 2018). Based these criteria, Zt is said to capture Minimum Necessary
3

---

Published as a conference paper at ICLR 2025
Information (MNI) between X and X2 if the following holds (Fischer, 2020):
r(x;; x²) T x²) = ;(
MNI characterizes an ideal scenario between X and X² where Z' captures complete information about X² with no extraneous information, i.c. : for an arbitrary joint distribution p(X*, X²2 (see Appendix E), particularly in general multimodal self-supervised settings (e.g. Figure and Appendix A). Despite its significance, a comprehensive discussion of representation optimality when MNI is unattainable is often overlooked in prior work.
- It may not be attainable
2.2.2 OPTIMAL SHARED REPRESENTATIONS: MNI ATTAINABLE OR NOT
We propose a definition of optimality for the shared representations that applies to both scenarios when MNI is attainable or not, as defined in Equation (I)*:
% arg min T(Z'; x 2x2), S.t. - 6e
21
(I)
2 arg min ; x² s.t. - x²
1
6c
22
Formally, minimizing the conditional mutual information, X(22; ensures that the shared
representation captures only the information that is truly common to both X and X2, while dis-
carding modality-specifie details unique to X Compared with ;(Z in IB, it provides a more precise measure of compression and more robust objective.
The constraint 1x';;²) -0(= - de ensures that 2 retains a substantial portion of the shared information between X and X2, controlling the difference within the limit de and preventing significant information loss. We utilize the IB curve F( (Kolchinsky et al.. 2018; Gilad-Bate Bachhrchh et al., 2003), representing the maximum T(Z x²) for given compression level T(X1: ) S 6, to illustrate the optimality in Figure 3. MNI is depicted point A, and 2 corresponding 8 is shown as point C.
71 ziy rxY X²)
TB curve for MNI
F[X;; 22) ((x),x))
H(X
When MNI is attainable, setting 6 = 0 achieves MNI.
Figure 3: IB Curve
In contrast, Achille & Soatto (2018) formulated the opti- mization as Z arg minz1.21, x1.xxx (xx);2),,, x 1(x'; x²), leading to MNI when attainable. This holds in supervised settings, assuming the data label X is deterministic function of X as used by previous methods (Kolchinsky et al., 2018; Fischer, 2020; Pan et al., 2021). However, in general multimodal self-supervised scenarios where MNI is not attainable, this results in point B in Figure 3, which includes information of X that has little relevance to X" to satisfy the equality constraint, causing a gap between the objective and the ideal representation.
2.2.3 OPTIMAL SPECIFIC REPRESENTATIONS: ENSURING COVERAGE AND DISENTANGLEMENT
Optimal speccfic representations. 2. and 23, should capture information unique each modality, being highly informative while minimizing redundancy with the shared representations. We hence define them via the optimization problems:
zi arg max 1 1(2'; x 1x3), s.t. 1(2), ) 6,
(2)
z arg max x21x'), s.t. [(2 2 > d,
Take Z!4 as example. By maximizing the conditional mutual information, T(Z | seek to extract the unique information inherent to modality X that is not redundant with what is
T(Zz x = T(Z* x' x2, x²)- ( = I(Z"; x²) - x²) ) 0, where the first equality is due to the property of conditional mutual information, and the second is due to the Markov structure Z 7'-X'_X²
5e denotes a small tolerance to account for non-attainability of MNI for the modality-specific representation. "The IB curve is concave Baaiaa---Bhrchhhheaa al., 2003), monotonically non-decreasing, and upper bounded by line T 1(2) x3) = I(Z'; x and line T(Z), x²) T(X'; ; x²) (see details and proofs in Appendix C).
4

---

Published as a conference paper at ICLR 2025
shared with X² Note that the same term is minimized in the shared representations described in Equation 1. Integrating both the shared and the modality-speciff representations together, the optimization objectives ensure full coverage of the entire information spectrum of each modality.
Meanwhile, the constraint I(Z'; 2 6, minimizes the overlap between modality -esaaitt--peciiic and shared representations, enhancing disentanglement by effectively separating the unique aspects of X from the components shared with X². The parameter 6, controls the trade-off between coverage and disentanglement.
2.3 DISENTANGLEDSSL: A STEP-RY-STEP OPTIMIZATION ALGORITHM
To achieve the optimal representations discussed in Section 2.2, we introduce two-step training procedure. The first step focuses on optimizing the shared latent representation, ensuring it captures the minimum necessary information as close possible. Building upon this, the second step utilizes the learned shared representations in step to facilitate the learning of modality-speciiff representations. This sequential approach is formalized in the optimization objectives given in Equations 3 and 4, with the pseudocode provided in Appendix I:
Step 1: Learn the shared latent representations by encouraging the shared representation encoded from one modality to be highly informative about the other modality, while minimizing redundancy.
2.0- arg max Lg arg max I I(Z x²) 3 - 1(,
2
2
(3)
22 arg max L arg max /
2
2
Step 2: Learn the modality-specifie latent representations based on the learned shared representations from step 1.
2/1 arg max L arg max T(Z), 22" x x
21
z1
23 = arg max L3 = arg max I I(Z2, 21h, -
Z
z
The hyperparameters 3 and À control the trade-off between relevance and redundancy for the shared and mnodllity-peecific representations respectively. We use the same values of / and X for both modalities since they operate similar information scales. Our sequential training approach, instead of a joint one, stems from the self-sufficient nature of each optimization procedure where one sub- optimal representation does not enhance the learning of the other. We offer a comprehensive analysis of the optimality guarantees of this step-by-ster method as follows.
2.3.1 OPTIMALITY GUARANTEE FOR THE LEARNED SHARED REPRESENTATIONS
This section explores how the step objective, T, optimizes the shared representation between modalities by balancing expressivity and redundancy. We discuss its effectiveness in both scenarios- when MNI is attainable or not.
L! seeks representation Z that maximizes the information shared between the modalities, i.e. ((2):²²) while minimizing the information unique each modality, i.e. 1(21; x²x²)), to capture only the essential shared content. This aligns with the conditional entropy bottleneck (CEB) objective (Fischer, 2020)5, an extension of the IB Lagrangian L = 1(2,; X²) ) xx2)-B1(Z1;; 31((\;'') (see details in Appendix B). While it serves remains underexplored in Fischer (2020), particularly when MNI unattainable. Additionally, Fischer (2020) focuses on the supervised scenario where X is the label of X whereas we address the multimodal case with X and X being two data modalities, demonstrating its effectiveness in both attainable and unattainable MNI scenarios.
robust objective for learning shared information, its optimality
When MNI is attainable, Proposition I states that the step optimization achieves MNI for any positive B. The proof is given in Appendix D.
'Note that T.f is the Lagrangian formulation of the constraint optimization in Equation 1, among which the term T(X): X²) irrelevant to the optimization target Z, and omitted from the objective.
5

---

Published as a conference paper at ICLR 2025
Proposition 1. Mf is attainable for random variable X' and X², maximizing L ; ; achieves MNI for any B3 > i.e. T T(2 ; T(2 x²) T(X1; X²), where
2/1 arg ma L.-
When MNI unattainable, the inequalities I(Z'; 200))(((2) xx2 and 1(X xx
cannot achieve equality simultaneously. This indicates inherent trade-off between capturing the entire shared information and avoiding the inclusion of modality-speciif details. More preciscly, under the condition of strict concavity of the ;( trade-off presented in Proposition 2. The proof is given Appendix D.
I(Z X information curve", such
Proposition For random variables X and X2, when the curve ; = = is strictly concave,
1) there exists a bijective mapping from B3 in L. to the value of information constraint óg in the definition of optimal shared latent 2,2" in Equation I;
0112)";8 :x)
"1(2]";;3 (x2) < O, where 7.3" is the optimal solution corresponding to a certain B.
2)
0,
03
03
The second property implies that 3 increases, the learned representation becomes less informative and redundant, demonstrating the trade-off between expressivity and redundancy when MNI is unattainable. Furthermore, the bijection mapping between 3 and 6e allows our step optimization to navigate the information frontier across various 3 values, facilitating soft control and the segmentation of shared information into different degrees of granularity.
2.3.2 OPTIMALITY GUARANTEE FOR THE LEARNED MODALITY-SPECIRC REPRESENTATIONS
In this section, we demonstrate the step 2 objective, T. ensures optimal coverage and disentanglement by showing equivalence (or nearly equivalence) to the Lagrangian of Equation 2.
L in Equation 4 learns the modality-speeiiff representation Z' based on the optimal shared repre- sentations - and 2 learned from step 1. It maximizes the information coverage of the data X through the combination of Zt and 2 i.e. 1(Z), 2,:2;;, 1 Simultaneously, it promotes disentan- glement by limiting overlap with the shared representation 2 of the same modality, indicated by T(Z!: " This objective is Lagrangian formulation of the constraint optimization in Equation 2, however replacing I(Z): x x with 1 ;11)
When MNI attainable, as justified in Proposition 3, this substitution results in an equivalent objective, allowing L3 to achieve optimal modality -sstaitty-peccficc representations (proof in Appendix D). Proposition 3. If MNI is attainable for random variables X and X?,
(1Z'x'1(') = arg max I(Z), -2²"x!
arg max Z1-Xi-X
1--X-X X1-X2 X2
where 2.2- is the representation of X2 that satisfies MNI, iee. T( x = T 1 x²) x²)= ((';,')
When MNI is unattainable, Proposition 4 shows such substitution yields almost equivalent objective, subject to the value de that corresponds to the 3 used in step optimization. Thus, optimizing Lg results in nearly optimal modality-speciinn representations (proof in Appendix D).
Proposition 4. For random variables X and x2,
0 < IIZZ,X'' x² 1(22,22- ;2;xx1 < 8
where 22 is the optimal representation of x with respect to d- as defined in Equation i.e. 2: = arg minga I(Z²; x²)x',, s.A. I(x"; x²) I(Z2x x²) S6.0
2.4 TRACTABLE TRAINING OBJECTIVES
Four terms are involved in DISENTANGLEDSSL. including maximizing the mutual information term 1[2 x²) and minimizing the conditional mutual information term 1 X |1)x for the shared
When the IB curve is not strictly concave (i.e. partially linear), the properties hold except the linear sections. In these cases, methods like squared TB (Kolchinsky al., 2018) can be used achieve bijection mapping.

---

Published as a conference paper at ICLR 2025
representations in step 1, as well as maximizing the joint mutual information term I(Z', 22"; and minimizing the mutual information term I(Z %g²)) for the modality-peecifie representations in step 2. In this section, we introduce the tractable training objectives for each term, with detailed formulations described in Appendix G.
For the inferred shared representations, we model the distributions 2 p(-1x²) and 22 p(- with neural network encoders. Following the common practice in Radford et al. (2021), we use the InfoNCE objective (Oord et al., 2018) as an estimation of the mutual information term I [ X²) in Lg- We implement the conditional mutual information term T(Z"; XX2) in L using an upper bound developed Federici et al. (2019), i.e. I(2,; x1|x) S DxL (p(2) x x²)))p((²(xxx)) While the conditional distributions of representations are modeled as the Gaussian distribution in Federici et al. (2019), we instead use the von Mises-Fiste Fisher (vMF) distribution for p( |X') and p(Z?|X?) to better align with the InfoNCE objective where the representations lie the sphere space. Specifically, Ze ~ vMF(u(X)) )),k), x), 22 vMF(u(X²²) x) where A a hyperparameter controlling for the uncertainty of the representations. Leveraging the formulation of the KL divergence between two vMF distributions, the training objective of L is to maximize:
L L + B Egi [m(z') (o(=))
The inferred modality-specine representations are encoded functions that takes both data ob- servations and the shared representations learned in step 1 as input to account for the dependence structure illustrated in Figure 2. The term 1(21, ; in L3 is optimized with the InfoNCE loss, where random augmentations of the data X form the two views. For the mutual information term between representations, T(Z ") in L., we implement as an orthogonal loss to encourage the marginal independence between the shared and poddllit--speciff representations, where the marginal
distribution is approximated across a training batch.
EXPERIMENTS
We conduct a simulation study and two real-world multimodal experiments to evaluate the efficacy of our proposed DISENTANGLEDSSL. We compare DISENTANGLEDSST against a disentangled variational autoencoder baseline, DMVAE (Lee & Pavlovic, 2021), as well as various multimodal contrastive learning methods, including CLIP (Radford et al., 2021). which aligns different modalities to learn the shared representations, and FOCAL (Liu et al., 2024b), FactorCI. (Liang et al., 2024) and SimMMDG (Dong et al., 2023), which capture both shared and modality-specifie information. Additionally, we evaluate a joint optimization variant of DISENTANGLEDSSL, JointOpt (Pan et al., 2021), to highlight the advantages of our step-by-step approach.
3.1 SIMULATION STUDY
Synthetic data generation. We generate synthetic data based on the graphical model in Figure
2. We sample d-dimensional true latents 2,, z33 and Z. independently from N(ua, Σ²). Using fixed transformations Ti and T2, we create X Ti !z! Za] and X = T2 [Z? Ze]. To simulate unattainable MNI, we add Gaussian noise to ensure the distribution has full support. Binary labels YYY and Y. are constructed from the true latents and used to evaluate the information content of learned representations via linear probing accuracy. Denote 2. as the combination of the learned shared representations of x' and X" i.e. 2.- = 12!, 2.3. Ideally, 2. should achieve high accuracy on Y. and low on y: and Y.?, while the modality -specific representations Z! and 23 should show the opposite pattern. Additional details on experimental settings can be found in Appendix H.
Results. We assess the performance of the learned shared and modality-speeiinn representations for different values of 3 and X, as shown in Figure 4. For comparison, we also evaluate JointOpt, DMVAE, FOCAL, and SimMMDG across different hyperparameter settings. Specifically, for JointOpt, we vary both a and X, where controls the joint mutual information terms 1(2) 22: X and 1(22, 2:; X²), and X adjusts the mutual information term between representations, similar to DISENTANGLEDDSL.
Figure 4a illustrates the performance of shared representation 2. learned by DISENTANGLEDSSL across different values of 3. For lower values of B, 2, captures both shared and specific features, as indicated by linear probing accuracy on Y1 Y22 and Y. exceeding 0.5. As increases, all accuracies
7

---

Published as a conference paper at ICLR 2025
decrease, reflecting the trade-off between expressivity and redundancy controlled by & when MNI is unattainable. This trend aligns well with Figure 3 and Proposition 2. A comparison of the performance of the shared representation with baseline methods is provided in Appendix H.1. We further examine such trade-off on synthetic data with varying levels of multimodal entanglement, where some dimensions aligned across modalities with MNI attainable and others remain entangled with MNI unattainable. Detailed results are provided in Appendix H.I.
Given the shared representations Z and 22 learned in step for a fixed 8, we then learn the corresponding modality-speciiff representations Z! and 23 with varying A. Figure 4b shows the performance DISENTANGLEDSSL in contrast to other baseline methods, where dots are connected according to a descending order of their corresponding À values". The ideal modality-speciff representation Z should maximize unique information from X shown by high accuracy Y,', while minimizing shared information with X², indicated by low accuracy on Y-. Therefore, bottom- outperforms
right point is preferred in Figure 4b. As illustrated in Figure 4b, DISENTANGLEDSS all other methods across various hyperparameter settings, especially JointOpt, demonstrating the effectiveness of the stepwise optimization procedure. Results on 2g are provided in Appendix H.1.
Oues,
Toa Accuracy of 2) on Y.
0.70
Duzs, 01z48::11 8=00
Test Accuracy
0.70
0.85
Seirtig,
X.
0.60
Jenthi a 1000
3.50
JinDsi,
1/1
DVVNE
1.2
POCAL
SIMMDO
3.0
35"
105
1071
10²
10
30a
10/
0.7
Test Accumey 2
(a) Performance of shared representation Ze learned (b) Performance of modality-specifie representation 2! learned with different and X, in comparison baselines.
with different values of B.
Figure 4: Simulation study results.
3.2 MULTIBENCII
Dataset deseription. We utilize the real- world multimodal benchmark from MultiBench (Liang et al., 2021), which includes curated datasets across various modalities, such as text, images, and tabular data, along with downstream label that expect shared and specific information to have varying importance for. These datasets cover a variety of domains including healthcare, affective computing and multimedia research areas. We follow the same setting (dataset splitting, encoder architecture, pre-extracted features) as in Liang et al. (2024). Additional details can be found in Appendix H.2.
Results. We evaluate the linear probing accuracy of rep- resentations learned by each pretrained model, as shown in Table 1. FactorCL-emb refers to the embeddings of FactorCL before projection heads, while FactorCL-proj uses the concatenation of all projection head outputs. All models use representations (or concatenation of them, if applicable) with the same dimensionality. We present the modality-
Representations
Prediction Accuracy
Shared
70
Speci
Bedh
60
50
MDMTC
MOSEI
MOST UR. FUNNMUSTARD
best results among three representations--hann specific, and the concatenation of both-for all methods in Table 1, with results for each representation in Ap- pendix H.2. DISENTANGLEDS?!
Figure Results of different representa- tions learned by DISENTANGI.EDSSL..
consistently outperforms baselines across datasets, demonstrating its ability capture valuable information for downstream tasks. Further, show the performance of each representation leamned through DISENTANGLEDSS in Figure 5. Combining shared and specific representations of DISENTANGLEDSSL improves performance most cases, showing that both
the hyperparameters and defined similarly to JointOpt, where controls the terms I(2: X ! and I (2?, x²) and À adjusts the orthogonal loss between shared and specific representations, For DMVAF, we tune À which denotes the weight of the KI. divergence term. We show the best- pers-performing results of FOCAL and DMVAF across hyperparameters in Figure 4b, with full results available in Appendix H.I.
FFor FOCAL,
8

---

Published as a conference paper at ICLR 2025
contribute to label prediction, while in MOSI and MUSTARD, only shared specific information contributes significantly. For example, for MUSTARD, specific representations capture nuances of sarcasm, such sardonic expressions ironic tone, which are crucial for sarcasm prediction.
Table I:
Prediction accuracy (%) of the representations learned by different methods on MultiBench
datasets and standard deviations over 3 random seeds.
Datasel
MIMIC
MOSEI
MOSI
UR-FUNNY
MUSTARD
CLIP
64.97 (0.60)
76.87 (0.45)
64.24 (0.88)
62.73 (0.92)
56.04 (4.19)
FactorCL-emb
65.25 (0.45)
71.80 (0.64)
62.97 (0.81)
63.29 (2.07)
56.76
FactorCL-proj
59,43 (1.70)
74.61 (1.65)
56.02 (1.26)
61.25(0.477
(2.18)
FOCAL
64.42 (0.34)
76.77 (0.51)
63.65 (1,09)
63.17 (0.96)
58.21 (2.21)
JointOpt
66.11 (0.64)
76.71 (0.14)
65.02 (1.96)
63,58 (1.45)
57.73 (4.12)
DISENTANGLEDSSL
66.44 (0.31)
77.45 (0.06)
65.16 (0.81)
64.24 (1.54)
61.60 (2.61)
3.3 HIGH-CONTER DRUG SCREENING
Dataset description. As characterized in Figure 1, we use two bigh-content drug screening datasets which provide phenotypic profiles after drug perturbation: RXRXI9a (Subramanian et al., 2017) containing cell imaging profiles, and LINCS (Cuccarese et al., 2020) containing L1000 gene expres- sion profiles. For RXRXI9a, we select data from the HRCE cell line under active SARS-CoVA condition, totaling 1,661 drugs and 10,117 profiles (~ 6 replicates per drug). Hand-crafted image features from CellProfiler (McQuin et al., 2018) are used as encoder inputs. For LINCS, we use the data curated in Wang et al. (2023a), selecting drugs with full observations across all 9 core cell lines and randomly sampling replicates per drug to minimize selection bias, resulting in 3,315 drugs and 86,833 profiles. We conduct train-vall dedin-valddation-tt pretrained to learn representations of molecular structures and their corresponding phenotypes. We provide additional details experimental settings in Appendix H.3.
splitting according to molecules. Models are
Table 2: Retrieving acccuracy and mean reciprocal rank (MRR) of molecule-phenotye retrieval.
Dataset
Top Ace (%)
N=1
N=5
N=10
N=20
N=30
MRR
Random
0.30
1.50
3.00
6.01
9,01
CLIP
3.3000.40
8.33(0.52)
11.59(0.20)
16.38(0.56)
20.22(0.67)
0.103(0.001)
DMVAE
3.85(0.36)
8. 8.76(0.30) 30)
11.840..2) 32)
16.20/0.83)
20.17(0.85)
0.106(0.002)
RXRX1%a
JointOpe
3.41(0.49)
8.54/0.
11.64(0.1))
16.79(0.27)
20.71(1.11)
0.01000.02)
FOCAL
3.61(0.51)
(.71()
(1.94(0.4))
16.86(0.22)
20.81(1.07) (10)
0,108(0.003)
DISENTANGLEDSSS (ff
3.39(0.54)
8.25(0.33)
(1.530.20)
16.28/0.10
20.20(0.45)
0.109(0,033)
DISENTANGLEDSSL
4.03(0.39)
9.62(0.20)
13.12(0.23)
18.36(0.42)
22.82(0.45)
0.111(0.001)
Random
0.15
0.75
1.51
3.02
4.52
CLIP
3.95(0.04)
(0.81(0.06)
15. 10(000))
21 (.((9))33)
29.44(0.30)
0,146(0.001)
DMVAE
4.31(0.09)
11.45(0.13)
15.88(0.17)
21.85())27)
29.41(0.33)
0.156(0.001)
LINCS
JointOpe
4.67(0.09)
11.60(0.11)
16.02(0.15)
22.01(0.21)
30.08(0.39) 39)
0.161 (0.001)
FOCAL
4.34(0.17)
11.2440.19)
15.74(0.26)
21 21.48(0.08)
29.32(0.08)
0.157(0.002)
DISENTANGLEDSSL (3 0)
4.36(0.13)
15.81(0.47) 11.2770.39)
21.71(0.48)
29.65(0.54)
0.158(0.001)
DISENTANGLEDSSL.
4,48(0.21)
11.70(0.40)
16,39(0.39)
22.68(0.58)
30.84(0.60)
0.16X00002)
pe retrieval using shared representations. We evaluate the shared representa- tions the polecule-phenotypn retrieval task, where the goal is to identify molecules from the whole test set that are most likely to induce a specific phenotype. The shared information, which connects the molecular structure and phenotype. plays a key role in this task. We tune 3 according to validation set performance and show results of top N accuracy 1N=1,5,10,20.33 and mean reciprocal rank (MRR) on the test set in Table 2. DISENTANGLEDS! datasets, effectively capturing relevant shared features while excluding irrelevant modality-specific information. Notably, compared to the variant without the information bottleneck constraint, i.e. DISENTANGLEDSSL (3 0), the full DISENTANGLEDSSL model preserves critical shared features, achieving superior performance in this retrieval task, where shared information is essential.
consistently outperforms basclines on both
Disentanglement measurement. To assess the effectiveness of learning modality-specifft represen- tations, we introduce the Reconstruction Gain (RG) metric, which quantifies the disentanglement and complementariness between shared and modality train 2-layer MLP decoders on the training set to reconstruct the original data from the shared and modality-speeifii representations, both individually and jointly, and compute the reconstruction R² on the test set for each case. A higher gain in R² when using combined representations in comparison to individual ones indicates lower redundancy and better disentanglement.
representations. Specifically, we

---

Published as a conference paper at ICLR 2025
As shown in Table 6, DISEN- TANGLEDSSL achieves the highest RG scores across both modalities and datasets, demonstrating superior dis- entanglement. In contrast, other methods show redun- dancy due to insufficient con- straints for disentanglement during pretraining. We also test the representations from the pretrained model on the counterfactual generation task, with detailed results provided in Appendix H.3.
Table 3: Reconstruction gain (RG) in R² of representations for each modality (molecular structure/phenntyye).
Dalaset
RXRX19:
LINCS
Metric
RG moleule
RG phenotype
RG RG-molecule
RG phenotype
FOCAL
0.122(0.001) 0.547(0.001) 0.117(0.006)
0.618(0.002)
DMVAR
0.545(0.003) 0.123(0.02))
0.139(0.002)
0.605(0.003)
JointOpt
0.130(0.001)
0.524(0.001)
0.103(0.000)
0.604(0.002)
DISENTANGLEDSSL
0.153(0.003)
0.591(0.001)
0.143(0.000)
0.622(0.002)
4 RELATED WORK
Disentangled representations in VAEs and GANs. Disentangled representation learning originated from works on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), focusing on isolating underlying data variations linked or independent of labels (e.g.. digit identity vs. writing style in MNIST) (Gonzalez-Garde) et al., 2018; Mathieu et al., 2016; Bouchacourt et al., 2018; Daunhawer al., 2021). This concept extended to multimodal data like text and image (Lee et al., 2021) and content and pose in video (Denton et al., 2017), typically using self- and cross- reconstruction with adversarial loss to learn shared and specific components. However, these methods often address simple cases with attainable MNI, lacking a comprehensive analysis for complex real-world multimodal scenarios where MNI is not attainable.
Information bottleneck and its variants. The information bottleneck (IB) principle has been used to analyze and optimize deep neural networks and the learned representations from the information theory perspective in both supervised and self-supervise settings (Tishby et al., 2000;
et al., 2003; Shwartz-Ziv & Tishby, 2017; Kolchinsky et al., 2018; Rodríguez Gálvez et al., 2020; Tsai et al., 2020; Wang et al., 2023b; Shwartz-Ziv & LeCun, 2023), with extensions like the conditional entropy bottleneck (CEB) (Fischer, 2020; Lee et al., 2021) being developed to better refine the information plane. IB has been applied to learn molecular representations from drug screening data (Liu et al., 2024a). However, these approaches primarily focus on extracting shared features while minimizing specific features, limiting their practical use. Other methods (Pan et al., 2021; Sanchez et al., 2020) learn disentangled representations by optimizing mutual information and using adversarial loss but are confined to settings with unattainable MNI.
scenarios and do not address complex multimodal
Multimodal disentanglement in self-supervised learning. Recent studies have explored disentan- gled representation learning in multimodal self-supervised contexts. Liang et al. (2024) factorizes and optimizes mutual information bounds to capture the union of shared and unique features, while not penalizing for redundancy between latents. Zhang al. (2024) learns disentangled representations of cell state for multimodal data in biological contexts. Liu et al. (2024b) and Li et al. (2024) use mutual information optimization with orthogonal/MMD regularizations for disentangling multimodal time-series signals and T-cell receptor segments, respectively. Li et al. (2022) and Dong et al. (2023) address the separation of shared and specific information, either among views of images or in the domain generalization setting. However, these methods lack a general framework for understanding information content, particularly when data modalities are deeply entangled in real-workes applications.
5 CONCLUSIONS
We present DISENTANGLEDSSL, a self-supervised approach for learning disentangled multimodal representations that effectively separates shared and modality-specific features. Built on a compre- hensive theoretical framework, DISENTANGLEDSS! addresses the challenging scenarios where the Minimum Necessary Information (MNI) is unattainable, enhancing its applications to com- plex real-world multimodal settings and improving interpretability and robustness. Our empirical results show that DISENTANGLEDSSL outperforms baselines across various synthetic and real- world datasets, including vision-language prediction tasks and molecule- pireculepphenotppe retrieval for high-content drug screens, demonstrating its ability to learn optimal disentangled representations for diverse applications.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
For the theoretical results presented the paper, we provide complete proofs in Appendix D. For the experimental results, we provide detailed explanations the algorithm implementations and experimental setup in Section 3, Appendix G, and Appendix H. For the datasets used in the experiments, we utilize publicly available datasets and follow the data processing procedures in prior works, with additional details elaborated in Section 3 and Appendix H. The code can be found in https: / thub. wcom//hleee ab/Di sentang edSSL.
ACKNOWLEDGEMENT
XZ, ST and CU were partially supported by the Eric and Wendy Schmidt Center at the Broad Institute, NCCIH/NIH (1DP2AT0012455, ONR (N00014-22-1-2116) and DOE (DE-SC0023187) SG and SJ acknowledge the support of the NSF Award CCF-2112665 (TILOS AI Institute), an Alexander von Humboldt fellowship, and Office of Naval Research grant N00014-20-1-205 1-2022 (MURI ML-SCOPE). CW and TJ acknowledge support from the Machine Learning for Pharmaceutical Discovery and Synthesis (MLPDS) consortium, the DTRA Discovery of Medical Countermeasures Against New and Emerging (DOMANE) threats program, and the NSF Expeditions grant (award 1918839) Understanding the World Through Code.

---

Published as a conference paper at ICLR 2025
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Michael F Cuccarese, Berton A Earnshaw, Katie Heiser, Ben Fogelson, Chadwick Davis, Peter F McLean, Hannah B Gordon, Kathleen-Rose Skelly, Fiona L Weathersby, Vlad Rodic, al. Functional immune mapping with deep-learning enabled phenomics applied to immunomodulatory and covid-19 19 drug discovery. Biorxiv, pp. 2020-08, 2020.
Imant Daunhawer, Thomas M Sutter, Ričards Marcinkevišs, and Julia E Vogt. Self-supervised disentanglement of modality-specife and shared factors improves multimodal generative models. In Pattern Recognition: 42nd DAGM German Conference, DAGM GCPR 2020, Tithnggen, Germany, September 28-October 1, 2020, Proceedings 42, pp. 459-473. Springer, 2021.
Emily I. Denton et al. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30, 2017.
Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. Simmmdg: A simple and effective framework for multi-modal domain generalization. Advances in Neural Information Processing 2023.
36:1ems,36:786744
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust
representations via multi-view information bottleneck. In International Conference on Learning Representations, 2019.
Ian Fischer. The conditional entropy bottleneck. Entropy. 22(9):999, 2020.
Ran Gilad- Bachrach, Amir Navot, and Naftali Tishby. An information theoretic tradeoff between complexity and accuracy. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, pp. 595-609 Springer, 2003.
Abel Gonzalez- Garcia, Joost Van De Weijer, and Yoshua Bengio. mmage-to-image translation for cross-domain disentanglement. Advances in neural information processing systems, 31, 2018.
Minyoung Huh, Brian Cheung. Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.
Sabrina Jaeger, Simone Fulle, and Samo Turk. Mol2vec: unsupervised machine learning approach with chemical intuition. Journal of chemical information and modeling, 58(1):27-35, 2018.
Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. arXiv preprint
07599, 2018.
Kuang-Hue' Lee, Anurag Amab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. Advances in Neural Information Processing Systems, 34: 14:19528-19552, 2021.
Mihec Lee and Vladimir Pavlovie. Private-shared disentangled multimodal vae for learning of latent representations. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 1692-1700. 2021.
Jiangmeng Li. Wenwen Qiang, Changwen Zheng, Bing Su, Farid Razzak, Ji-Rong Wen, and Hui Xiong. Modeling multiple views via implicitly preserving global consistency and local comple- mentarity. IEFE Transactions Knowledge and Data Engineering, 35(7):7220-7238 2022.
12

---

Published as a conference paper at ICLR 2025
Tianxiao Li, Hongyu Guo, Filippo Grazioli, Mark Gerstein, and Martin Renqiang Min. Disentangled wasserstein autocncoder for t-cell receptor engineering, Advances in Neural Information Processing Systems, 36, 2024.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. arXiv preprint urXiv:2107.07502, 2021.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint arXiv:2209.03430 2022a.
Paul Pu Liang, Zihao Deng, Martin Q Ma, James Y Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36, 2024.
Victor Weixin Liang, Yuhui Zhang, Yongehan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 15:17612-17625, 2022b.
Gang Liu, Srijit Seal, John Arevalo, Zhenwen Liang, Anne E Carpenter, Meng Jiang, and Shantanu Singh. Learning molecular representation in cell. arXiv preprint arXiv:2406. 24t.::2600..20566, 2024a.
Shengzhong Liu, Tomoyoshi Kimura, Dongxin Liu, Ruijie Wang. Jinyang Li, Suhas Diggavi, Mani Srivastava, and Tarek Abdelzaher. Focal: Contrastive learning for multimodal time-series sensing signals in factorized orthogonal latent space. Advances in Neural Information Processing Systems, 36, 2024b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for aoooo--nd-lanuuss tasks. Advances in neural information processing systems, 32, 2019.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. Advances in neural information processing systems, 29, 2016.
Claire McQuin, Allen Goodman, Vasiliy Chernyshev, Lee Kamentsky, Beth A Cimini, Kyle W Karhohs, Minh Doan, Liya Ding, Susanne M Rafelski, Derek Thirstrup, et al. Cellprofiler 3.0: Next-generation image processing for biology. PLoS biology, 16(7):e2005970, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint / /807.03748, 2018
Ziqi Pan, Li Niu, Jianfu Zhang, and Liqing Zhang. Disentangled information bottleneck. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9285-9293, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 9748-8763. PMLR, 2021.
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. Accept the modality gap: An exploration in the hyperbolic space. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 27263-27272 2024.
Borja Rodríguez Gálvez, Ragnar Thobaben, and Mikael Skoglund. The convex information bottleneck lagrangian. Entropy, 22(1):98, 2020.
Eduardo Hugo Sanchez, Mathicu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pp. 205-221 Springer, 2020.
Ravid Shwartz- Ziv and Yann LeCun. To compress or not to compress self celf-supervised learning and information theory: A review. arXiv preprint arXiv:2304.03335 09355, 2023.
13

---

Published as a conference paper at ICLR 2025
Ravid Shwartz-ZZ Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint FreXiv::00300000 2017.
Karthik Sridharan and Sham M Kakade. An information theoretic framework for multi-view learning. In COLT, pp. 403-414, 2008.
Aravind Subramanian, Rajiv Narayan, Steven M Corsello, David Peck, Ted E Natoli, Xiaodong Lu, Joshua Gould, John F Davis, Andrew A Tubelli, Jacob K Asiedu, et al. A next generation connectivity map: L1000 platform and the first 1,000,000 profiles. Cell, 171(6):1437-144
2017.
Naftali Tishby, Fernando Percira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206. PMLR. 2021.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Moreney. Self- learning from multi-view perspective. In International Conference on Learning Representations 2020.
Chenyu Wang, Sharut Gupta, Caroline Uhler, and Tommi S Jaakkola. Removing biases from
molecular representations via information maximization. The Twelfth International Conference Learning Representations, 2023a.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929 9939. PMLR, 2020.
Ying Wang, Tim GJ Rudner, and Andrew Gordon Wilson. Visual explanations of image-text representations via multi-modal information bottleneck attribution. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b.
Karren Dai Yang, Anastasiya Belyaeva, Saradha Venkatachalapathy, Karthik Damodaran, Abigail Katcoff, Adityanarayanan Radhakrishnan, GV Shivashankar, and Caroline Uhler. Multi-domain translation between single-cell imaging and sequencing data using autoencoders. Nature Commu- nications, 12(1):31, 2021.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Xinyi Zhang, Xiao Wang, GV Shivashankar, and Caroline Uhler. Graph-based autoencoder integrates spatial transcriptomics with chromatin images and identifies joint biomarkers for Alzheimer's disease. Nature Communications, 13(1):7480, 2022.
Xinyi Zhang, GV Shivashankar, and Caroline Uhler. Partially shared multi-modal embed- ding learns holistic representation of cell state. Under revision, preprint available at https:
stweeettt///t//eeeeeeeeeeeeeesee wwwssshrrrttt// 10.224....00 01. 1119771119 2024.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXAMPLES OF UNATTAINABLE MNI
In this section, we present two additional examples where the Minimum Necessary Information (MNI) unattainable. These examples offer clear understanding of how unattainable MNI manifests in data, providing practical insights and making the concept more accessible.
Firstly, show example in simple mathe- matical case. As illustrated in Figure 6, we have true latent variables are independently drawn from a Bernoulli distribution, i.e. 7 Ze, 722 Bernouli(00.5) and observations X and X² are generated according to X OR(Z, Ze, 23-) and X2 OR (Z. Z3). In this scenario, we observe X = 1, we are not able to distinguish whether this result is due Z. 1, or Z or both. Thus, extracting purely shared features from the observations is infeasible.
True Latents.
Ze
Z
X2
X
Figure Example unattainble MNI.
Further, provide example in 3D geometry, illustrated in Figure 7. For information such as the height of the cone and the number of spheres, one modality conveys the complete information while the other conveys partial information. This ambiguity makes it challenging to classify the information as purely shared or modality-speciife placing it in an uncertain category.
Modality X
Modality 2: X²
X!
Heigght the come
X2
of spleces
Image taken from the
Text deseription
Color of the sphere the right
Radins of the blue
Ciolor
Reality
sphere red cone, and a green sphore to the right of
Modalit specitic Entangled
Figure 7: Image modality (X₂) and text modality (X2) of underlying reality. The Venn diagram illustrates shared and specific information between modalities X; and X2: shared content is shown in red, modality-specifie content in green, and entangled content due unattainable MNI in orange. For example, for the cone's s height, the image conveys full information, while the text (i.e. "tall red cone") provides partial information. Similarly, for the number of spheres, the text specifies it completely, whereas the image indicates there is at least one sphere.
B BACKGROUND ON INFORMATION BOTTLENECK
The Information Bottleneck (IB) principle provides a powerful theoretical foundation for our method. IB objective seeks to find representation Z of random variable X that optimally trades off the preservation of relevant information with compression (Tishby et al., 2000). Relevance defined as the mutual information, 1(Z X²), between the representation Z and another target variable X Compression is enforced by constraining the mutual information between the representation and the original data, I(X'; Z to lie below specified threshold. This can be formalize in of the following constrained optimization problem:
x²), 1(2) ( 5 5
arg max Z
where Z from a set of random variables that obey the Markov chain Z ++ X1 <+ X2, In practice, Equation can be optimized by minimizing the IB Lagrangian in Equation 6.
I(2 -
Here, the Lagrangian multiplier 3 controls the emphasis placed on compression expressiveness.
15

---

Published as a conference paper at ICLR 2025
Fischer (2020) extends IB the conditional entropy bottleneck (CEB) objective, which utilizes the conditional mutual information term I(Z); xx in place of the mutual information term [[Z': ( in Equation 6:
L x= -
While being equivalent to the IB Lagrangian in terms of the information criterion, CEB rectifies the information plane precisely measure compression using the conditional mutual information, and offers clear assessment of how close the compression is to being optimal.
Note that although CEB aligns with our step objective in Equation 3, its optimality has been fully examined in Fischer (2020), particularly under scenarios where MNI is unattainable. Furthermore, Fischer (2020) primarily addresses supervised learning context, where X serves as the label for X In contrast, our study tackles the more complex multimodal setting, where X and X are two distinct data modalities. We extend the analysis to both attainable and unattainable MNI cases, demonstrating the efficacy of our approach in capturing shared and modality-specine information under these challenging scenarios. This broadens the applicability of CEB beyond traditional label-based supervised settings to multimodal data with complex modality entanglements.
C PROPERTIES OF THE IB CURVE
The information plane is a helpful visualization of the information bottleneck principle, which utilizes I(X': 2 and I Z ) its coordinates that represents the trade-off between compression and prediction (Tishby et al., 2000). The frontier of all possible Zt is called the IB curve or F(r)
(Kolchinsky et al., 2018) as follows:
F(r) = zz.Z X²
1(2); s.t. ((2; ST
The IB curve has the following properties:
The IB curve is concave (Lemma 5 in Gilad-Bar Bachrach et al. (2003)) and monotonically non- decreasing.
The IB curve is upper bounded by line I(X2; Z I(X!: and line ((x; II(X; according to the Markov relationship - X2.
When the MNI point attainable for random variables X and X2, the IB curve has the following formula (Rodríguez Gálvez et al., 2020; Kolchinsky et al., 2018);
if if I(Z);x!),
I x2= = (()); if i
1(Z);x') S H(X!)
and the Gateaux derivative of 1(X2; with respect to p==1(=²) (as used in (Pan et al., 2021))
doesn't 't exist the MNI point (see details in Appendix F).
D PROOFS
D.1 PROOF OF PROPOSITION 1
Proposition 1. MNI is attainable for random variable X and X?, maximizing L/ = I(Z"; x achieves MNI any 3 > 0, i.e. 1(21";X') = ; = 1(x1; where 21 :- arg max X L.-
Proof. Based on data processing inequality and the Markov relationship 25 - x T(2). x²) < T(X1; (2). Based on the non-negativity of conditional mutual information, I( x [X2) 2 0. Thus, for 3>0 3>0, 0,
L 12! - -- x < 1(x1; v
where the equality holds when 1(22; X2) T(XI; X2) and 1 1 x
16

---

Published as a conference paper at ICLR 2025
Mcanwhile,
T( = = -1 1( x2)
where the second equality due the conditional independence 2 11. X2| in the Markov relationship.
Therefore, L achieves maximality X²x i/f ((2;; achieves MNI.
1(x:; x2) ;
1(26; X²), i.e. 2
L
D.2 PROOF OF PROPOSITION 2
Proposition 2. For random variables X and X2, when the IB curve x2) = F(T(Z); strictly concave,
1) there exists bijective mapping from [3r in Le to the value of information constraint de in the definition of optimal shared latent 2 in Equation I;
a1(2 ) 0,
01(2) xx²)
2)
O, where Z/3" is the optimal solution corresponding to certain B.
03
09
Proof. Since the IB curve T(Z!; x²) SIB( (I='; x!)) is monotonically non-decreasing and strictly concave, is monotonically increasing. When Z XX,, JIB achieves the maximum point (H("),, I(X;; X²)). Thus, / 1(x;; 2 I(x+; (²) - 1(26 ;²) 2 0 and monotonically de- creasing. Then there is bijective mapping between 5 = x(x T(2 x²) E [0, T(X X²)]
and points in the IB curve.
Since 1(2), xx = - -
we have
xx fin x3)) -1=
-1 Since fIB(r) S r, the equality holds when r = and fIB where SCIB(r) (fi -r)
strictly concave, we have
die < dr r=0 1. Thus dr
diB
4/d dr
and
dfCIB
1)-1 11
dr
dr
dip dr
Furthermore, since
< 0 (strict concavity of fIB), we have
df
dfc 2r
<
di
1)2
Nie
Therefore, fCIB is also monotonically increasing and strictly concave. Then there is bijective mapping between 5 I(x - I(Z!"; X²) [o, I(X!; X²)] and points in the CIB curve fCIB-
Since fCIB is increasing, the inequality constraint can be replaced by the equality constraint, i.c.
2!
J(2 (x)))) I(Z':''|x3>- (x)))-
arg min
arg min
The corresponding Lagrangian function is
L = 1==
I(X'; x²) +8 3 (T(XI; x(x+xx²) - 8)
-
=
Based on the Lagrangian multiplier theorem, the optimal -!! achieved when
dL
(I 3 3=0
di(z)
dl d1
17

---

Published as a conference paper at ICLR 2025
dL -1 + - d3
Thus,
(T xx3)
; 1(22-x)==((x------
L.e. is the slope of fein when ; IXX x 6. Therefore, there is bijective mapping
between, 3 = 3- in Lgg and points in fCIB, and thus 6.
For any B1, 32 > 0 and B1 > 82, we have 31 = = < 32 = 32. According to formula 7,
B1 df df (x(2)))))
((2 [12xx)=11222xx33
1(2xx))=1123 x²), B2 dl dI 1 X2)
dT dI X2)
Since 31
32 and fCIB is strictly concave function, JCiB is strictly convex and we have T( X2)). Furthermore, since fIB is monotonically increasing, have 1(Zg, X²). Therefore,
T X²)
1(Zg, X²)
a1(Z;; x²)
(1(Z); X²)
<0,
03
03
D.3 PROOF OF PROPOSITION 3
Proposition 3. If MNI is attainable for random variables X and X²,
arg max 1(2";x²(x3) arg max 1[Z 22x ; -I-X--X2
-1-X1-X2
where 22- is the representation based on X² that satisfies MNI, iee. ( 1 1(X;; )²).
Proof. Based on the graphical model, 28 11 X²2 Z have I 12;; x²) 0. Thus,
1 = ( +12
=
;2;; I(X x
(8)
Meanwhile,
- 1(x
= = 2 (2,) 2212)
where the second equality is due to the conditional independence X 11 | 22 in the graphical model.
When 2 is the MNI point 12x,((x): (2-) 1(X'; Thus.
T(XI; X2 x2(2)) ) T T(X x (26-) [(X!; (XX;X² 1(X1; 22))
where the first equality is due to the Markov relationship X <> X2 2?2
Then we have (xx X 2 = 1(x?; Due to the non-negativity of conditional mutual information,
-
Based on formula 8 and 9,
T(Z x Since > ,
and 1(x1; is a constant value irrelevant
=
to z1, maximizing 112 is equivalent to maximizing 1(21, x2; X1) 1(22, 22 X1).
18

---

Published as a conference paper at ICLR 2025
D.4 PROOF OF PROPOSITION 4
Proposition 4. For random variables X and X²,
0 S 1<(Z x , ) ; S dc
where 22 is the optimal representation based on X" with respect to 6, as defined in Equation 1, i.e. 22 = arg min za (122Xx² s.1. 1(x x²) ; - 6
Proof. According to formula and 9,
[(z',²2 1( =12= 22)
x |3 = -xx - x (2)
Since 22- is the optimal latent under 6, [ - ((22"x')) 5. Thus,
1(x x² 22:) = ; - ( 8
Meanwhile, ((21x2112) > 0. Then we have x(xt;x²1z', (2) 8- 10-1 s 8.
Therefore,
x² 0. 1(2), x²; 1(2), 22, 22, 1 22) <
iee. x2: - [(2), 12: x1) x 5 Since T(Z); X |
1(2) (1,'X2,X)) T(XI:x2) and ( is constant value ir-
relevant to Zt, maximizing xx is equivalent to maximizing 1(22 e
[T(Z], 22"; x , ;²)
E SUFFICIENT CONDITIONS FOR MNI
In this section, we explore the conditions for both MNI attainable and unattainable cases. Here, we use X and Y to represent the two modalities, rather than X and X in other sections. Although prior works have not provided a comprehensive necessary and sufficient condition for the attainability of MNI, and such analysis is beyond the scope of this paper, we provide several sufficient conditions for both MNI attainable and unattainable scenarios.
We first present the sufficient conditions under which MNI is attainable. As outlined in Proposition 5, Proposition 6, and Proposition 7, MNI attainable when the relationships between and Y either entirely deterministic or completely independent across each sub-domain. While the deterministic mapping might hold true when Y is the data label, it generally does not hold when X and Y are high-dimensional data modalities. In such cases, a deterministic relationship between the two modalities would imply that one can be fully inferred from the other, leaving no room for modality speecifi information in one of the modalities.
Proposition 5. For random variables X, Y, and representation derived from X (i.e., the Markov chain Z + X + Y holds), sufficient condition for MNI to be attainable is that X - Y mapping deterministic (Fischer, 2020).
Proof. Denote Y -(X) as the deterministic X mapping. the encoder is powerful enough, it can learn to reproduce the deterministic function f. i.e. Z = =(x) = Y. Thus T(X; 2) T(X;Y)= = HYY) = (Y:Z).
While Fischer (2020) identifies this as necessary condition for attainable MNI, there scenarios where X + Y is not deterministic, yet MNI is still attainable. We provide additional sufficient conditions for these more general cases in Proposition 6 and Proposition 7.
Proposition 6. For random variables X, Y, and representation Z derived from X (i.e., the Markov chain Z + X + Y holds), a sufficient condition for MNI to be attainable is that Y -t X mapping is deterministic.
19

---

Published as a conference paper at ICLR 2025
Proof.
Denote X f(Y) the deterministic Y X mapping. Then, for any Z with Z + X +
Y.
p(zly) / p(z)r))(zl))dd== / p(ae=
More formally, p(zly) p(z|X f(v)), and
p(Y -, pY M v, -,)x= =) pYY p(=|X f(w))
Thus the conditional entropy be written
H(ZY) = - -11 p(u, 2) log p(=l))dydz = J plu) / p(z|X = (og))) = f(y))dyd=
H(ZIX) =- - p(x)) / p(=(2c) logp(z (z(p(=))ddd:
=-
Thus, H(Z (()))), i.e. T)Y: I(X;Y) = (X), we achieve the MNI point.
Z): By selecting Z such that X)
Proposition's 7. For random variable X,Y in X x y with joint distribution p(X, Y), a sufficient condition for the existence of MNI is:
In any sub- uub-domain X, x Yo of X x y where p(X, Y) has the full support, iee V x, E X. X Vn, p(x, v) 0, X and Y are independent, i.e. X YK((XY)) X, x V.).
Proof. Construct the learned representation Z as the conditional probability of Y given X, i.e. Z = XX) Z is deterministic function based o X. We will show that such Z satisfies the MNI condition.
First, we show that ((2 I(X;Y). Denote p(Y| = x). Since & fully describes the conditional distribution p(Y X I), we have p(Y XX = x) = p = 2). Thus, HY X = x) H(Y 2). Therefore,
[(| f (())H((1X 2)dx= =x))= (())H(Y| =2)d
= ( p(x)dD)
H(YIZ =)d= = / (()))((112= = =2)dz H(YIZ)
Thus I(Z;Y) H(Y) H(Y H(Y) - H(Y X) = I(X;Y).
Second, we show that X(Z = I(Z:Y). Since I(Z:X) = H(Z) H(Z I(Z:Y) = 0, I(2;X) I(Z;Y) is equivalent H(Z 0, i.e.
( H())Y), and H(]
= (Y is determined by the value Y.
The condition is V X, V.,, X IL VH((X, Y) X, Vs), which indicates that
XX, x Vo, Vx,ye E X, Ve, p(y(z) p(v)
x())((a)) p()).)
For any E D, choose V. = {v). and X. = fo : p(zly) > 0}. we have p(y|z) = Px..
PQL) p(y), V x E X, Thus, V I1 sampled from p(X = y), we have p(y|=1) = 22) Therefore, p(Y x/X) is fully determined given the value of y and is irrelevant to the value of X. Then we show that T(Z; X)- T(ZYY).
is easy to see that deterministic mapping between X and Y is a special case of this condition.
Regarding the sufficient conditions for MNI being unattainable, outlined in Lemma 6 of Gilad- Bachrach et al. (2003), when variables X, have full support, ie. p(x,y) y) > 0, V x. y, MNI is unattainable.
20

---

Published as a conference paper at ICLR 2025
F DIFFERENTIABILITY OF THE IB CURVE
Definition 1. Let V and W be Banach spaces, Q an open set in V and F a function that maps Q2 into W. Then the Gateaux derivative of at æ E Qt in the direction h E V defined as
F(x + eh) F(x)
dF(x;; h)
1im c-0
F(x + ehi cm0
provided that this limit exists for all h E
In this section, tions. In our setting, the domain is the Banach space corresponding to the product space representing the set of pairs of joint probability distributions i.e. 2 x V. The functional where Q is the product space representing the set of pairs of joint probability distributions for which this mutual information defined. Following the Markov structure in our graphical model, we have
X and Y to represent the two modalities, rather than X and X²2 in other
(): : + R,
p(z,v) log p(z,4) p(z)p(u)
T(Z:Y)
U.z
p(z|0, v)p(2 v) log Je p(2|2, v)p(ale) Sa ((e(o)p(()
L. ((cl)))((al))
p(=lx)p(z11)) log z.y0= F(f)where f p(z(x)
J. ((z()))(())
Proposition 8. If the MNI point exists, then the Gateaux derivative of I(Y; Z) with respect to p(2(æ) (as used in (Pan et al., )) doesn't I exist at the MNI point.
Proof. Denote the MNI point between X and Y where I(X = ((Z;Y)= = X): When the MNI point exists, the curve can be represented as (Rodríguez Gálvez et al., 2020; Kolchinsky et al., 2018):
I(Z:Y) (1) < H(Y)
if < T(Z;X) s H(X)
or equivalently
((llv)= = p(zlz), if $ [1=g(=>), if HYY) < ((2;X) H(X)
Based on this characterization, we know that there exist a direction such that on perturbing p(z|æ) along this directions, either have 7(Z; Y) I(Z; H(Y) T T[Z;Y) T(X (Y) < T(;;)) Without loss of generality, assume that perturbing by ch, where e < 0 corresponds the former case and eh, where e 0 the latter. Consider the right directional derivative of I(Y:Z)
at in the direction h, we have
F(p + ch) -(())
d4 F(pph)) = lim +
Since have p(2ly) p(2[z) when p is perturbed in the direction by eh, where 0, we consider I(Z; Y) as an identity function of X); In such a case, F(p + eh1) = HYY + ch. As consequence, perturbing p(2|æ) to get p/ is equivalent the resulting perturbation in p(=ly) also
21

---

Published as a conference paper at ICLR 2025
being p'. As result, we have
F(p') - (p))
d_ d_F(p;h)
lim 6---
F(o(zly) = =p²) F())
lim
c-0-
d Flp + chi)
de
==0
de J. (p(zly) + ch)p(v) log p(=/y) + eh
d
jy,((zy)pp()) ==0
J hp(x) log p(zlu) + ch ( 2.
h
J,p((ly))(())
(p(=lv) eh p(z)
c==
h h 0
Clearly, the left and right limits exist but aren't equal, hence proving the non-differentiabiiityy
G TRACTABLE TRAINING OBJECTIVES
Four terms are involved in DISENTANGLEDSSL, including maximizing the mutual information term T(Z); x²) and minimizing the conditional mutual information term x| for the shared representations in step 1, as well as maximizing the joint mutual information term I(Z), ; and minimizing the mutual information term / 1(Z "2-) for the modality representations in step 2. We introduce detailed formulations of the tractable training objectives for each of the four terms in this section.
For the inferred shared representations, model the distributions 2 p(-IX)) and 22 p(-IX?) with neural network encoders. Following the common practice in Radford et al. (2021), we use the InfoNCE objective (Oord et al., 2018) as estimation of the mutual information term T(Z X2) in T.
exp(2!T 22+ /T)
LinfANCE
E 21 (2) H.
log
exp(x! 22 222/7) EN (p(2iT=?-/11
where is the temperature hyperparameter, 2² is the representation of the positive sample cor- responding to the joint distribution p p(X² X²), and {2] { ) are representations of N negative -
samples from the marginal distribution p p(X²).
We implement the conditional mutual information term [ x x in L using an upper bound developed in Federici et al. (2019), i.c. I [ x X²) DxL (p(2 2 x [p(2 X²)). While the conditional distributions of representations are modeled as the Gaussian distribution Federici et al. (2019), we instead the von Mises-Fisher (vMF) distribution for p( p(Z!! p(Z' and p p(22 better align with the InfoNCE objective where the representations lie on the sphere space. Specifically, ?! vMF(((X") k)) 2 ~ vMF(p((²), k) where к is a hyperparameter controlling for the uncertainty of the representations. Leveraging the formulation of the KL divergence between two vMF distributions, the training objective of T is to maximize:
4._ - LiMONNEE + E.EE (M(Zt) m(x²)
Note that this objective establish connections with the alignment versus uniformity framework dis- cussed in Wang Isola (2020), where the conditional information bottleneck constraint corresponds to higher weight on the alignment term, in contrast to the uniformity term.
The inferred modality-specific representations are encoded as functions 2! p(-|X1 2%) and 23 p(- 22) with deterministic encoders, that takes both data observations and the shared
22

---

Published as a conference paper at ICLR 2025
representations learned in step input to account for the dependence structure illustrated in Figure 2. The term T T(Z), n2, x²) in T.g is optimized with the InfoNCE loss, where random augmentations of the data X form the views. Denote the concatenation of Z and its corresponding Ze as 2 the InfoNCE objective has the following formula:
exp(21T21+)) /T) exp(3ITz(+/r) Σ exp(21T //T))
LRoNNE Bgt,at ( (E
log
(10)
where z1+ is the representation of the positive sample corresponding the augmented view of X and [2, - Tim1 are representations of N negative samples from the marginal distribution p(X
For the mutual information term between representations, I [Z' = in Li we implement it as an orthogonal loss to encourage the marginal independence between the shared and representations, where the marginal distribution is approximated across training batch.
Tonbaggona = |
(11)
where B is the batch size, | and [2 ===1 are the concatenations of all the representations in mini-batch, and I|F is the Frobenius norm the pairwise cosine similarities between each latent
dimensions. The training objective of L3 in step 2 is to maximize:
L - Lin X Lorthogonal
(12)
H EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
To evaluate the efficacy of our proposed DISENTANGLEDSSL. we conduct simulation study and two real-world multimodal experiments to address the following key questions:
How do B and À affect DISENTANGLEDSSS 's learned representations compared to baselines in controlled simulation setting with known ground truth?
Does DISENTANGLEDSS! achieve effective coverage of multimodal information in downstream prediction tasks from MultiBench (Liang et al., 2021), using a combination of shared and modality- specific representations?
How do DISENTANGLEDSSL 's representations perform in high- high-content drug screening datasets (Subramanian et al., 2017: Cuccarese al., 2020), assessed by polecule-phenoyye retrieval (for the shared) and disentanglement measurement (for modelity-specifie)
Each experiment was conducted on NVIDIA RTX A5000 GPU, each with 24GB of accelerator RAM. All experiments were implemented using the PyTorch deep learning framework.
H.1 SIMULATION STUDY.
Experimental details. We generate synthetic data X and X based on the graphical model in Figure 2, with the dimensionality of 100 and dataset size being 90,000. To be specific, sample 50-dimensional true latents Z2, Z and Z. independently from (O50. 0.5 Iso). Then we sample the transformation weights Ti and from uniform distribution Uniform( (1, 1), and generate x = 71 [z!, Ze] and X" = T2 [ Ze]. We randomly split 80% data into the training set and the rest into the test set. To simulate unattainable MNI, we add Gaussian noise and random dropout during training to ensure the distribution has full support. We use 3-layer multi-layer perceptron (MLP) with a hidden dimension 512 as encoders for all methods. For DMVAE, we employ MLPs with the same architecture as decoders. We report the average linear probing accuracy the test set over random seeds.
We all the methods the synthetic data with combinations of different hyperparameter values. For DISENTANGLEDSSL, we use 3 E (0.0, 0.001, 0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0, 300.0, 500.0, 1000.0) and À E {0.0, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0).
23

---

Published as a conference paper at ICLR 2025
For JointOpt, hyperparameter controls the joint mutual information terms 1(2), Ze: x²) and 1(22, 2,; x²), and À adjusts the mutual information term between represettttessn,nnionn,,, i.c.
2c 22, 22" arg max (12
(1(Z:: +1(22
We use a E {0.01, 0.1, 1.0, 10.0, 100.0, 1000.0) and E {0.0, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0).
For FOCAL, we tune the hyperparameters and defined similarly to JointOpt, where controls the terms 1(2);; ) and r(22; X²) and À adjusts the orthogonal loss between shared and specific representations. We use the same set of and that in JointOpt.
For DMVAE, we tune x which denotes the weight of the KI, divergence term in contrast to the reconstruction loss. We use À E {10-1, 10-6, 10 10 10-3 10 10-2..
For SimMMDG, we tune the hyperparameters and X, where a controls the weight of the cross-modal translation loss and À adjusts the distance loss between shared and specific representations. We use the same set of a and À as that in JointOpt.
Additional results. We provide comparison of the performance of the shared representation with baseline methods in Figure 8. Denote Ze the concatenation of the learned shared representations of X and X2, i.c. Z, [2g- 22). The ideal 2, should maximize the shared information between X and X2, shown by high accuracy on Yz, while minimizing unique information of X and X indicated by low accuracy on Y! and Y?? Therefore, a top-left point is preferred in Figure
As shown in Figure 8, DISENTANGLEDSSL consistently outperforms other methods across various hyperparameter settings, showcasing its ability to effectively capture shared information. The only exception occurs in Figure 8a when /3 is set very high and the accuracy on Y to drop to around 0.50 (equivalent to random guessing, indicating this scenario, DMVAE performs better. This happens because high values of 3 cause the decoder-free contrastive objectives collapse, with most representations converging nearly the same point, commonly-knen -nnoon issue in contrastive self-supervised learning (Oord et al., 2018).
information of 2, modality-speciffi features). In
feix00 601 Jetoryy
Jecakgs 10.1
Acpuracy of y.
Jecat0s am 1060
Jetx a 1000
0.75
0.75
Jntrrte 1060
POCAL,
066
FOCAL,
065
JOCAL,
FOCAL e- 16.0
(38 Avcure
FOCAL, o- 160
FOCAL, am 180.0
POCAL, o 100.
FocAL, n- 10000
FOCALL 10080
0.5
0.5
StaMMMI o 601
StMMMMK c:
Ses04MXX c:
StsMMDD 1.0
0.5
0.6
07
05
0.
07
106.3
Teal Accuracy of on V
SuMMMIX 1000
Tes Accutacy 2, on
StmitMXX 10600
Oun
Our
(a) Test accuracy of 2. on Y. versus V
(b) Test accuracy of 2. on Y. versus Y?
Figure 8: Performance of shared representation 2, for different models.
For the modality-speeifee representations, as supplement to Figure 4b, we provide the complete results for both representations 2 and 2 on full set of hyperparameters in Figure 9. DISENTAN- GLEDSSL outperforms all other methods across various hyperparameter settings.
Analysis for different levels of entanglement. We further examine the trade-off between expressivity and redundancy for the learned shared representations synthetic data with varying levels of multimodal entanglement. In this scenario, some dimensions align across modalities with MNI attainable, while others remain entangled with MNI unattainable. Specifically, dimensional true shared latent Z, into two parts: the first 35 dimensions, denoted as Z and the last 15 dimensions, denoted Zeure We then generate 100-dimensional observations X and X2, where
split the 50-
24

---

Published as a conference paper at ICLR 2025
Oxt,
9-01
Jeas m0
5-08
0.75
Jeardy m11
Jans 11
Accurccccyyo 2 V-
Jen0 1000
0--E1
Juan 1060
0.70
Jear (060
0-05
DMWAE
ForAl.
0.68
10
0.55
POCAL, 101
POCAL.
FOCAL
0.50
0=1020
5-500
FoCAL.+-1000
3-2020
0.55
0.55
1-1000.
Accuracy 23, on
0.55
0:60
Test Accurayy of 22 on V
(a) Performance of 2 for DISENTANGLEDSSL
across varying values of
(b) 2! performance DISENTANGLEDSSL and baselines.
JencOpaa
JencOpu
2-20
Jaandy;
2-3001
0.70
Jaandy: a
0.675
Jaandy: 1066
6.600
P-15
2-2.1
0.65
DOWAE
FOCAL +000
0.925
2-50
POCAL, a
POCAL, x-18
2-160
FOCAL.
2:300
FOCAL. 1060
0.75
7-100.
FOCAL. (060
Tel
2- 100
SasH0MDG c-001
0.50
0.55
SacH0MDG c-al
(-
SasMMMG e: 1.0
0.70
SasMMMDG 160
Accurny of 23
SasMMDG 1660
0.50
0.55
0.60
0.65
0,70
0.75
Accuracy 22 Y?
(c) Performance of 23 for DISENTANGLEDSSL. across varying values of B.
(d) / performance for DISENTANGLEDSSL. and baselines.
Figure Performance of coddalty-sppecifee representation 2 and 23 for different models.
the first 85 dimensions generated under the same procedure before, i.c. X1 =i [Z, 2 and X2 T2 [Z2, m followed by adding Gaussian noise and random dropout, while the last 15 dimensions are directly 2
We present the results on the synthetic data with mixed level of entanglement in Figure 10. Figure 10a shows the test accuracy of 2! on Y.. Y, and Y the left axis, alongside the MLP weight ratio on X and X2 on the right axis, for varying values of 3. To compute the MLP weight ratio, first extract the diagonal of the inner product of MLP encoder's 's first layer weight matrix, then calculate the ratio between the average of the last 15 "pure" dimensions and the first 85 "mixed" dimensions. This ratio indicates how much attention the encoder gives to the "pure" versus "mixed" dimensions, with higher values signifying greater focus on the "pure" dimensions. Figure 10b shows the corresponding test accuracy
versus Y in line plot.
Using such data with mixed entanglement levels, DISENTANGLEDSSL demonstrates a clear pattern where the learned information plateaus certain / values. As illustrated in Figure 10, the MLP weight ratio initially rises sharply from around 20 to nearly 80, then drops to 1 when 3 becomes very large, indicating the collapse of the learned representations. With large value of 3 (while before the model collapses, i.e. / x 10), the encoders focus mainly on the "pure" dimensions. This is because a stronger information bottleneck constraint discourages the extraction of shared components from the "mixed" dimensions, which inevitably include modality-speciiff information due to unattainable MNI, and favors the "pure" shared components with no extra cost.
In addition, we evaluated our method synthetic data with varying noise levels (i.e. variance of the Gaussian noise and rate of random dropout). As shown in Figure 11, as the noise level decreases, the learned shared representation 2r contains less modality-specific information under the same 33 and the test accuracy on Y, and Y2 decrease to close 0.5 even with small B. Meanwhile, the frontier of
25

---

Published as a conference paper at ICLR 2025
0.80
Ratio on X
0.75
Test Accuracy
Ratio X
60
Ratio
Test Accuracy on
50
40
0.65
20
Y
Y
Y?
10
Test Accuracy Y
(a) Test accuracy 2 and the MLP weight ratio with varying B.
Test accuracy on Y. versus Y?"
Figure 10: Performance of 2, on synthetic data with mixed level of entanglement.
test accuracy Y. versus Y1 shrinks towards the top left corner, indicating decreasing expressivity- redundancy tradeoff and easier disentanglement, These trends align well with our theoretical analysis.
0.675
large
0.75
Tost Accuray on V5
Test Accuracy on
0.000
0.70
0.595
0.65
large moise
mid noise
small noise
10
107
10
16
10
10²
0.500
0.525
0.550
0,575
2,100
0.628
0.650 0.675
Tesl Accurazy on X
(a) Performance of Z. Y1.
(b) Test accuracy on Y. versus Yi.
large noise
mid noise
stall lI noise
Test Avuursyy on
Ted Accursy on
0.70
0.600
0:575
0.6
0.550
large noise
0.525
0.60
mid noise
small II
10"
10"
0.500
00535
0.500
0.575
0,600
0.625
0.600 0.675
Tost Accurazy
(c) Performance 2. on Yz.
(d) Test accuracy Yo versus Y/
Figure 11: Performance shared representation 2, learned with different noise level. As the noise level decreases, 2e contains less modality-speccife information, and the frontier test accuracy on Y. versus Y, shrinks toward the top left, reflecting reduced expressivity-reduudangg tradeoff.
H.2 MULTIBENCH.
Dataset description. We utilize the real-world multimodal benchmark from MultiBench (Liang al., 2021). We provide brief description of each dataset below.
26

---

Published as a conference paper at ICLR 2025
MIMIC is large-scale dataset in healthcare, which contains over 40,000 ICU patient records. The two modalities are the time series patient records measured by hours and tabular static data (age, gender, etc.). The binary task label is whether the patient fits any ICD-9 code in group 7.
MOSEI a sentence-level multimodal sentiment and emotion benchmark with 23,000 monologue videos. The two modalities are the vision and text modalities, and the label is on whether the sentiment positive or negative corresponding to each video.
MOSI is a similar multimodal sentiment analysis dataset based on 2,199 YouTube video clips, with vision and text modalities and the sentiment label.
UR-FUNNY is humor detection dataset in human speech, which consists of samples from TED talk videos, with vision and text modalities and binary label on whether there is humor or not.
MUSTARD is corpus of 690 videos from popular TV shows for sarcasm detection. The vision and text modalities are utilized to classify a binary label of sarcasm.
Experimental details. We follow the same dataset splitting, and utilize the same encoder architecture and pre-extracted features Liang et al. (2024). All models use representations (or concatenation of them, for FactorCL-proj, FOCAL, JointOpt and DISENTANGLEDSSL) with the same dimensionality, We set the latent dimension to 300 across all datasets, except for MUSTARD, where it is set 100 due to the smaller dataset size, and for MIMIC, where we use 360 to align with the output dimension of the GRU encoders. For FactorCL, hyperparameters are tuned based on validation set performance. For DISENTANGLEDSSY we use 3 1.0 and À 10 3 for all datasets, except for MOSI where 0.01. For FOCAL and JointOpt, we use a = 1 and À 10 -3 across all datasets. We report the mean and standard deviation of the linear prohing aceuracy on prediction labels from the test set over random seeds.
use their default hyperparameter settings. For other methods,
Table 4: Prediction accuracy (%) of the representations learned by different methods on MultiBench
datasets and standard deviations over 3 random seeds.
Datuset
MIMIC
MOSEI
MOSI
UR-FUNNY
MUSTARD
CLIP
64.97 (0.60)
76.87 (0.45)
64.24 (0.88)
62.73 (0.92)
(4.19)
FactorCL-emb
65.25 (0.45)
71.80 (0.64)
62.97 (0.81)
63.29 (2.07)
56.76 (4.66)
FactorCL-proj
59.43 (1.70)
74.61 (1.65)
56.02 (1.26)
61.25 (0.47)
55.80
FOCAL (shared)
62.69 (1.46)
75.93 (0.23)
62.10
62.38 (0.58)
54.83 (3.02)
FOCAL (specific)
62.13 (1.49)
75.41 (0.54)
63.65 (1.27)
63.17 (0.96)
(2.21)
FOCAL (both)
64.42 (0.34)
76.77 (0.51)
62.98 (1.52)
(0.00)
JointOpt (shared)
63.01 (0.59)
76.69 (0.28)
(1.96)
62.51 (1.02)
54.83 (4.82)
JointOpt (specific)
65.81 (0.49)
74.40 (0.94)
53.89 (0.80)
62.13 (0.69)
57.73 (4.12)
JointOpt (hoth)
66.11 (0.64)
76.71 (0.14)
64.24
63.58 (1.45
56.52 (2.61)
DISENTANGLEDSSL (shared)
63.16 (0.48)
76.94 (0.22)
65.16 (0.81)
64.14 (1.53)
54.11 (1.51)
DISENTANGLEDSSSI (specific)
65.73 (0.09)
75.99 (0.60)
51.70 (0.72)
60.27 (1.28)
61.60 (2.61)
DISENTANGLEDSSL (both)
66.44 (0.31)
77.45 (0.06)
65.11 (0.80)
64.24 (1.54)
56.52 (2.18)
Additional results. As a complement to Table 1, we present results of FOCAL and JointOpt for the learned shared and modality-specific representations, both combined (i.e. "both") and separately in Table 4. Although they exhibit a similar trend as DISENTANGLEDSS. are more important for MOSI and specific representations are crucial for MUSTARD, the distinction is less pronounced. Moreover, the overall performance is inferior to that of DISENTANGLEDSSL. indicating that DISENTANGLEDSS! achieves better balance between coverage and disentanglement in the leamned representations.
where shared representations
Ablation study. We conduct an ablation study of DISENTANGLEDSSL on the MultiBench datasets to examine the impact of the hyperparameters 3 and X in Table 5. As 3 increases. the shared repre- sentations preserve only the most closely shared information and discard other features. Therefore, on datasets where the loosely shared information contributes to the label (eg., UR-FUNNY, MOSI, MIMIC), the prediction accuracy decreases with increasing 3. For the MUSTARD dataset where there exist contradictions between language and video (Liang et al., 2024), the accuracy of the shared representations increases with very high 3 since the contradicted parts are discarded and only clear signals remain. Similarly, as À increases, the independence constraint between the shared and specific representations is emphasized more. Thus, the moddllity-speciff representations contains less information, potentially related to the label, making their prediction accuracy decrease.
27

---

Published as a conference paper at ICLR 2025
Table 5: Prediction accuracy (%) of the representations learned by DISENTANGLEDSSL with different 33 and À and standard deviations over 3 random seeds.
Datasct
Shared
Specific
Both
0.01
0.001
64.08 (1.48)
62.04 (0.66)
64.43 (1.14)
1.0
0.0
64.14 (1.53)
61.81 (0.08)
64.24 (.96)
UR-FUNNY
1.0
0.001
64.14 (1,53)
60.27 (1.28)
64.24 (1.54)
1.0
0.1
64.14 (1.53)
60.46 (0.32)
64,46 (0.74)
100.0
0.001
62.70 (0.70)
58.76 (0.35)
61.72 (.83)
0,0001
0.001
65,16 (0.60)
51.70 (0.77)
64.97 (0.34)
0.01
0.0
65.16
51.36 (1.37)
64.97 (0.50)
MOSI
0.01
0.001
65.16 (0.81)
51.70 (0.72)
65.11 (0.80)
0.01
0.1
65.16 (0.81)
50.53 (1.31)
64.72 (0.83)
1.0
0.001
63.31 (0.70)
50.87 (1.15)
63.95 (0.77)
0.01
0.001
62.73 (0.26)
66.05 (0.39)
66.25 (0.36)
1.0
0.0
63.16 (0.48)
65.74 (0.20)
66.26 (0.35)
MIMIC
1.0
0.001
63.16
65.73 (0.09)
66.44 (0.31)
1.0
0.1
63.16 (0.48)
65.37 (0.48)
66.38 (0.19)
100.0
0.001
61.94 (0.29)
62.82 (0.85)
62.70 (0.63)
0.01
0.001
76.68 (0.21)
75.71 (0.43)
77.00 (0.14)
1.0
0.0
76.94 (0.22)
75.97 (0.52)
77.54 (0.19)
MOSEI
1.0
0.001
76.94 (0.22)
75.99 (0.60)
77.45 (0.06)
1.0
0.1
76.94 (0.22)
75.83 (0.39)
77.56 (0.11)
100.0
0.001
77.20 (0.16)
75.35 (0.72)
77.26 (0.27)
0.01
0.001
56.28 (1.90
58,70 (1.57)
55.80 (2.05)
1.0
0.0
54.11 (1.51)
60.14 (0.59)
56.28 (4.16)
MUSTARD
1.0
0.001
54.11 (1-51)
(2.61)
56.52 (2.18)
1.0
0.1
54.11 (1.51)
61.35 (2.08)
57.49 (1.49)
100.0
0.001
57.25 (3.13)
(4.7444.88
57.97 (2.05)
H.3 TIGH-CONTENT DRUG SCREENING
Experimental details. Following the setup in Wang et al. (2023a), we utilize Mol2vec (Jaeger et al., 2018) to featurize the molecular structures into 300-dimensional feature vectors. For both molecular structures and phenotypes, we employ 3-layer MLP encoders with a hidden dimension of 2560. The dimensionality for shared and modality-specifft representations is set to 32 across all methods and datasets. We tune B based on validation set performance, with 3 = 5.0 for RXRX19a and B = 1.0 for LINCS, and set À 0.01 for both datasets. For FOCAL and JointOpt, we set = 1.0 and A = 0.01. For DMVAE, we set the coefficient of the KI. divergence term as 10-5. In addition, as in Lee & Pavlovic (2021), I), we introduce an InfoNCE loss for the shared representations to DMVAE in the high-content drug screening experiments to enhance its retrieving accuracy. We report the mean and standard deviation of all results on the test set over 3 random seeds.
Counterfactual generation. To further evaluate the learned modality sppecfff- representations, different combinations of the shared and modality-speciftt representations to generate counterfactual samples, i.e. predicting phenotype of a drug a different cell with its molecule shared latent but the phenotype specific latent of a different cell perturbed by other drugs. Since the factual observations of counterfactual generation are unavailable, we measure the performance distributional- wise, introducing the difference in Frechet distance (Diff-FD) as a metric, i.c. Diff-FD-c measuring the gain of using the correct batch of molecules, thus the information of the shared latent; Diff-FD-s measures the gain of using the correct batch of cells, thus the information of the specific latent.
To be specific, we train decoders with the factual combination of the shared latents from molecular structures and the modality-speciff latents from phenotypes on the training set. We use "val recon" to denote the samples generated using the representations attained from validation set molecules and validation set phenotypes as input. Similarly, "test recon" refers to the samples generated using test set molecules paired with test set phenotypes, while "counterfactual" represents those generated using test set molecules paired with validation set phenotypes. Formally, Diff-FD-c is defined as FD(val recon; test recon)- FD(counterfactual; test recon), and Diff-FD-s as FD(val recon; test recon) FD(counterfactual; val recon). Note that non-informative specific latent can lead to very high -Dff-FD-c and an overly informative modality-specific latent can lead to very high Diff-FD-s, thus take both metrics into consideration together.
28

---

Published as a conference paper at ICLR 2025
As shown in Table 6, DISENTANGLEDSSL outperforms JointOpt in both metrics and surpasses FOCAL in Dift-FD-s. While FOCAL shows a high Dift-FD-s, it learns overly informative modality- specific latents. which contains significant shared information, as highlighted in the simulation study results (i.e. the modality-specific representations of FOCAL have high accuracy on Y, in Figure 9), leading effectiveness in capturing both shared and modality-ppecifie information, while maximizing the separation between them, indicated by its high values in both metrics.
low value of Diff-FD-c. In contrast, DISENTANGLEDSS
demonstrate better
Table 6: Results on counterfactual generation.
Dataset Metric
RXRX19a
LINCS
Diff-FD-c
Diff-FD-s
Diff-FD-c
Diff-FD-s
FOCAL
15.30(0.85)
22.89(0.28)
0.246(0.039)
0.765)
JointOpt
20.34(0.47)
9.40(0.39)
0.248(0.029)
0.333(0.050)
DISENTANGLEDSSL
20.92(0.71)
11.36(0.36)
0.277(0.031)
0,401(0.085)
I ALGORITHM PSEUDOCODE
Algorithm PyTorch-baseed pseudocode for DISENTANGLEDSSL
I: Notations: fe, f² represents the backhone encoder network for the shared representations. f1, f
represents the backbone encoder network for the modality-specific representations. 3 and X are the Lagrangian weights for the shared and sppeiitt--ppecific objectives respectively. vMF represents Mises-Fisher sampling.
2: / Step 1: Learn shared representations
3: for minibatch (x+ x²2) in dattallee oader do
4: &L VMF (f. (x') ), x), & vME (x
('Compute InfoNCE ListaNCE infonce (at.eooneessse_e__... 22))
shared representations",
7: / /"Shop 1: Objjcttiv for shared representations" /
8: L. = LiaaNCE B Elfo (x+) [=(2)
9:
backwwwarrd)
step_ LopmiLopmiizzztttep
11: / Step
modality regpllayyypppciiiceeeeenntttttn repnsentations
minibatch (a* _² datalcader
13:
Generate augmented views of the data (x) x²) and (x), 2z))
14:
1e Calculate the learned shared representations
15:
2 t (x.),, 22 + (x2)) Rd.b + (xt), &2, f- (x3)
16:
1" FEncude modality-sperifie representations conditioned on shared components /
$ + (= 2,, 22, +- (xāa 22 22,.), 21,6 f fo (ab. 2) ,), 22, + f2 20,6)
18:
1" CCompute modality-specifee losses")
19:
infonce Citence_Loss(concattto (2 2d concat| (0,))
+
cosocc_ccls(c.c. 22,)), concat *),
20:
Lorhh = Eze(a.b) Eie(, orthogonal_losseeeeeo
21:
22:
ba.backward()
23:
step 2 optimizer.seep
J NOTATIONS
29

---

Published as a conference paper at ICLR 2025
Table 7: Notations and mathematical definitions
Variahle(s)
Definition
Variabler of Data Generation
[X', (x3)
two modalities (such as image and text pairs) Shared latent representation capturing common information Modality-specifee latent representations for observations X,
observation or
Ze
zi
Inferred representations
25
Shared latent representation inferred only from X, i.c. 2 ~ p(-[X))
25
Modality-spe latent representations inferred jointly from observations X; and the inferred shared representations 2: i.c. 2; ~ p(-|X', 2
25-
Optimal shared latent representation inferred only from X;
2
Optimal modality-specifff latent representation inferred jointly from observations X; and the
optimal shared representations 2::
Optimization Parametery
6.
Small tolerunce to account for non-attainability of MNI for the shared representation Small tolerance coverage- coverage-disentangeee disentanglement trade-off' modality-speeiffc latent?
Lagrangian coefficient controlling trade-of''s in optimization for the shared representation Lagrangian coefficient controlling tradc-offs in optimization for the modality-specifat repre-
A
sentation
Objective be maximized learning the shared representation using Objective to be maximized for the learning the modality specific representation for modality i
17
Standard Notations and Definitions
MNI
Minimum Necessury Information Mutual information between variables Multivariate normal distribution Von Mises-Fisher distribution used modeling representations
I(:))
Ma, E]
VMF(p,))
30


Reviews:
[{"review_id": "AFlTSMyJR9", "reviewer": "Reviewer_nJHk", "summary": "The paper introduces a self-supervised learning approach to disentangle shared and modality-specific information in multimodal data. The authors also explain that the optimal case in prior work is not doable based on Minimum Necessary Information (MNI), as the comprehensive analysis of the optimality. The experiments on vision-language and molecule-phenotype retrieval tasks demonstrate its effectiveness.", "questions": "Please check the weakness section.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents a rigorous, information-theoretic framework for disentangling data for different domains and modalities under conditions where MNI is unattainable.\n\n2. Based on the theoretical analysis, the authors designed a two-step training algorithm, and provided an optimality guarantee for this proposed method theoretically.", "weaknesses": "Weakness:\nFor the experiments on synthetic data, they only contain the data for unattainable MNI. The synthetic experiments for attainable MNI are also essential here to demonstrate the reliability of the theoretical bound. My suggestion is to add Gaussian noise produced by different variances as the degrees of MNI attainable or unattainable.\n\n\n1. In Figure 4 (b), the superiority of DISENTANGLEDSSL is not significant compared to other baselines, such as JointOPT. The hyperparameter front of JointOPT is closer to the front of DISENTANGLEDSSL.\n\n2. Table 1 shows the three versions of DISENTANGLEDSSL are unstable. For example, DISENTANGLEDSSL (shared) can achieve promising performance on MOSI while failing on MUSTARD. It would be better to explain why such a phenomenon exists, probably focusing on the main difference between these three versions and properties of multi-modal data. \n\n3. The performance gain shown in Table 2 is quite limited. Could the author explain if small improvements might be important in this domain? In addition, it would be better if the authors could provide the reason for this tiny improvement. \n\n4. The paper does not satisfy the page limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lT2QQV0Mfo", "reviewer": "Reviewer_H7Jq", "summary": "In this paper, the authors proposed a novel self-supervised representation learning method for multimodal representation learning. The proposed method disentangles the representation of multimodal information into shared information and modality-specific information, and uses a separate neural network to learn each representation. The authors supported their proposed method through extensive theoretical analysis and proofs. The proposed method was evaluated on both synthetic datasets and real-world multimodal datasets, where the proposed method achieved top performance over baselines in most tasks.", "questions": "Although in most cases DisentangledSSL-both achieves highest performance, there are also several tasks where either  DisentangledSSL-shared or  DisentangledSSL-specific alone achieves top performance (and sometimes by quite a large margin over  DisentangledSSL-both, such as Mustard). Do you observe any trend / suggest any guidelines for when it is best to use each configuration?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a new perspective in how to perform effective multimodal representation learning. Specifically, the paper demonstrated that disentanglement of modality-specific and shared information into 2 separate representations can be effective in improving downstream task performance, which seems to be original and novel.\n\n2. The authors provided extensive theoretical justifications and proofs for their proposed approach.\n\n3. The experiments are quite comprehensive. They include both synthetic and real-world datasets, and the proposed method is evaluated against several multimodal SSL baselines on different tasks from drastically different domains. The proposed method achieved top performance in almost all tasks.\n\n4. All experiment details are presented in the Appendix (high reproducibility).", "weaknesses": "While the extensive use of information-theory notations allowed rigorous proof of the proposed approach, it isn't really easy for readers who isn't interested in the proofs and just want to learn about the concrete algorithm/methodology. Perhaps the authors should consider including a Pseudocode/Alg block either in the main text or in Appendix to clearly demonstrate the training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVr5LGnyCN", "reviewer": "Reviewer_DHvL", "summary": "The authors believe that \"shared information between modalities is precisely what is relevant for downstream tasks\" and \"the modality gap ... results in misalignment between modalities, restricting the application of these methods in numerous real-world multimodal scenarios.\" Hence they motivate \"the need for a disentangled representation space that captures both shared and modality-specific information\". To address this need, they propose DISENTANGLEDSSL, an information-theoretic framework designed to learn disentangled shared and modality-specific representations of multimodal data. The authors have conducted a theoretical analysis to evaluate the quality of disentanglement, which can be applied even in cases where Minimum Necessary Information (MNI) is unattainable. The efficacy of DISENTANGLEDSSL is demonstrated across a range of synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.", "questions": "Please respond to the weaknesses listed above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It makes sense to learn disentangled shared and modality-specific representations of multimodal data. \n\nThe theoretical analysis to evaluate the quality of disentanglement, which can be applied even in cases where Minimum Necessary Information (MNI) is unattainable, is provided.\n\nThe efficacy of DISENTANGLEDSSL is demonstrated across a range of synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.", "weaknesses": "Some of the key assertions motivating this work are inaccurate, and the proposed graphical model is flawed, which together raise questions about the validity of the resulting framework.\n\nFor instance, the statement \"shared information between modalities is precisely what is relevant for downstream tasks\" is not universally accurate. What is relevant depends heavily on the specific downstream tasks. Furthermore, \"shared information\" is a vague term, especially when applied across modalities, making it problematic in this context.\n\nThe assertion that \"the modality gap ... results in misalignment between modalities, restricting the application of these methods in numerous real-world multimodal scenarios\" is also oversimplified. The modality gap is not the sole or primary cause of misalignment; other factors such as alignment methods, distributional shifts, domain shifts, sampling biases, and more can contribute significantly.\n\nI suggest the authors to refine their motivation to more accurately reflect the complexities of multimodal relationships, while still maintaining the core idea of disentangling shared and modality-specific information.\n\nThe graphical model in Figure 2, intended to represent the generative process (as claimed in line 128), has two major issues. First, the authors have conflated the generative and inference processes. Only the pathway from Z to X represents the generative model, while the path from X to \\hat{Z} corresponds to inference, which should not be included in the data-generating graphical model. This issue is straightforward to resolve by removing the inference process from the model, which would not affect the algorithm or results, or to create separate diagrams, one for the generative process and the other for inference to clarify.\n\nThe second issue is more severe: the assumption of a shared latent variable Z_c existing between two modalities may not hold. This assumption lacks foundation, as it is more likely that the relationship takes the form of two variables Z_c^1 and Z_c^2, with potential connections between them:  (1)  Z_c^1 -> Z_c^2, (2)  Z_c^1 <- Z_c^2, or (3) or Z_c^1 <-C-> Z_c^2, depending on the modalities involved. This discrepancy significantly undermines the justification for the proposed algorithm.  I encourage the authors to discuss the implications of using separate but potentially connected latent variables (Z_c^1 and Z_c^2) instead of a single shared Z_c, and how might this change affect the proposed algorithm and results. This could lead to a more nuanced and realistic model of multimodal relationships.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Fq5AWd9wB", "reviewer": "Reviewer_bzKx", "summary": "This paper addresses the disentanglement of modality-shared and modality-specific information in multi-modal self-supervised learning and proposes DisentangledSSL. From an information-theoretic perspective, the modality-shared information is optimized using a conditional entropy bottleneck (CEB). Correspondingly, the authors formulate an optimization problem to isolate modality-specific information. Theoretical analysis of the optimality of each disentangled representation, particularly when Minimum Necessary Information is unattainable, along with experimental results, demonstrates the superiority of DisentangledSSL.", "questions": "Are there any alternative approaches for implementing the $I(Z^1;\\hat{Z}_{c}^{1*})$ in $L_s^1$? If so, how does their performance compare to that of the orthogonal loss implementation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.The idea of applying conditional entropy bottleneck (CEB) in multi-modal self-supervised learning is novel.\n\n2.Comprehensive theoretical analysis is conducted to prove the optimality of both modality-specific and modality-shared information.\n\n3.DisentangledSSL demonstrates superior performance across tasks, including vision-language prediction and molecule-phenotype retrieval.", "weaknesses": "1.How does the optimization objective for the modality-shared representation change from Eq. (1) to Eq. (3)? Why is $I(X^1;X^2)$ omitted?\n\n2.Related studies, such as CoCoNet [1] and SimMMDG [2], which also address the separation of modality-shared and modality-specific information, are recommended for inclusion and comparison. Note that SimMMDG [2] can be easily adapted to a self-supervised setting by substituting supervised contrastive learning with the original self-supervised counterpart.\n\n3.Ablation studies on the impact of $\\beta$ and $\\lambda$ in the MultiBench datasets should be provided to demonstrate the importance of separating modality-specific and modality-shared information in real-world applications. \n\n4.Some typos.\n1)$Z^2$ should be $X^2$ in Eq.(6) and in the equation below Eq.(6).\n2)$I(X^1;Z^1)$ should be $I(X^1;X^2)$ in Figure 3.\n \n[1] Li J, Qiang W, Zheng C, et al. Modeling multiple views via implicitly preserving global consistency and local complementarity. TKDE, 2022.\n[2] Dong H, Nejjar I, Sun H, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization. NIPS, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)