An Information Criterion for Controlled Disentanglement of Multimodal Data

## Abstract

Abstract Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose Disentangled S elf- S upervised L earning ( DisentangledSSL ), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called Minimum Necessary Information (MNI) point is not attainable. We demonstrate that DisentangledSSL successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data.

## 1 Introduction

Humans understand and interact with the world using multiple senses, each providing unique and complementary information essential for forming a comprehensive mental representation of the environment.
To emulate human-like perception,
multimodal representation learning (Bengio et al., ; Liang et al., ) seeks to decipher complex systems by combining information from multiple modalities into holistic representations, showcasing significant applications across fields from vision-language (Yuan et al., ; Lu et al., ; Radford et al., ) to biology (Yang et al., ; Zhang et al., ; Wang et al., ).
Large multimodal representation learning models such as CLIP (Radford et al., ), trained through self-supervised learning, maximally capture the mutual information shared across multiple modalities, exploiting the assumption of multi-view redundancy (Tosh et al., ; Sridharan & Kakade, ).
This property indicates that shared information between modalities is exactly what is relevant for downstream tasks.

However, the modality gap, rooted in the inherent differences in representational nature and information content across modalities (Liang et al., ; Ramasinghe et al., ; Huh et al., ), leads to the misalignment between modalities and restricts the application of these methods in many real-world multimodal scenarios.
This highlights the need for a disentangled representation space that captures both shared and modality-specific information effectively, elaborated as follows:

- •
Coverage: Distinct modalities often contribute unique, complementary information crucial for specific tasks (Liang et al., ; Liu et al., ), making it essential to capture modality-specific features effectively. For instance, in multimodal sentiment analysis, text conveys explicit sentiment, while vocal tone and facial expressions provide nuanced emotional cues.
- •
Disentanglement: As emphasized by Zhang et al. () in biological contexts, separating shared from modality-specific information is vital for interpretability and decision-making. However, maximizing mutual information across modalities can blur this separation
(Fischer, ), reducing stability and robustness for tasks like cross-modal translation and counterfactual generation.

Disentangled representation learning in multimodal data traces back to seminal works in Variational Autoencoders and Generative Adversarial Networks (Lee & Pavlovic, ; Daunhawer et al., ; Denton et al., ; Gonzalez-Garcia et al., ), aiming to isolate the underlying factors of variation within the data.
Recent studies have increasingly explored self-supervised learning methods to capture both shared and modality-specific information (Liu et al., ; Li et al., ; Zhang et al., ). In particular, information-theoretic approaches model modality-specific information as the complement of shared information, learning these representations either sequentially (Sanchez et al., ) or jointly (Liang et al., ; Pan et al., ). However, none of these methods provide rigorous definitions or theoretical guarantees on the optimality of disentangled representations, particularly under the multimodal scenario where the Minimum Necessary Information (MNI) point between the two modalities, as introduced in Fischer (), is often unattainable.
In simple terms, the MNI criterion characterizes $Z$ as a representation of $X^{1}$ that perfectly captures all the important information needed to understand $X^{2}$, without containing any unnecessary details.
Essentially, knowing $Z$ gives you the same understanding of $X^{2}$ as directly knowing $X^{1}$, and vice versa(^1^11Formally, $I(Z;X^{1})=I(Z;X^{2})=I(X^{1};X^{2})$ (Fischer, ), which we further explain in Section 2.2.1.).
However, the shared and modality-specific information is deeply entangled in a wide range of real-world applications, i.e. the shared and modality-specific components
are intertwined and result in similar observations, leading to unattainable MNI. Figure 1 illustrates an example
in the context of high-content drug screens. Specifically, the two modalities capture distinct but related aspects of drug mechanism and cancer cell viability, and extracting precise shared features from each modality is not feasible.
We provide additional examples of unattainable MNI in Appendix A.

Figure: Figure 1: Post-perturbation phenotype ($X_{1}$) (i.e., cellular images or gene expression after the application of a drug to cells) and molecular structure ($X_{2}$) of an underlying drug perturbation system where cancer cells are targeted and killed while healthy cells remain unaffected. The Venn diagram illustrates shared and specific information between modalities $X_{1}$ and $X_{2}$: shared content is shown in red, modality-specific content in green, and entangled content due to unattainable MNI in orange. For example, for the drug mechanism, the molecular structure conveys full information, while the phenotype provides partial information (i.e. mechanisms causing cell death). Similarly, for the states of healthy cells, the phenotype specifies their cell states, whereas the molecular structure only indicates that the cells are unaffected without detailing their specific states.
Refer to caption: extracted/5969051/arxiv/figs/mainfig1_bio.png

In this work, we propose Disentangled Self Supervised Learning (DisentangledSSL), a self-supervised representation learning approach for multimodal data that effectively separates shared and modality-specific information. Building on principles of information theory, we devise a step-by-step optimization strategy to learn these representations and maximize the variational lower bound on our proposed objectives during model training. Unlike existing works, we specifically address the challenging and frequently encountered scenarios where MNI is unattainable. We further articulate a formal analysis of the optimality of each disentangled representation and offer theoretical guarantees for our algorithm’s ability to achieve optimal disentanglement regardless of the attainability of the MNI point. To the best of our knowledge, this establishes the first set of analyses on the disentangled representations under such settings.
Empirically, we demonstrate that DisentangledSSL successfully achieves both distinct coverage and disentanglement for representations on a suite of synthetic datasets and multiple real-world multimodal datasets. It consistently outperforms baselines on prediction tasks in the multimodal benchmarks proposed in Liang et al. (), as well as molecule-phenotype retrieval tasks in high-content drug screens datasets: LINCS gene expression profiles (Subramanian et al., ) and RXRX19a cell imaging profiles (Cuccarese et al., ).
To summarize, the main contributions of our work are:

- •
We propose DisentangledSSL, an information-theoretic framework for learning the disentangled shared and modality-specific representations of multimodal data.
- •
We present a comprehensive theoretical framework to study the quality of disentanglement that generalizes to settings when the Minimum Necessary Information (MNI) is unattainable. We prove that DisentangledSSL is guaranteed to learn the optimal disentangled representations.
- •
Empirically, we demonstrate the efficacy of DisentangledSSL across diverse synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.

## 2 Method

In this section, we detail our proposed method, DisentangledSSL, for learning disentangled representations in multimodal data.
We begin by outlining the graphical model that formalizes the problem in Section 2.1 and defining the key properties of “desirable” representations in Section 2.2. Subsequently, we describe the DisentangledSSL framework in Section 2.3 and provide theoretical guarantees on the optimality of the learned representations. Finally, in Section 2.4, we present the tractable training objectives that facilitate efficient learning for each term of our method.

### 2.1 Multimodal Representation Learning with Disentangled Latent Space

DisentangledSSL learns disentangled representations in latent space, separating modality-specific information from shared factors across paired observations ($X^{1}$, $X^{2}$). This generative process is modeled in Figure 2. Each observation is generated from two distinct latent representations:
the modality-specific representations ($Z_{s}^{1}$ and $Z_{s}^{2}$) that contain information exclusive to their respective modalities, and a shared representation ($Z_{c}$) that contains information common to both modalities. We refer to these as the true latents.

Figure: Figure 2: Graphical model.

DisentangledSSL infers the shared representation from both modalities independently, i.e. $\hat{Z}^{1}_{c}\sim p(\cdot|X^{1})$ and $\hat{Z}^{2}_{c}\sim p(\cdot|X^{2})$. The modality-specific information for each modality is encoded by variables $\hat{Z}^{1}_{s}$ and $\hat{Z}^{2}_{s}$. Note that for the true latents, $Z_{s}^{1}$ and $Z_{c}$ are conditionally dependent on $X^{1}$ due to the V-structure in the graphical model. To preserve such dependencies in the inferred latents,
$\hat{Z}^{1}_{s}$ and $\hat{Z}^{2}_{s}$ are conditioned on both the respective observations and the inferred shared representations, with $\hat{Z}^{1}_{s}\sim p(\cdot|X^{1},\hat{Z}^{1}_{c})$ and $\hat{Z}^{2}_{s}\sim p(\cdot|X^{2},\hat{Z}^{2}_{c})$.

### 2.2 Information Criteria for the Optimal Inferred Representations

We establish information-theoretic criteria to ensure the shared and modality-specific representations are informative and disentangled, capturing key features while minimizing redundancy.

#### 2.2.1 Information Bottleneck Principle and Minimum Necessary Information

The shared representations $\hat{Z}_{c}^{1}$ and $\hat{Z}_{c}^{2}$ should effectively balance compactness and expressivity, as studied by the information bottleneck (IB) principle in both supervised and self-supervised settings (Tishby et al., ; Shwartz-Ziv & Tishby, ; Shwartz-Ziv & LeCun, ).
The IB objective seeks to optimize the representation $Z^{1}$ of an observation $X^{1}$ in relation to a target variable $X^{2}$, following the Markov chain $Z^{1}\leftarrow X^{1}\leftrightarrow X^{2}$. It balances the trade-off between preserving relevant information about $X^{2}$, i.e. $I(Z^{1};X^{2})$, and compressing the representation, i.e. $I(Z^{1};X^{1})$ (see more details in Appendix B).
The optimal representation should be both sufficient, i.e. $I(Z^{1};X^{2})=I(X^{1};X^{2})$ (Achille & Soatto, ; Shwartz-Ziv & LeCun, ), and minimal (Achille & Soatto, ).
Based on these criteria, $Z^{1}$ is said to capture Minimum Necessary Information (MNI) between $X^{1}$ and $X^{2}$ if the following holds (Fischer, ):

$$ $I(X^{1};X^{2})=I(Z^{1};X^{2})=I(Z^{1};X^{1})$ $$

MNI characterizes an ideal scenario between $X^{1}$ and $X^{2}$, where $Z^{1}$ captures complete information about $X^{2}$ with no extraneous information, i.e. $I(Z^{1};X^{1}|X^{2})=0$(^2^22 $I(Z^{1};X^{1}|X^{2})=I(Z^{1};X^{1},X^{2})-I(Z^{1};X^{2})=I(Z^{1};X^{1})-I(Z^{1 };X^{2})=0$, where the first equality is due to the property of conditional mutual information, and the second is due to the Markov structure $Z^{1}\text{-}X^{1}\text{-}X^{2}$.). It may not be attainable for an arbitrary joint distribution $p(X^{1},X^{2})$ (see Appendix E), particularly in general multimodal self-supervised settings (e.g. Figure 1 and Appendix A). Despite its significance, a comprehensive discussion of representation optimality when MNI is unattainable is often overlooked in prior work.

#### 2.2.2 Optimal Shared Representations: MNI Attainable or not

We propose a definition of optimality for the shared representations that applies to both scenarios – when MNI is attainable or not, as defined in Equation 1:

$$ $\begin{split}\hat{Z}_{c}^{1*}&=\operatorname*{arg\,min}_{Z^{1}}I(Z^{1};X^{1}|X ^{2}),\text{ s.t. }I(X^{1};X^{2})-I(Z^{1};X^{2})\leq\delta_{c}\\ \hat{Z}_{c}^{2*}&=\operatorname*{arg\,min}_{Z^{2}}I(Z^{2};X^{2}|X^{1}),\text{ s.t. }I(X^{1};X^{2})-I(Z^{2};X^{1})\leq\delta_{c}\end{split}$ (1) $$

Formally, minimizing the conditional mutual information, $I(Z^{1};X^{1}|X^{2})$, ensures that the shared representation captures only the information that is truly common to both $X^{1}$ and $X^{2}$, while discarding modality-specific details unique to $X^{1}$. Compared with $I(Z^{1};X^{1})$ in IB, it provides a more precise measure of compression and a more robust objective.

Figure: Figure 3: IB Curve
Refer to caption: extracted/5969051/arxiv/figs/IBcurve.png

The constraint $I(X^{1};X^{2})-I(Z^{1};X^{2})\leq\delta_{c}$ ensures that $\hat{Z}_{c}^{1*}$ retains a substantial portion of the shared information between $X^{1}$ and $X^{2}$, controlling the difference within the limit $\delta_{c}$ and preventing significant information loss.
We utilize the IB curve $F(\delta)$(^3^33The IB curve is concave (Gilad-Bachrach et al., ), monotonically non-decreasing, and upper bounded by line $I(Z^{1};X^{2})=I(Z^{1};X^{1})$ and line $I(Z^{1};X^{2})=I(X^{1};X^{2})$ (see details and proofs in Appendix C).) (Kolchinsky et al., ; Gilad-Bachrach et al., ), representing the maximum $I(Z^{1};X^{2})$ for a given compression level $I(X^{1};Z^{1})\leq\delta$, to illustrate the optimality in Figure 3. MNI is depicted as point A, and $\hat{Z}_{c}^{1*}$ corresponding to $\delta_{c}$ is shown as point C. When MNI is attainable, setting $\delta_{c}=0$ achieves MNI.

In contrast,
Achille & Soatto () formulated the optimization as $Z^{1}=\operatorname*{arg\,min}_{Z^{1}:Z^{1}\text{-}X^{1}\text{-}X^{2}}I(X^{1};
Z^{1}),\textit{s.t. }I(Z^{1};X^{2})=I(X^{1};X^{2})$, leading to MNI when attainable. This holds in supervised settings, assuming the data label $X^{2}$ is a deterministic function of $X^{1}$, as used by previous methods (Kolchinsky et al., ; Fischer, ; Pan et al., ).
However, in general multimodal self-supervised scenarios where MNI is not attainable, this results in point B in Figure 3, which includes information of $X^{1}$ that has little relevance to $X^{2}$ to satisfy the equality constraint, causing a gap between the objective and the ideal representation.

#### 2.2.3 Optimal Specific Representations: Ensuring Coverage and Disentanglement

Optimal modality-specific representations, $\hat{Z}_{s}^{1}$ and $\hat{Z}_{s}^{2}$, should capture information unique to each modality, being highly informative while minimizing redundancy with the shared representations. We hence define them via the optimization problems

$$ $\begin{split}\hat{Z_{s}^{1*}}&=\operatorname*{arg\,max}_{Z^{1}}I(Z^{1};X^{1}|X ^{2}),\text{ s.t. }I(Z^{1};\hat{Z_{c}^{1*}})\leq\delta_{s}\\ \hat{Z_{s}^{2*}}&=\operatorname*{arg\,max}_{Z^{2}}I(Z^{2};X^{2}|X^{1}),\text{ s.t. }I(Z^{2};\hat{Z_{c}^{2*}})\leq\delta_{s}\end{split}$ (2) $$

Take $\hat{Z}_{s}^{1*}$ as an example. By maximizing the conditional mutual information, $I(Z^{1};X^{1}|X^{2})$, we seek to extract the unique information inherent to modality $X^{1}$ that is not redundant with what is shared with $X^{2}$. Note that the same term is minimized in the shared representations as described in Equation 1. Integrating both the shared and the modality-specific representations together, the optimization objectives ensure full coverage of the entire information spectrum of each modality.

Meanwhile, the constraint $I(Z^{1};\hat{Z}_{c}^{1*})\leq\delta_{s}$ minimizes the overlap between modality-specific and shared representations, enhancing disentanglement by effectively separating the unique aspects of $X^{1}$ from the components shared with $X^{2}$.
The parameter $\delta_{s}$ controls the trade-off between coverage and disentanglement.

### 2.3 DisentangledSSL : a Step-by-Step Optimization Algorithm

To achieve the optimal representations discussed in Section 2.2, we introduce a two-step training procedure. The first step focuses on optimizing the shared latent representation, ensuring it captures the minimum necessary information as close as possible.
Building upon this, the second step utilizes the learned shared representations in step 1 to facilitate the learning of modality-specific representations. This sequential approach is formalized in the optimization objectives given in Equations 3 and 4:

Step 1: Learn the shared latent representations by encouraging the shared representation encoded from one modality to be highly informative about the other modality, while minimizing redundancy.

$$ $\begin{split}\hat{Z}_{c}^{1*}&=\operatorname*{arg\,max}_{Z^{1}}L_{c}^{1}= \operatorname*{arg\,max}_{Z^{1}}I(Z^{1};X^{2})-\beta\cdot I(Z^{1};X^{1}|X^{2}) \\ \hat{Z}_{c}^{2*}&=\operatorname*{arg\,max}_{Z^{2}}L_{c}^{2}=\operatorname*{arg \,max}_{Z^{2}}I(Z^{2};X^{1})-\beta\cdot I(Z^{2};X^{2}|X^{1})\end{split}$ (3) $$

Step 2: Learn the modality-specific latent representations based on the learned shared representations from step 1.

$$ $\begin{split}\hat{Z}_{s}^{1*}&=\operatorname*{arg\,max}_{Z^{1}}L_{s}^{1}= \operatorname*{arg\,max}_{Z^{1}}I(Z^{1},\hat{Z}_{c}^{2*};X^{1})-\lambda\cdot I (Z^{1};\hat{Z}_{c}^{1*})\\ \hat{Z}_{s}^{2*}&=\operatorname*{arg\,max}_{Z^{2}}L_{s}^{2}=\operatorname*{arg \,max}_{Z^{2}}I(Z^{2},\hat{Z}_{c}^{1*};X^{2})-\lambda\cdot I(Z^{2};\hat{Z}_{c} ^{2*})\end{split}$ (4) $$

The hyperparameters $\beta$ and $\lambda$ control the trade-off between relevance and redundancy for the shared and modality-specific representations respectively. We use the same values of $\beta$ and $\lambda$ for both modalities since they operate on similar information scales. Our sequential training approach, instead of a joint one, stems from the self-sufficient nature of each optimization procedure where one sub-optimal representation does not enhance the learning of the other. We offer a comprehensive analysis of the optimality guarantees of this step-by-step method as follows.

#### 2.3.1 Optimality Guarantee for the Learned Shared Representations

This section explores how the step 1 objective, $L_{c}^{1}$, optimizes the shared representation between modalities by balancing expressivity and redundancy. We discuss its effectiveness in both scenarios–when MNI is attainable or not.

$L_{c}^{1}$ seeks representation $Z^{1}$ that maximizes the information shared between the modalities, i.e. $I(Z^{1};X^{2})$, while minimizing the information unique to each modality, i.e. $I(Z^{1};X^{1}|X^{2})$, to capture only the essential shared content. This aligns with the conditional entropy bottleneck (CEB) objective (Fischer, ), an extension of the IB Lagrangian $L=I(Z^{1};X^{2})-\tilde{\beta}I(Z^{1};X^{1})$ (see details in Appendix B). While it serves as a robust objective for learning shared information, its optimality remains underexplored in Fischer (), particularly when MNI is unattainable. Additionally, Fischer () focuses on the supervised scenario where $X^{2}$ is the label of $X^{1}$, whereas we address the multimodal case with $X^{1}$ and $X^{2}$ being two data modalities, demonstrating its effectiveness in both attainable and unattainable MNI scenarios.

When MNI is attainable, Proposition 1 states that the step 1 optimization achieves MNI for any positive $\beta$. The proof is given in Appendix D.

###### Proposition 1 .

If MNI is attainable for random variable $X^{1}$ and $X^{2}$, maximizing $L_{c}^{1}=I(Z^{1};X^{2})-\beta I(Z^{1};X^{1}|X^{2})$ achieves MNI for any $\beta>0$, i.e. $I(\hat{Z}_{c}^{1*};X^{1})=I(\hat{Z}_{c}^{1*};X^{2})=I(X^{1};X^{2})$, where $\hat{Z}_{c}^{1*}:=\operatorname*{arg\,max}_{Z^{1}-X^{1}-X^{2}}L_{c}^{1}$.

When MNI is unattainable, the inequalities $I(Z^{1};X^{1})\geq I(Z^{1};X^{2})$ and $I(X^{1};X^{2})\geq I(Z^{1};X^{2})$ can not achieve equality simultaneously.
This indicates an inherent trade-off between capturing the entire shared information and avoiding the inclusion of modality-specific details. More precisely, under the condition of strict concavity of the $I(Z^{1};X^{2})-I(Z^{1};X^{1})$ information curve(^4^44When the IB curve is not strictly concave (i.e. partially linear), the properties hold except the linear sections. In these cases, methods like squared IB (Kolchinsky et al., ) can be used to achieve bijection mapping.), such a trade-off is presented in Proposition 2. The proof is given in Appendix D.

###### Proposition 2 .

For random variables $X^{1}$ and $X^{2}$, when the IB curve $I(Z^{1};X^{2})=F(I(Z^{1};X^{1}))$ is strictly concave,
1) there exists a bijective mapping from $\beta$ in $L_{c}^{1}$ to the value of information constraint $\delta_{c}$ in the definition of optimal shared latent $\hat{Z}_{c}^{1*}$ in Equation 1;
2) $\frac{\partial I(Z_{\beta}^{1*};X^{1})}{\partial\beta}<0,\ \frac{\partial I(Z_
{\beta}^{1*};X^{2})}{\partial\beta}<0$, where $Z_{\beta}^{1*}$ is the optimal solution corresponding to a certain $\beta$.

The second property implies that as $\beta$ increases, the learned representation becomes less informative and redundant, demonstrating the trade-off between expressivity and redundancy when MNI is unattainable. Furthermore, the bijection mapping between $\beta$ and $\delta_{c}$ allows our step 1 optimization to navigate the information frontier across various $\beta$ values, facilitating soft control and the segmentation of shared information into different degrees of granularity.

###### Proposition 1 .

###### Proposition 2 .

#### 2.3.2 Optimality Guarantee for the Learned Modality-Specific Representations

In this section, we demonstrate the step 2 objective, $L_{s}^{1}$, ensures optimal coverage and disentanglement by showing its equivalence (or nearly equivalence) to the Lagrangian of Equation 2.

$L_{s}^{1}$ in Equation 4 learns the modality-specific representation $Z^{1}$ based on the optimal shared representations $\hat{Z}_{c}^{1*}$ and $\hat{Z}_{c}^{2*}$ learned from step 1. It maximizes the information coverage of the data $X^{1}$ through the combination of $Z^{1}$ and $\hat{Z}_{c}^{2*}$, i.e. $I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$. Simultaneously, it promotes disentanglement by limiting overlap with the shared representation $\hat{Z}_{c}^{1*}$ of the same modality, indicated by $I(Z^{1};\hat{Z}_{c}^{1*})$. This objective is a Lagrangian formulation of the constraint optimization in Equation 2, however replacing $I(Z^{1};X^{1}|X^{2})$ with $I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$.

When MNI is attainable, as justified in Proposition 3, this substitution results in an equivalent objective, allowing $L_{s}^{1}$ to achieve optimal modality-specific representations (proof in Appendix D).

###### Proposition 3 .

If MNI is attainable for random variables $X^{1}$ and $X^{2}$,

$$ $\operatorname*{arg\,max}_{Z^{1}-X^{1}-X^{2}}I(Z^{1};X^{1}|X^{2})=\operatorname *{arg\,max}_{Z^{1}-X^{1}-X^{2}}I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ $$

where $\hat{Z}_{c}^{2*}$ is the representation of $X^{2}$ that satisfies MNI, i.e. $I(\hat{Z}_{c}^{2*};X^{1})=I(\hat{Z}_{c}^{2*};X^{2})=I(X^{1};X^{2})$.

When MNI is unattainable, Proposition 4 shows such substitution yields an almost equivalent objective, subject to the value of $\delta_{c}$ that corresponds to the $\beta$ used in step 1 optimization. Thus, optimizing $L_{s}^{1}$ results in nearly optimal modality-specific representations (proof in Appendix D).

###### Proposition 4 .

For random variables $X^{1}$ and $X^{2}$,

$$ $0\leq I(Z^{1},X^{2};X^{1})-I(Z^{1},\hat{Z}_{c}^{2*};X^{1})\leq\delta_{c}$ $$

where $\hat{Z}_{c}^{2*}$ is the optimal representation of $X^{2}$ with respect to $\delta_{c}$ as defined in Equation 1, i.e. $\hat{Z}_{c}^{2*}=\operatorname*{arg\,min}_{Z^{2}}I(Z^{2};X^{2}|X^{1}),\text{ s
.t. }I(X^{1};X^{2})-I(Z^{2};X^{1})\leq\delta_{c}$.

###### Proposition 3 .

###### Proposition 4 .

### 2.4 Tractable Training Objectives

Four terms are involved in DisentangledSSL, including maximizing the mutual information term $I(Z^{1};X^{2})$ and minimizing the conditional mutual information term $I(Z^{1};X^{1}|X^{2})$ for the shared representations in step 1, as well as maximizing the joint mutual information term $I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ and minimizing the mutual information term $I(Z^{1};\hat{Z}_{c}^{1*})$ for the modality-specific representations in step 2.
In this section, we introduce the tractable training objectives for each term, with detailed formulations described in Appendix G.

For the inferred shared representations, we model the distributions $\hat{Z}^{1}_{c}\sim p(\cdot|X^{1})$ and $\hat{Z}^{2}_{c}\sim p(\cdot|X^{2})$ with neural network encoders. Following the common practice in Radford et al. (), we use the InfoNCE objective (Oord et al., ) as an estimation of the mutual information term $I(Z^{1};X^{2})$ in $L_{c}^{1}$.
We implement the conditional mutual information term $I(Z^{1};X^{1}|X^{2})$ in $L_{c}^{1}$ using an upper bound developed in Federici et al. (), i.e. $I(Z^{1};X^{1}|X^{2})\leq D_{\text{KL}}(p(Z^{1}|X^{1})||p(Z^{2}|X^{2}))$. While the conditional distributions of representations are modeled as the Gaussian distribution in Federici et al. (), we instead use the von Mises-Fisher (vMF) distribution for $p(Z^{1}|X^{1})$ and $p(Z^{2}|X^{2})$ to better align with the InfoNCE objective where the representations lie on the sphere space. Specifically, $\hat{Z}^{1}_{c}\sim\text{vMF}(\mu(X^{1}),\kappa),\hat{Z}^{2}_{c}\sim\text{vMF}
(\mu(X^{2}),\kappa)$ where $\kappa$ is a hyperparameter controlling for the uncertainty of the representations. Leveraging the formulation of the KL divergence between two vMF distributions, the training objective of $L_{c}^{1}$ is to maximize:

$$ $L_{c}^{1}=-L_{\text{InfoNCE}}^{c}+\beta\cdot\mathbb{E}_{x^{1},x^{2}}\left[\mu( x^{1})^{\top}\mu(x^{2})\right]$ $$

The inferred modality-specific representations are encoded as functions
that takes both data observations and the shared representations learned in step 1 as input to account for the dependence structure illustrated in Figure 2. The term $I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ in $L_{s}^{1}$ is optimized with the InfoNCE loss, where random augmentations of the data $X^{1}$ form the two views.
For the mutual information term between representations, $I(Z^{1};\hat{Z}_{c}^{1*})$ in $L_{s}^{1}$, we implement it as an orthogonal loss to encourage the marginal independence between the shared and modality-specific representations, where the marginal distribution is approximated across a training batch.

## 3 Experiments

To evaluate the efficacy of our proposed DisentangledSSL, we conduct a simulation study and two real-world multimodal experiments to address the following key questions:

- •
How do $\beta$ and $\lambda$ affect DisentangledSSL’s learned representations compared to baselines in a controlled simulation setting with known ground truth?
- •
Does DisentangledSSL achieve effective coverage of multimodal information in downstream prediction tasks from MultiBench (Liang et al., ), using a combination of shared and modality-specific representations?
- •
How do DisentangledSSL’s representations perform in high-content drug screening datasets (Subramanian et al., ; Cuccarese et al., ), assessed by molecule-phenotype retrieval (for the shared) and a disentanglement measurement (for modality-specific)?

We compare DisentangledSSL against a disentangled variational autoencoder baseline, DMVAE (Lee & Pavlovic, ), as well as various multimodal contrastive learning methods, including CLIP (Radford et al., ), which aligns different modalities to learn the shared representations, and FOCAL (Liu et al., ) and FactorCL (Liang et al., ), which capture both shared and modality-specific information. Additionally, we evaluate a joint optimization variant of DisentangledSSL, JointOpt (Pan et al., ), to highlight the advantages of our step-by-step approach.

### 3.1 Simulation Study

Synthetic data generation.
We generate synthetic data based on the graphical model in Figure 2. We sample $d$-dimensional true latents $Z_{s}^{1}$, $Z_{s}^{2}$, and $Z_{c}$ independently from $\mathcal{N}(\mu_{d},\Sigma^{2}_{d})$. Using fixed transformations $T_{1}$ and $T_{2}$, we create $X^{1}=T_{1}\cdot[Z_{s}^{1},Z_{c}]$ and $X^{2}=T_{2}\cdot[Z_{s}^{2},Z_{c}]$. To simulate unattainable MNI, we add Gaussian noise to ensure the distribution has full support. Binary labels $Y_{s}^{1}$, $Y_{s}^{2}$, and $Y_{c}$ are constructed from the true latents and used to evaluate the information content of learned representations via linear probing accuracy. Denote $\hat{Z}_{c}$ as the combination of the learned shared representations of $X^{1}$ and $X^{2}$, i.e. $\hat{Z}_{c}=[\hat{Z}_{c}^{1},\hat{Z}_{c}^{2}]$. Ideally, $\hat{Z}_{c}$ should achieve high accuracy on $Y_{c}$ and low on $Y_{s}^{1}$ and $Y_{s}^{2}$, while the modality-specific representations $\hat{Z}_{s}^{1}$ and $\hat{Z}_{s}^{2}$ should show the opposite pattern. Additional details on experimental settings can be found in Appendix H.1.

Results. We assess the performance of the learned shared and modality-specific representations for different values of $\beta$ and $\lambda$, as shown in Figure 4. For comparison, we also evaluate JointOpt, DMVAE, and FOCAL across different hyperparameter settings. Specifically, for JointOpt, we vary both $a$ and $\lambda$, where $a$ controls the joint mutual information terms $I(\hat{Z}_{s}^{1},\hat{Z}_{c}^{2};X^{1})$ and $I(\hat{Z}_{s}^{2},\hat{Z}_{c}^{1};X^{2})$, and $\lambda$ adjusts the mutual information term between representations, similar to DisentangledSSL.

Figure 4(a) illustrates the performance of shared representation $\hat{Z}_{c}$ learned by DisentangledSSL across different values of $\beta$. For lower values of $\beta$, $\hat{Z}_{c}$ captures both shared and specific features, as indicated by linear probing accuracy on $Y_{s}^{1}$, $Y_{s}^{2}$, and $Y_{c}$ exceeding 0.5. As $\beta$ increases, all accuracies decrease, reflecting the trade-off between expressivity and redundancy controlled by $\delta_{c}$ when MNI is unattainable. This trend aligns well with Figure 3 and Proposition 2. A comparison of the performance of the shared representation with baseline methods is provided in Appendix H.1.
We further examine such trade-off on synthetic data with varying levels of multimodal entanglement, where some dimensions are aligned across modalities with MNI attainable and others remain entangled with MNI unattainable. Detailed results are provided in Appendix H.1.

Given the shared representations $\hat{Z}_{c}^{1}$ and $\hat{Z}_{c}^{2}$ learned in step 1 for a fixed $\beta$, we then learn the corresponding modality-specific representations $\hat{Z}_{s}^{1}$ and $\hat{Z}_{s}^{2}$ with varying $\lambda$. Figure 4(b) shows the performance of DisentangledSSL in contrast to other baseline methods, where dots are connected according to a descending order of their corresponding $\lambda$ values(^5^55For FOCAL, we tune the hyperparameters $a$ and $\lambda$, defined similarly to JointOpt, where $a$ controls the terms $I(\hat{Z}_{s}^{1};X^{1})$ and $I(\hat{Z}_{s}^{2};X^{2})$ and $\lambda$ adjusts the orthogonal loss between shared and specific representations. For DMVAE, we tune $\lambda$ which denotes the weight of the KL divergence term. We show the best-performing results of FOCAL and DMVAE across hyperparameters in Figure 4(b), with full results available in Appendix H.1.).
The ideal modality-specific representation $\hat{Z}_{s}^{1}$ should maximize unique information from $X^{1}$, shown by high accuracy on $Y_{s}^{1}$, while minimizing shared information with $X^{2}$, indicated by low accuracy on $Y_{c}$. Therefore, a bottom-right point is preferred in Figure 4(b). As illustrated in Figure 4(b), DisentangledSSL outperforms all other methods across various hyperparameter settings, especially JointOpt, demonstrating the effectiveness of the stepwise optimization procedure. Results on $\hat{Z}_{s}^{2}$ are provided in Appendix H.1.

Figure: (a) Performance of shared representation $\hat{Z}_{c}$ learned with different values of $\beta$.
Refer to caption: x1.png

### 3.2 MultiBench

Dataset description. We utilize the real-world multimodal benchmark from MultiBench (Liang et al., ), which includes curated datasets across various modalities, such as text, images, and tabular data, along with a downstream label that we expect shared and specific information to have varying importance for. These datasets cover a variety of domains including healthcare, affective computing and multimedia research areas. We follow the same setting (dataset splitting, encoder architecture, pre-extracted features) as in Liang et al. (). Additional details can be found in Appendix H.2.

Results. We evaluate the linear probing accuracy of representations learned by each pretrained model, as shown in Table 1. FactorCL-emb refers to the embeddings of FactorCL before projection heads, while FactorCL-proj uses the concatenation of all projection head outputs.
All models use representations (or concatenation of them, if applicable) with the same dimensionality. We present results for the combined shared and specific representations for FOCAL and JointOpt in Table 1, with results for individual representations in Appendix H.2.
DisentangledSSL consistently outperforms baselines across datasets, demonstrating its ability to capture valuable information for downstream tasks. Combining shared and specific representations of DisentangledSSL improves performance in most cases, showing that both contribute to label prediction, while in MOSI and MUSTARD, only shared or specific information contributes significantly.

**Table 1: Prediction accuracy (%) of the representations learned by different methods on MultiBench datasets and standard deviations over 3 random seeds.**
| Dataset | MIMIC | MOSEI | MOSI | UR-FUNNY | MUSTARD |
| --- | --- | --- | --- | --- | --- |
| CLIP | 64.97 (0.60) | 76.87 (0.45) | 64.24 (0.88) | 62.73 (0.92) | 56.04 (4.19) |
| FactorCL-emb | 65.25 (0.45) | 71.80 (0.64) | 62.97 (0.81) | 63.29 (2.07) | 56.76 (4.66) |
| FactorCL-proj | 59.43 (1.70) | 74.61 (1.65) | 56.02 (1.26) | 61.25 (0.47) | 55.80 (2.18) |
| FOCAL | 64.42 (0.34) | 76.77 (0.51) | 63.65 (1.09) | 62.98 (1.52) | 54.35 (0.00) |
| JointOpt | 66.11 (0.64) | 76.71 (0.14) | 64.24 (1.75) | 63.58 (1.45) | 56.52 (2.61) |
| DisentangledSSL (shared) | 63.16 (0.48) | 76.94 (0.22) | 65.16 (0.81) | 64.14 (1.53) | 54.11 (1.51) |
| DisentangledSSL (specific) | 65.73 (0.09) | 75.99 (0.60) | 51.70 (0.72) | 60.27 (1.28) | 61.60 (2.61) |
| DisentangledSSL (both) | 66.44 (0.31) | 77.45 (0.06) | 65.11 (0.80) | 64.24 (1.54) | 56.52 (2.18) |

### 3.3 High-Content Drug Screening

Dataset description. As characterized in Figure 1, we use two high-content drug screening datasets which provide phenotypic profiles after drug perturbation: RXRX19a (Subramanian et al., ) containing cell imaging profiles, and LINCS (Cuccarese et al., ) containing L1000 gene expression profiles. For RXRX19a, we select data from the HRCE cell line under active SARS-CoV-2 condition, totaling 1,661 drugs and 10,117 profiles ($\sim 6$ replicates per drug). Hand-crafted image features from CellProfiler (McQuin et al., ) are used as encoder inputs. For LINCS, we use the data curated in Wang et al. (), selecting drugs with full observations across all 9 core cell lines and randomly sampling 3 replicates per drug to minimize selection bias, resulting in 3,315 drugs and 86,833 profiles. We conduct train-validation-test splitting according to molecules. Models are pretrained to learn representations of molecular structures and their corresponding phenotypes. We provide additional details on experimental settings in Appendix H.3.

**Table 2: Retrieving acccuracy and mean reciprocal rank (MRR) of molecule-phenotype retrieval.**
| Dataset | Top N Acc (%) | N=1 | N=5 | N=10 | N=20 | N=30 | MRR |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Random | 0.30 | 1.50 | 3.00 | 6.01 | 9.01 | - |
|  | CLIP | 3.30(0.40) | 8.33(0.52) | 11.59(0.20) | 16.38(0.56) | 20.22(0.67) | 0.103(0.001) |
|  | DMVAE | 3.85(0.36) | 8.76(0.30) | 11.84(0.32) | 16.20(0.83) | 20.17(0.85) | 0.106(0.002) |
| RXRX19a | JointOpt | 3.41(0.49) | 8.54(0.14) | 11.64(0.14) | 16.79(0.27) | 20.71(1.11) | 0.110(0.002) |
|  | FOCAL | 3.61(0.51) | 8.71(0.69) | 11.94(0.74) | 16.86(0.62) | 20.81(1.07) | 0.108(0.003) |
|  | DisentangledSSL ($\beta=0$) | 3.39(0.54) | 8.25(0.33) | 11.53(0.20) | 16.28(0.10) | 20.20(0.45) | 0.109(0.003) |
|  | DisentangledSSL | 4.03(0.39) | 9.62(0.20) | 13.12(0.23) | 18.36(0.42) | 22.82(0.45) | 0.111(0.001) |
|  | Random | 0.15 | 0.75 | 1.51 | 3.02 | 4.52 | - |
|  | CLIP | 3.95(0.04) | 10.81(0.06) | 15.10(0.20) | 21.08(0.39) | 29.44(0.30) | 0.146(0.001) |
|  | DMVAE | 4.31(0.09) | 11.45(0.13) | 15.88(0.17) | 21.85(0.27) | 29.41(0.33) | 0.156(0.001) |
| LINCS | JointOpt | 4.67(0.09) | 11.60(0.11) | 16.02(0.15) | 22.01(0.21) | 30.08(0.39) | 0.161(0.001) |
|  | FOCAL | 4.34(0.17) | 11.24(0.19) | 15.74(0.26) | 21.48(0.08) | 29.32(0.08) | 0.157(0.002) |
|  | DisentangledSSL ($\beta=0$) | 4.36(0.13) | 11.27(0.39) | 15.81(0.47) | 21.71(0.48) | 29.65(0.54) | 0.158(0.001) |
|  | DisentangledSSL | 4.48(0.21) | 11.70(0.40) | 16.39(0.39) | 22.68(0.58) | 30.84(0.66) | 0.163(0.002) |

Molecule-phenotype retrieval using shared representations.
We evaluate the shared representations in the molecule-phenotype retrieval task, where the goal is to identify molecules from the whole test set that are most likely to induce a specific phenotype. The shared information, which connects the molecular structure and phenotype, plays a key role in this task.
We tune $\beta$ according to validation set performance and show results of top N accuracy (N=1,5,10,20,30) and mean reciprocal rank (MRR) on the test set in Table 2.
DisentangledSSL consistently outperforms baselines on both datasets, effectively capturing relevant shared features while excluding irrelevant modality-specific information. Notably, compared to the variant without the information bottleneck constraint, i.e. DisentangledSSL ($\beta=0$), the full DisentangledSSL model preserves critical shared features, achieving superior performance in this retrieval task, where shared information is essential.

Disentanglement measurement. To assess the effectiveness of learning modality-specific representations, we introduce the Reconstruction Gain (RG) metric, which quantifies the disentanglement and complementariness between shared and modality-specific representations. Specifically, we train 2-layer MLP decoders on the training set to reconstruct the original data from the shared and modality-specific representations, both individually and jointly, and compute the reconstruction $R^{2}$ on the test set for each case. A higher gain in $R^{2}$ when using combined representations in comparison to individual ones indicates lower redundancy and better disentanglement.

**Table 3: Reconstruction gain (RG) in $R^{2}$ of representations for each modality (molecular structure/phenotype).**
| Dataset | RXRX19a | LINCS |  |  |
| --- | --- | --- | --- | --- |
| Metric | RG-molecule | RG-phenotype | RG-molecule | RG-phenotype |
| FOCAL | 0.117(0.006) | 0.547(0.001) | 0.122(0.001) | 0.618(0.002) |
| DMVAE | 0.123(0.002) | 0.545(0.003) | 0.139(0.002) | 0.605(0.003) |
| JointOpt | 0.130(0.001) | 0.524(0.001) | 0.103(0.000) | 0.604(0.002) |
| DisentangledSSL | 0.153(0.003) | 0.591(0.001) | 0.143(0.000) | 0.622(0.002) |

As shown in Table 3, DisentangledSSL achieves the highest RG scores across both modalities and datasets, demonstrating superior disentanglement. In contrast, other methods show redundancy due to insufficient constraints for disentanglement during pretraining.
We also test the representations from the pretrained model on the counterfactual generation task, with detailed results provided in Appendix H.3.

## 4 Related Work

Disentangled representations in VAEs and GANs. Disentangled representation learning originated from works on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), focusing on isolating underlying data variations linked to or independent of labels (e.g., digit identity vs. writing style in MNIST) (Gonzalez-Garcia et al., ; Mathieu et al., ; Bouchacourt et al., ; Daunhawer et al., ). This concept extended to multimodal data like text and image (Lee et al., ) and content and pose in video (Denton et al., ), typically using self- and cross-reconstruction with adversarial loss to learn shared and specific components. However, these methods often address simple cases with attainable MNI, lacking a comprehensive analysis for complex real-world multimodal scenarios where MNI is not attainable.

Information bottleneck and its variants. The information bottleneck (IB) principle has been used to analyze and optimize deep neural networks and the learned representations from the information theory perspective in both supervised and self-supervised settings (Tishby et al., ; Gilad-Bachrach et al., ; Shwartz-Ziv & Tishby, ; Kolchinsky et al., ; Rodríguez Gálvez et al., ; Tsai et al., ; Wang et al., ; Shwartz-Ziv & LeCun, ), with extensions like the conditional entropy bottleneck (CEB) (Fischer, ; Lee et al., ) being developed to better refine the information plane. IB has been applied to learn molecular representations from drug screening data (Liu et al., ). However, these approaches primarily focus on extracting shared features while minimizing specific features, limiting their practical use. Other methods (Pan et al., ; Sanchez et al., ) learn disentangled representations by optimizing mutual information and using adversarial loss but are confined to single-modality scenarios and do not address complex multimodal settings with unattainable MNI.

Multimodal disentanglement in self-supervised learning. Recent studies have explored disentangled representation learning in multimodal self-supervised contexts. Liang et al. () factorizes and optimizes mutual information bounds to capture the union of shared and unique features, while not penalizing for redundancy between latents. Zhang et al. () learns disentangled representations of cell state for multimodal data in biological contexts. Liu et al. () and Li et al. () use mutual information optimization with orthogonal/MMD regularizations for disentangling multimodal time-series signals and T-cell receptor segments, respectively. However, these methods lack a general framework for understanding information content, particularly when data modalities are deeply entangled in real-world applications.

## 5 Conclusions

We present DisentangledSSL, a self-supervised approach for learning disentangled multimodal representations that effectively separates shared and modality-specific features. Built on a comprehensive theoretical framework, DisentangledSSL addresses the challenging scenarios where the Minimum Necessary Information (MNI) is unattainable, enhancing its applications to complex real-world multimodal settings and improving interpretability and robustness. Our empirical results show that DisentangledSSL outperforms baselines across various synthetic and real-world datasets, including vision-language prediction tasks and molecule-phenotype retrieval for high-content drug screens, demonstrating its ability to learn optimal disentangled representations for diverse applications.

## Acknowledgement

Xinyi Zhang, Sana Tonekaboni and Caroline Uhler were partially supported by the Eric and Wendy Schmidt Center at the Broad Institute, NCCIH/NIH (1DP2AT012345), ONR (N00014-22-1-2116) and DOE (DE-SC0023187). SG and SJ acknowledge the support of the NSF Award CCF-2112665 (TILOS AI Institute) and an Alexander von Humboldt fellowship. CW and TJ acknowledge support from the Machine Learning for Pharmaceutical Discovery and Synthesis (MLPDS) consortium, the DTRA Discovery of Medical Countermeasures Against New and Emerging (DOMANE) threats program, and the NSF Expeditions grant (award 1918839) Understanding the World Through Code.

## Appendix A Additional Examples of Unattainable MNI

In this section, we present two additional examples where the Minimum Necessary Information (MNI) is unattainable. These examples offer a clear understanding of how unattainable MNI manifests in data, providing practical insights and making the concept more accessible.

Figure: Figure 5: Example of unattainble MNI.

Firstly, we show an example in a simple mathematical case. As illustrated in Figure 5, we have true latent variables are independently drawn from a Bernoulli distribution, i.e. $Z_{s}^{1},Z_{c},Z_{s}^{2}\sim\text{Bernoulli}(0.5)$, and observations $X^{1}$ and $X^{2}$ are generated according to $X^{1}=\text{OR}(Z_{c},Z_{s}^{1})$ and $X^{2}=\text{OR}(Z_{c},Z_{s}^{2})$. In this scenario, if we observe $X^{1}=1$, we are not able to distinguish whether this result is due to $Z_{c}=1$, or $Z_{s}^{1}=1$, or both. Thus, extracting purely shared features from the observations is infeasible.

Further, we provide an example in 3D geometry, as illustrated in Figure 6. For information such as the height of the cone and the number of spheres, one modality conveys the complete information while the other conveys partial information.
This ambiguity makes it challenging to classify the information as purely shared or modality-specific, placing it in an uncertain category.

Figure: Figure 6: Image modality ($X_{1}$) and text modality ($X_{2}$) of an underlying reality. The Venn diagram illustrates shared and specific information between modalities $X_{1}$ and $X_{2}$: shared content is shown in red, modality-specific content in green, and entangled content due to unattainable MNI in orange. For example, for the cone’s height, the image conveys full information, while the text (i.e. "tall red cone") provides partial information. Similarly, for the number of spheres, the text specifies it completely, whereas the image indicates there is at least one sphere.
Refer to caption: extracted/5969051/arxiv/figs/mainfig1.png

## Appendix B Background on Information Bottleneck

The Information Bottleneck (IB) principle provides a powerful theoretical foundation for our method. IB objective seeks to find a representation $Z^{1}$ of a random variable $X^{1}$ that optimally trades off the preservation of relevant information with compression (Tishby et al., ). Relevance is defined as the mutual information, $I(Z^{1};X^{2})$, between the representation $Z^{1}$ and another target variable $X^{2}$. Compression is enforced by constraining the mutual information between the representation and the original data, $I(X^{1};Z^{1})$, to lie below a specified threshold. This is can be formalize in terms of the following constrained optimization problem:

$$ $\operatorname*{arg\,max}_{Z^{1}}I(Z^{1};X^{2}),\textit{s.t. }I(Z^{1};X^{1})\leq\delta$ (5) $$

where $Z^{1}$ is from a set of random variables that obey the Markov chain $Z^{1}\leftrightarrow X^{1}\leftrightarrow X^{2}$. In practice, Equation 5 can be optimized by minimizing the IB Lagrangian in Equation 6.

$$ $\mathcal{L}=I(Z^{1};X^{1})-\beta I(Z^{1};Z^{2})$ (6) $$

Here, the Lagrangian multiplier $\beta$ controls the emphasis placed on compression versus expressiveness.

Fischer () extends IB to the conditional entropy bottleneck (CEB) objective, which utilizes the conditional mutual information term $I(Z^{1};X^{1}|X^{2})$ in place of the mutual information term $I(Z^{1};X^{1})$ in Equation 6:

$$ $\mathcal{L}=I(Z^{1};X^{1}|X^{2})-\beta I(Z^{1};Z^{2})$ $$

While being equivalent to the IB Lagrangian in terms of the information criterion, CEB rectifies the information plane to precisely measure compression using the conditional mutual information, and offers a clear assessment of how close the compression is to being optimal.

Note that although CEB aligns with our step 1 objective in Equation 3, its optimality has not been fully examined in Fischer (), particularly under scenarios where MNI is unattainable. Furthermore, Fischer () primarily addresses a supervised learning context, where $X^{2}$ serves as the label for $X^{1}$. In contrast, our study tackles the more complex multimodal setting, where $X^{1}$ and $X^{2}$ are two distinct data modalities. We extend the analysis to both attainable and unattainable MNI cases, demonstrating the efficacy of our approach in capturing shared and modality-specific information under these challenging scenarios. This broadens the applicability of CEB beyond traditional label-based supervised settings to multimodal data with complex modality entanglements.

## Appendix C Properties of the IB curve

The information plane is a helpful visualization of the information bottleneck principle, which utilizes $I(X^{1};Z^{1})$ and $I(X^{2};Z^{1})$ as its coordinates that represents the trade-off between compression and prediction (Tishby et al., ). The frontier of all possible $Z^{1}$s is called the IB curve or $F(r)$ (Kolchinsky et al., ) as follows:

$$ $F(r):=\max_{Z^{1}:Z^{1}-X^{1}-X^{2}}I(Z^{1};X^{2})\ \text{s.t.}\ I(X^{2};Z^{1} )\leq r$ $$

The IB curve has the following properties:

- •
The IB curve is concave (Lemma 5 in Gilad-Bachrach et al. ()) and monotonically non-decreasing.
- •
The IB curve is upper bounded by line $I(X^{2};Z^{1})=I(X^{1};Z^{1})$ and line $I(X^{2};Z^{1})=I(X^{1};X^{2})$, according to the Markov relationship $Z^{1}-X^{1}-X^{2}$.
- •
When the MNI point is attainable for random variables $X^{1}$ and $X^{2}$, the IB curve has the following formula (Rodríguez Gálvez et al., ; Kolchinsky et al., ):
$I(Z^{1};X^{2})=\left\{\begin{aligned} I(Z^{1};X^{1}),\ &\quad\text{if}\ I(Z^{1
};X^{1})\leq I(X^{1};X^{2})\\
I(X^{1};X^{2}),\ &\quad\text{if}\ I(X^{1};X^{2})<I(Z^{1};X^{1})\leq H(X^{1})
\end{aligned}\right.$
and the Gateaux derivative of $I(X^{2};Z^{1})$ with respect to $p(z^{1}|x^{1})$ (as used in (Pan et al., )) doesn’t exist at the MNI point (see details in Appendix F).

## Appendix D Proofs

### D.1 Proof of Proposition 1

###### Proposition 1 .

If MNI is attainable for random variable $X^{1}$ and $X^{2}$, maximizing $L_{c}^{1}=I(Z^{1};X^{2})-\beta I(Z^{1};X^{1}|X^{2})$ achieves MNI for any $\beta>0$, i.e. $I(\hat{Z}_{c}^{1*};X^{1})=I(\hat{Z}_{c}^{1*};X^{2})=I(X^{1};X^{2})$, where $\hat{Z}_{c}^{1*}:=\operatorname*{arg\,max}_{Z^{1}-X^{1}-X^{2}}L_{c}^{1}$.

###### Proof.

Based on data processing inequality and the Markov relationship $\hat{Z}_{c}^{1}\leftarrow X^{1}\leftrightarrow X^{2}$, $I(\hat{Z}_{c}^{1};X^{2})\leq I(X^{1};X^{2})$. Based on the non-negativity of conditional mutual information, $I(\hat{Z}_{c}^{1};X^{1}|X^{2})\geq 0$. Thus, for $\beta>0$,

$$ $L_{c}^{1}=I(\hat{Z}_{c}^{1};X^{2})-\beta\cdot I(\hat{Z}_{c}^{1};X^{1}|X^{2}) \leq I(X^{1};X^{2}),\ \forall\ \hat{Z}_{c}^{1}$ $$

where the equality holds when $I(\hat{Z}_{c}^{1};X^{2})=I(X^{1};X^{2})$ and $I(\hat{Z}_{c}^{1};X^{1}|X^{2})=0$

Meanwhile,

$$ $I(\hat{Z}_{c}^{1};X^{1}|X^{2})=I(\hat{Z}_{c}^{1};X^{1},X^{2})-I(\hat{Z}_{c}^{1 };X^{2})=I(\hat{Z}_{c}^{1};X^{1})-I(\hat{Z}_{c}^{1};X^{2})$ $$

where the second equality is due to the conditional independence $\hat{Z}_{c}^{1}\perp\!\!\!\perp X^{2}|X^{1}$ in the Markov relationship.

Therefore, $L_{c}^{1}$ achieves maximality $I(X^{1};X^{2})$ *iff* $I(\hat{Z}_{c}^{1};X^{2})=I(X^{1};X^{2})=I(\hat{Z}_{c}^{1};X^{1})$, i.e. $\hat{Z}_{c}^{1}$ achieves MNI.

∎

###### Proposition 1 .

###### Proof.

### D.2 Proof of Proposition 2

###### Proposition 2 .

For random variables $X^{1}$ and $X^{2}$, when the IB curve $I(Z^{1};X^{2})=F(I(Z^{1};X^{1}))$ is strictly concave,
1) there exists a bijective mapping from $\beta$ in $L_{c}^{1}$ to the value of information constraint $\delta_{c}$ in the definition of optimal shared latent $\hat{Z}_{c}^{1*}$ in Equation 1;
2) $\frac{\partial I(Z_{\beta}^{1*};X^{1})}{\partial\beta}<0,\ \frac{\partial I(Z_
{\beta}^{1*};X^{2})}{\partial\beta}<0$, where $Z_{\beta}^{1*}$ is the optimal solution corresponding to a certain $\beta$.

###### Proof.

Since the IB curve $I(Z^{1};X^{2})=f_{IB}(I(Z^{1};X^{1}))$ is monotonically non-decreasing and strictly concave, it is monotonically increasing. When $Z^{1}=X^{1}$, $f_{IB}$ achieves the maximum point $(H(X^{1}),I(X^{1};X^{2}))$. Thus, $I(X^{1};X^{2})\geq I(X^{1};X^{2})-I(\hat{Z}_{c}^{1*};X^{2})\geq 0$ and is monotonically decreasing. Then there is a bijective mapping between $\delta=I(X^{1};X^{2})-I(\hat{Z}_{c}^{1*};X^{2})\in[0,I(X^{1};X^{2})]$ and points in the IB curve.

Since $I(Z^{1};X^{1}|X^{2})=I(Z^{1};X^{1})-I(Z^{1};X^{2})$, we have

$$ $I(Z^{1};X^{1}|X^{2})=f_{IB}^{-1}(I(Z^{1};X^{2}))-I(Z^{1};X^{2}):=f_{CIB}^{-1}( I(Z^{1};X^{2}))$ $$

where $f_{CIB}(r)=(f_{IB}^{-1}(r)-r)^{-1}$. Since $f_{IB}(r)\leq r$, the equality holds when $r=0$, and $f_{IB}$ is strictly concave, we have $\frac{df_{IB}}{dr}\leq\frac{df_{IB}}{dr}|_{r=0}<1$. Thus $\frac{df_{IB}^{-1}}{dr}>1$ and

$$ $\frac{df_{CIB}}{dr}=(\frac{df_{IB}^{-1}}{dr}-1)^{-1}>0$ $$

Furthermore, since $\frac{df^{2}_{IB}}{dr^{2}}<0$ (strict concavity of $f_{IB}$), we have

$$ $\frac{df^{2}_{CIB}}{dr^{2}}=-\frac{1}{(\frac{df_{IB}^{-1}}{dr}-1)^{2}}\cdot(- \frac{\frac{df^{2}_{IB}}{dr^{2}}}{(\frac{df_{IB}}{dr})^{2}})<0$ $$

Therefore, $f_{CIB}$ is also monotonically increasing and strictly concave. Then there is a bijective mapping between $\delta=I(X^{1};X^{2})-I(\hat{Z}_{c}^{1*};X^{2})\in[0,I(X^{1};X^{2})]$ and points in the CIB curve $f_{CIB}$.

Since $f_{CIB}$ is increasing, the inequality constraint can be replaced by the equality constraint, i.e.

$$ $\hat{Z}_{c}^{1*}=\operatorname*{arg\,min}_{I(Z^{1};X^{2})\geq I(X^{1};X^{2})- \delta}I(Z^{1};X^{1}|X^{2})=\operatorname*{arg\,min}_{I(Z^{1};X^{2})=I(X^{1};X ^{2})-\delta}I(Z^{1};X^{1}|X^{2})$ $$

The corresponding Lagrangian function is

$$ $\displaystyle L$ $\displaystyle=I(Z^{1};X^{1}|X^{2})-\tilde{\beta}\cdot(I(Z^{1};X^{2})-I(X^{1};X ^{2})+\delta)$ $\displaystyle=f^{-1}_{CIB}(I(Z^{1};X^{2}))-\tilde{\beta}\cdot I(Z^{1};X^{2})+ \tilde{\beta}\cdot(I(X^{1};X^{2})-\delta)$ $$

Based on the Lagrangian multiplier theorem, the optimal $\hat{Z}_{c}^{1*}$ is achieved when

$$ $\frac{dL}{dI(Z^{1};X^{2})}=\frac{df^{-1}_{CIB}(I(Z^{1};X^{2}))}{dI(Z^{1};X^{2} )}-\tilde{\beta}=0$ $$

$$ $\frac{dL}{d\tilde{\beta}}=-I(Z^{1};X^{2})+I(X^{1};X^{2})-\delta=0$ $$

Thus,

$$ $\tilde{\beta}=\frac{df^{-1}_{CIB}(I(Z^{1};X^{2}))}{dI(Z^{1};X^{2})}|_{I(Z^{1}; X^{2})=I(X^{1};X^{2})-\delta}$ (7) $$

i.e. $\tilde{\beta}$ is the slope of $f^{-1}_{CIB}$ when $I(Z^{1};X^{2})=I(X^{1};X^{2})-\delta$. Therefore, there is a bijective mapping between $\beta=\tilde{\beta}^{-1}$ in $L_{c}^{1}$ and points in $f_{CIB}$, and thus $\delta$.

For any $\beta_{1},\beta_{2}>0$ and $\beta_{1}>\beta_{2}$, we have $\tilde{\beta}_{1}=\beta_{1}^{-1}<\beta_{2}^{-1}=\tilde{\beta}_{2}$. According to formula 7,

$$ $\tilde{\beta}_{1}=\frac{df^{-1}_{CIB}(I(Z^{1};X^{2}))}{dI(Z^{1};X^{2})}|_{I(Z^ {1};X^{2})=I(Z_{\beta_{1}}^{*};X^{2})},\ \tilde{\beta}_{2}=\frac{df_{CIB}^{-1} (I(Z^{1};X^{2}))}{dI(Z^{1};X^{2})}|_{I(Z^{1};X^{2})=I(Z_{\beta_{2}}^{*};X^{2})}$ $$

Since $\tilde{\beta}_{1}<\tilde{\beta}_{2}$ and $f_{CIB}$ is a strictly concave function, $f^{-1}_{CIB}$ is strictly convex and we have $I(Z_{\beta_{1}}^{*};X^{2})<I(Z_{\beta_{2}}^{*};X^{2})$. Furthermore, since $f_{IB}$ is monotonically increasing, we have $I(Z_{\beta_{1}}^{*};X^{1})<I(Z_{\beta_{2}}^{*};X^{1})$. Therefore,

$$ $\frac{\partial I(Z_{\beta}^{*};X^{2})}{\partial\beta}<0,\ \frac{\partial I(Z_{ \beta}^{*};X^{1})}{\partial\beta}<0$ $$

∎

###### Proposition 2 .

###### Proof.

### D.3 Proof of Proposition 3

###### Proposition 3 .

If MNI is attainable for random variables $X^{1}$ and $X^{2}$,

$$ $\operatorname*{arg\,max}_{Z^{1}-X^{1}-X^{2}}I(Z^{1};X^{1}|X^{2})=\operatorname *{arg\,max}_{Z^{1}-X^{1}-X^{2}}I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ $$

where $\hat{Z}_{c}^{2*}$ is the representation based on $X^{2}$ that satisfies MNI, i.e. $I(\hat{Z}_{c}^{2*};X^{1})=I(\hat{Z}_{c}^{2*};X^{2})=I(X^{1};X^{2})$.

###### Proof.

Based on the graphical model, $\hat{Z}_{c}^{2}\perp\!\!\!\perp X^{1}|X^{2},Z^{1}$, we have $I(\hat{Z}_{c}^{2};X^{1}|Z^{1},X^{2})=0$.
Thus,

$$ $\displaystyle I(Z^{1},X^{2};X^{1})$ $\displaystyle=I(Z^{1},X^{2};X^{1})+I(\hat{Z}_{c}^{2};X^{1}|Z^{1},X^{2})$ $\displaystyle=I(Z^{1},X^{2},\hat{Z}_{c}^{2};X^{1})=I(Z^{1},\hat{Z}_{c}^{2};X^{ 1})+I(X^{1};X^{2}|Z^{1},\hat{Z}_{c}^{2})$ (8) $$

Meanwhile,

$$ $I(X^{1};X^{2}|\hat{Z}_{c}^{2})-I(X^{1};X^{2}|Z^{1},\hat{Z}_{c}^{2})=I(X^{2};Z^ {1}|\hat{Z}_{c}^{2})-I(X^{2};Z^{1}|X^{1},\hat{Z}_{c}^{2})=I(X^{2};Z^{1}|\hat{Z }_{c}^{2})$ $$

where the second equality is due to the conditional independence $X^{2}\perp\!\!\!\perp Z^{1}|X^{1},\hat{Z}_{c}^{2}$ in the graphical model.

When $\hat{Z}_{c}^{2}$ is the MNI point $\hat{Z}_{c}^{2*}$, $I(X^{1};\hat{Z}_{c}^{2*})=I(X^{1};X^{2})$. Thus,

$$ $I(X^{1};X^{2}|\hat{Z}_{c}^{2*})=I(X^{1};X^{2},\hat{Z}_{c}^{2*})-I(X^{1};\hat{Z }_{c}^{2*})=I(X^{1};X^{2})-I(X^{1};\hat{Z}_{c}^{2*})=0$ $$

where the first equality is due to the Markov relationship $X^{1}\leftrightarrow X^{2}\rightarrow\hat{Z}_{c}^{2*}$.

Then we have $-I(X^{1};X^{2}|Z^{1},\hat{Z}_{c}^{2*})=I(X^{2};Z^{1}|\hat{Z}_{c}^{2*})$. Due to the non-negativity of conditional mutual information,

$$ $I(X^{1};X^{2}|Z^{1},\hat{Z}_{c}^{2*})=I(X^{2};Z^{1}|\hat{Z}_{c}^{2*})=0$ (9) $$

Based on formula D.3 and 9,

$$ $I(Z^{1},X^{2};X^{1})=I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ $$

Since $I(Z^{1};X^{1}|X^{2})=I(Z^{1},X^{2};X^{1})-I(X^{1};X^{2})$ and $I(X^{1};X^{2})$ is a constant value irrelevant to $Z^{1}$, maximizing $I(Z^{1};X^{1}|X^{2})$ is equivalent to maximizing $I(Z^{1},X^{2};X^{1})=I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$.

∎

###### Proposition 3 .

###### Proof.

### D.4 Proof of Proposition 4

###### Proposition 4 .

For random variables $X^{1}$ and $X^{2}$,

$$ $0\leq I(Z^{1},X^{2};X^{1})-I(Z^{1},\hat{Z}_{c}^{2*};X^{1})\leq\delta_{c}$ $$

where $\hat{Z}_{c}^{2*}$ is the optimal representation based on $X^{2}$ with respect to $\delta_{c}$ as defined in Equation 1, i.e. $\hat{Z}_{c}^{2*}=\operatorname*{arg\,min}_{Z^{2}}I(Z^{2};X^{2}|X^{1}),\text{ s
.t. }I(X^{1};X^{2})-I(Z^{2};X^{1})\leq\delta_{c}$.

###### Proof.

According to formula D.3 and D.3,

$$ $I(Z^{1},X^{2};X^{1})=I(Z^{1},\hat{Z}_{c}^{2};X^{1})+I(X^{1};X^{2}|Z^{1},\hat{Z }_{c}^{2})$ $$

$$ $I(X^{1};X^{2}|Z^{1},\hat{Z}_{c}^{2})=I(X^{1};X^{2}|\hat{Z}_{c}^{2})-I(Z^{1};X^ {2}|\hat{Z}_{c}^{2})$ $$

Since $\hat{Z}_{c}^{2*}$ is the optimal latent under $\delta$, $I(X^{1};X^{2})-I(\hat{Z}_{c}^{2*};X^{1})\leq\delta$. Thus,

$$ $I(X^{1};X^{2}|\hat{Z}_{c}^{2*})=I(X^{1};X^{2})-I(X^{1};\hat{Z}_{c}^{2*})\leq\delta$ $$

Meanwhile, $I(Z^{1};X^{2}|\hat{Z}_{c}^{2})\geq 0$. Then we have $I(X^{1};X^{2}|Z^{1},\hat{Z}_{c}^{2})\leq\delta-I(Z^{1};X^{2}|\hat{Z}_{c}^{2})\leq\delta$.
Therefore,

$$ $0\leq I(Z^{1},X^{2};X^{1})-I(Z^{1},\hat{Z}_{c}^{2};X^{1})=I(X^{1};X^{2}|Z^{1}, \hat{Z}_{c}^{2})\leq\delta$ $$

i.e. $|I(Z^{1},X^{2};X^{1})-I(Z^{1},\hat{Z}_{c}^{2};X^{1})|\leq\delta$

Since $I(Z^{1};X^{1}|X^{2})=I(Z^{1},X^{2};X^{1})-I(X^{1};X^{2})$ and $I(X^{1};X^{2})$ is a constant value irrelevant to $Z^{1}$, maximizing $I(Z^{1};X^{1}|X^{2})$ is equivalent to maximizing $I(Z^{1},X^{2};X^{1})\in[I(Z^{1},\hat{Z}_{c}^{2*};X^{1}),I(Z^{1},\hat{Z}_{c}^{2
*};X^{1})+\delta]$.

∎

###### Proposition 4 .

###### Proof.

## Appendix E Sufficient Conditions for MNI

In this section, we explore the conditions for both MNI attainable and unattainable cases. Here, we use $X$ and $Y$ to represent the two modalities, rather than $X^{1}$ and $X^{2}$ in other sections. Although prior works have not provided a comprehensive necessary and sufficient condition for the attainability of MNI, and such analysis is beyond the scope of this paper, we provide several sufficient conditions for both MNI attainable and unattainable scenarios.

We first present the sufficient conditions under which MNI is attainable. As outlined in Proposition 5, Proposition 6, and Proposition 7, MNI is attainable when the relationships between $X$ and $Y$ are either entirely deterministic or completely independent across each sub-domain.
While the deterministic mapping might hold true when $Y$ is the data label, it generally does not hold when $X$ and $Y$ are high-dimensional data modalities. In such cases, a deterministic relationship between the two modalities would imply that one can be fully inferred from the other, leaving no room for modality-specific information in one of the modalities.

###### Proposition 5 .

For random variables $X$, $Y$, and representation $Z$ derived from $X$ (i.e., the Markov chain $Z\leftarrow X\leftrightarrow Y$ holds), a sufficient condition for MNI to be attainable is that $X\rightarrow Y$ mapping is deterministic (Fischer, ).

###### Proof.

Denote $Y=f(X)$ as the deterministic $X\rightarrow Y$ mapping. If the encoder is powerful enough, it can learn to reproduce the deterministic function $f$, i.e. $Z=f(X)=Y$. Thus $I(X;Z)=I(X;Y)=H(Y)=I(Y;Z)$.
∎

While Fischer () identifies this as a necessary condition for attainable MNI, there are scenarios where $X\rightarrow Y$ is not deterministic, yet MNI is still attainable. We provide additional sufficient conditions for these more general cases in Proposition 6 and Proposition 7.

###### Proposition 6 .

For random variables $X$, $Y$, and representation $Z$ derived from $X$ (i.e., the Markov chain $Z\leftarrow X\leftrightarrow Y$ holds), a sufficient condition for MNI to be attainable is that $Y\rightarrow X$ mapping is deterministic.

###### Proof.

Denote $X=f(Y)$ as the deterministic $Y\rightarrow X$ mapping. Then, for any $Z$ with $Z\leftarrow X\leftrightarrow Y$,

$$ $p(z|y)=\int_{x}p(z|x)p(x|y)dx=\int_{x}p(z|x)\mathbf{1}_{x=f(y)}dx=p(z|x=f(y))$ $$

More formally, $p(z|y)=p(z|X=f(y))$, and

$$ $p(Y=y,Z=z)=p(Y=y,X=f(y),Z=z)=p(Y=y)\cdot p(z|X=f(y))$ $$

Thus the conditional entropy can be written as

$$ $H(Z|Y)=-\int_{Y,Z}p(y,z)\log p(z|y)dydz=-\int_{Y}p(y)\int_{Z}p(z|X=f(y))\log p (z|X=f(y))dydz$ $$

$$ $H(Z|X)=-\int_{X}p(x)\int_{Z}p(z|x)\log p(z|x)dxdz$ $$

Thus, $H(Z|Y)=H(Z|X)$, i.e. $I(Y;Z)=I(X;Z)$. By selecting $Z$ such that $I(Z;X)=I(X;Y)=H(X)$, we achieve the MNI point.
∎

###### Proposition 7 .

For random variable $X,Y$ in $\mathcal{X}\times\mathcal{Y}$ with joint distribution $p(X,Y)$, a sufficient condition for the existence of MNI is:

In any sub-domain $\mathcal{X}_{s}\times\mathcal{Y}_{s}$ of $\mathcal{X}\times\mathcal{Y}$ where $p(X,Y)$ has the full support, i.e. $\forall\ x,y\in\mathcal{X}_{s}\times\mathcal{Y}_{s},p(x,y)>0$, $X$ and $Y$ are independent, i.e. $X\perp\!\!\!\perp Y|\{(X,Y)\in\mathcal{X}_{s}\times\mathcal{Y}_{s}\}$.

###### Proof.

Construct the learned representation $Z$ as the conditional probability of $Y$ given $X$, i.e. $Z=p(Y|X)$. $Z$ is a deterministic function based o $X$. We will show that such $Z$ satisfies the MNI condition.

First, we show that $I(Z;Y)=I(X;Y)$. Denote $z=p(Y|X=x)$. Since $z$ fully describes the conditional distribution $p(Y|X=x)$, we have $p(Y|X=x)=p(Y|Z=z)$. Thus, $H(Y|X=x)=H(Y|Z=z)$. Therefore,

$$ $\displaystyle H(Y|X)$ $\displaystyle=\int p(x)H(Y|X=x)dx=\int p(x)H(Y|Z=z)dx$ $\displaystyle=\int\left(\int_{\{x:p(Y|X=x)=z\}}p(x)dx\right)H(Y|Z=z)dz=\int p( z)H(Y|Z=z)dz=H(Y|Z)$ $$

Thus $I(Z;Y)=H(Y)-H(Y|Z)=H(Y)-H(Y|X)=I(X;Y)$.

Second, we show that $I(Z;X)=I(Z;Y)$. Since $I(Z;X)=H(Z)-H(Z|X)$, $I(Z;Y)=H(Z)-H(Z|Y)$, and $H(Z|X)=0$, $I(Z;X)=I(Z;Y)$ is equivalent to $H(Z|Y)=0$, i.e. $Z=p(Y|X)$ is determined by the value of $Y$.

The condition is $\forall\ \mathcal{X}_{s}\times\mathcal{Y}_{s},X\perp\!\!\!\perp Y|\{(X,Y)\in
\mathcal{X}_{s}\times\mathcal{Y}_{s}\}$, which indicates that

$$ $\forall\ \mathcal{X}_{s}\times\mathcal{Y}_{s},\forall\ x,y\in\mathcal{X}_{s} \times\mathcal{Y}_{s},\frac{p(y|x)}{p(\mathcal{Y}_{s}|\mathcal{X}_{s})}=\frac{ p(y)}{p(\mathcal{Y}_{s})}$ $$

For any $y\in\mathcal{Y}$, choose $\mathcal{Y}_{s}=\{y\}$, and $\mathcal{X}_{s}=\{x:p(x|y)>0\}$, we have $p(y|x)=\frac{p(\mathcal{Y}_{s}|\mathcal{X}_{s})}{p(\mathcal{Y}_{s})}p(y),
\forall\ x\in\mathcal{X}_{s}$. Thus, $\forall\ x_{1},x_{2}$ sampled from $p(X|Y=y)$, we have $p(y|x1)=p(y|x2)$. Therefore, $p(Y=y|X)$ is fully determined given the value of $y$ and is irrelevant to the value of $X$. Then we show that $I(Z;X)=I(Z;Y)$.

It is easy to see that deterministic mapping between $X$ and $Y$ is a special case of this condition.

∎

Regarding the sufficient conditions for MNI being unattainable, as outlined in Lemma 6 of Gilad-Bachrach et al. (), when variables $X,Y$ have full support, i.e. $p(x,y)>0,\ \forall\ x,y$, MNI is unattainable.

###### Proposition 5 .

###### Proof.

###### Proposition 6 .

###### Proof.

###### Proposition 7 .

###### Proof.

## Appendix F Differentiability of the IB Curve

###### Definition 1 .

Let $V$ and $W$ be Banach spaces, $\Omega$ an open set in $V$ and $F$ a function that maps $\Omega$ into $W$. Then the Gateaux derivative of $F$ at $x\in\Omega$ in the direction $h\in V$ is defined as

$$ $dF(x;h)=\left.\lim_{\epsilon\rightarrow 0}\frac{F(x+\epsilon h)-F(x)}{\epsilon }=\frac{d}{d\epsilon}F(x+\epsilon h)\right|_{\epsilon=0}$ $$

provided that this limit exists for all $h\in V$

In this section, we use $X$ and $Y$ to represent the two modalities, rather than $X^{1}$ and $X^{2}$ in other sections.
In our setting, the domain is the Banach space corresponding to the product space representing the set of pairs of joint probability distributions i.e. $\mathcal{Z}\times\mathcal{Y}$. The functional $F=I(Z;Y):\Omega\rightarrow\mathbb{R}$, where $\Omega$ is the product space representing the set of pairs of joint probability distributions for which this mutual information is defined. Following the Markov structure in our graphical model, we have

$$ $\displaystyle I(Z;Y)$ $\displaystyle=\int_{y,z}p(z,y)\log\frac{p(z,y)}{p(z)p(y)}$ $\displaystyle=\int_{x,y,z}p(z|x,y)p(x|y)\log\frac{\int_{x}p(z|x,y)p(x|y)}{\int _{x}p(z|x)p(x)}$ $\displaystyle=\int_{x,y,z}p(z|x)p(x|y)\log\frac{\int_{x}p(z|x)p(x|y)}{\int_{x} p(z|x)p(x)}$ $\displaystyle=F(f)\text{where $f=p(z|x)$}$ $$

###### Proposition 8 .

If the MNI point exists, then the Gateaux derivative of $I(Y;Z)$ with respect to $p(z|x)$ (as used in (Pan et al., )) doesn’t exist at the MNI point.

###### Proof.

Denote the MNI point between $X$ and $Y$ as $p$ where $I(X;Y)=I(Z;Y)=I(Z;X)$. When the MNI point exists, the IB curve can be represented as (Rodríguez Gálvez et al., ; Kolchinsky et al., ):

$$ $I(Z;Y)=\left\{\begin{aligned} I(Z;X),\ &\text{if}\ I(Z;X)\leq H(Y)\\ H(Y),\ &\text{if}\ H(Y)<I(Z;X)\leq H(X)\end{aligned}\right.$ $$

or equivalently

$$ $p(z|y)=\left\{\begin{aligned} p(z|x),\ &\text{if}\ I(Z;X)\leq H(Y)\\ \mathbf{1}_{y=g(z)},\ &\text{if}\ H(Y)<I(Z;X)\leq H(X)\end{aligned}\right.$ $$

Based on this characterization, we know that there exist a direction $h$ such that on perturbing $p(z|x)$ along this directions, we either have $I(Z;Y)=I(Z;X)<H(Y)$ or $I(Z;Y)=I(X;Y)=H(Y)<I(Z;X)$. Without loss of generality, assume that perturbing by $\epsilon h$, where $\epsilon<0$ corresponds to the former case and $\epsilon h$, where $\epsilon>0$ to the latter. Consider the right directional derivative of I(Y:Z) at $p$ in the direction $h$, we have

$$ $\displaystyle d_{+}F(p;h)$ $\displaystyle=\lim_{\epsilon\rightarrow 0^{+}}\frac{F(p+\epsilon h)-F(p)}{ \epsilon}=0$ $$

Since we have $p(z|y)=p(z|x)$ when $p$ is perturbed in the direction by $\epsilon h$, where $\epsilon<0$, we can consider $I(Z;Y)$ as an identity function of $I(Z;X)$. In such a case, $F(p+\epsilon h_{1})=H(Y)+\epsilon h$. As a consequence, perturbing $p(z|x)$ to get $p^{\prime}$ is equivalent to the resulting perturbation in $p(z|y)$ also being $p^{\prime}$. As a result, we have

$$ $\displaystyle d_{-}F(p;h)$ $\displaystyle=\lim_{\epsilon\rightarrow 0^{-}}\frac{F(p^{\prime})-F(p)}{\epsilon}$ $\displaystyle=\lim_{\epsilon\rightarrow 0^{-}}\frac{F(p(z|y)=p^{\prime})-F(p)} {\epsilon}$ $\displaystyle=\left.\frac{d}{d\epsilon}F(p+\epsilon h)\right|_{\epsilon=0}$ $\displaystyle=\left.\frac{d}{d\epsilon}\int_{z,y}(p(z|y)+\epsilon h)p(y)\log \frac{p(z|y)+\epsilon h}{\int_{y}p(z|y)p(y)}\right|_{\epsilon=0}$ $\displaystyle=\left.\int_{z,y}hp(y)\log\frac{p(z|y)+\epsilon h}{\int_{y}p(z|y) p(y)}+(p(z|y)+\epsilon h)p(y)\Big{(}\frac{h}{p(z|y)+\epsilon h}-\frac{1}{p(z)} \Big{)}\right|_{\epsilon=0}$ $\displaystyle=h\neq 0$ $$

Clearly, the left and right limits exist but aren’t equal, hence proving the non-differentiability.

∎

###### Definition 1 .

###### Proposition 8 .

###### Proof.

## Appendix G Tractable Training Objectives

Four terms are involved in DisentangledSSL, including maximizing the mutual information term $I(Z^{1};X^{2})$ and minimizing the conditional mutual information term $I(Z^{1};X^{1}|X^{2})$ for the shared representations in step 1, as well as maximizing the joint mutual information term $I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ and minimizing the mutual information term $I(Z^{1};\hat{Z}_{c}^{1*})$ for the modality-specific representations in step 2.
We introduce detailed formulations of the tractable training objectives for each of the four terms in this section.

For the inferred shared representations, we model the distributions $\hat{Z}^{1}_{c}\sim p(\cdot|X^{1})$ and $\hat{Z}^{2}_{c}\sim p(\cdot|X^{2})$ with neural network encoders. Following the common practice in Radford et al. (), we use the InfoNCE objective (Oord et al., ) as an estimation of the mutual information term $I(Z^{1};X^{2})$ in $L_{c}^{1}$.

$$ $L_{\text{InfoNCE}}^{c}=\mathbb{E}_{z^{1},z^{2+},\{z_{i}^{2-}\}_{i=1}^{N}}\left [-\log\frac{\exp({z^{1}}^{\top}z^{2+}/\tau)}{\exp({z^{1}}^{\top}z^{2+}/\tau)+ \sum_{i=1}^{N}\exp({z^{1}}^{\top}z^{2-}_{i}/\tau)}\right]$ $$

where $\tau$ is the temperature hyperparameter, $z^{2+}$ is the representation of the positive sample corresponding to the joint distribution $p(X^{1},X^{2})$, and $\{{z_{i}^{2-}\}_{i=1}^{N}}$ are representations of $N$ negative samples from the marginal distribution $p(X^{2})$.

We implement the conditional mutual information term $I(Z^{1};X^{1}|X^{2})$ in $L_{c}^{1}$ using an upper bound developed in Federici et al. (), i.e. $I(Z^{1};X^{1}|X^{2})\leq D_{\text{KL}}(p(Z^{1}|X^{1})||p(Z^{2}|X^{2}))$. While the conditional distributions of representations are modeled as the Gaussian distribution in Federici et al. (), we instead use the von Mises-Fisher (vMF) distribution for $p(Z^{1}|X^{1})$ and $p(Z^{2}|X^{2})$ to better align with the InfoNCE objective where the representations lie on the sphere space. Specifically, $\hat{Z}^{1}_{c}\sim\text{vMF}(\mu(X^{1}),\kappa),\hat{Z}^{2}_{c}\sim\text{vMF}
(\mu(X^{2}),\kappa)$ where $\kappa$ is a hyperparameter controlling for the uncertainty of the representations. Leveraging the formulation of the KL divergence between two vMF distributions, the training objective of $L_{c}^{1}$ is to maximize:

$$ $L_{c}^{1}=-L_{\text{InfoNCE}}^{c}+\beta\cdot\mathbb{E}_{x^{1},x^{2}}\left[\mu( x^{1})^{\top}\mu(x^{2})\right]$ $$

Note that this objective establish connections with the alignment versus uniformity framework discussed in Wang & Isola (), where the conditional information bottleneck constraint corresponds to a higher weight on the alignment term, in contrast to the uniformity term.

The inferred modality-specific representations are encoded as functions
$\hat{Z}^{1}_{s}\sim p(\cdot|X^{1},\hat{Z}^{1}_{c})$ and $\hat{Z}^{2}_{s}\sim p(\cdot|X^{2},\hat{Z}^{2}_{c})$ with deterministic encoders,
that takes both data observations and the shared representations learned in step 1 as input to account for the dependence structure illustrated in Figure 2. The term $I(Z^{1},\hat{Z}_{c}^{2*};X^{1})$ in $L_{s}^{1}$ is optimized with the InfoNCE loss, where random augmentations of the data $X^{1}$ form the two views. Denote the concatenation of $Z^{1}$ and its corresponding $\hat{Z}_{c}^{2*}$ as $\tilde{Z}^{1}$, the InfoNCE objective has the following formula:

$$ $L_{\text{InfoNCE}}^{s}=\mathbb{E}_{\tilde{z}^{1},\tilde{z}^{1+},\{\tilde{z}_{i }^{1-}\}_{i=1}^{N}}\left[-\log\frac{\exp(\tilde{z}^{1\top}\tilde{z}^{1+}/\tau) }{\exp(\tilde{z}^{1\top}\tilde{z}^{1+}/\tau)+\sum_{i=1}^{N}\exp(\tilde{z}^{1 \top}\tilde{z}^{1-}_{i}/\tau)}\right]$ (10) $$

where $\tilde{z}^{1+}$ is the representation of the positive sample corresponding to the augmented view of $X^{1}$, and $\{{z_{i}^{1-}\}_{i=1}^{N}}$ are representations of $N$ negative samples from the marginal distribution $p(X^{1})$.

For the mutual information term between representations, $I(Z^{1};\hat{Z}_{c}^{1*})$ in $L_{s}^{1}$, we implement it as an orthogonal loss to encourage the marginal independence between the shared and modality-specific representations, where the marginal distribution is approximated across a training batch.

$$ $L_{\text{orthogonal}}=||{[Z^{1}_{i}]_{i=1}^{B}}^{\top}\cdot[\hat{Z}^{1*}_{ci}] _{i=1}^{B}||_{F}$ (11) $$

where $B$ is the batch size, $[Z^{1}_{i}]_{i=1}^{B}$ and $[\hat{Z}^{1*}_{ci}]_{i=1}^{B}$ are the concatenations of all the representations in a mini-batch, and $||\cdot||_{F}$ is the Frobenius norm of the pairwise cosine similarities between each latent dimensions.
The training objective of $L_{s}^{1}$ in step 2 is to maximize:

$$ $L_{s}^{1}=-L_{\text{InfoNCE}}^{s}-\lambda\cdot L_{\text{orthogonal}}$ (12) $$

## Appendix H Experimental Details and Additional Results

Each experiment was conducted on 1 NVIDIA RTX A5000 GPU, each with 24GB of accelerator RAM. All experiments were implemented using the PyTorch deep learning framework.

### H.1 Simulation Study.

Experimental details. We generate synthetic data $X^{1}$ and $X^{2}$ based on the graphical model in Figure 2, with the dimensionality of 100 and dataset size being 90,000. To be specific, we sample 50-dimensional true latents $Z_{s}^{1}$, $Z_{s}^{2}$, and $Z_{c}$ independently from $\mathcal{N}(\mathbf{0}_{50},0.5\times\mathbf{I}_{50})$. Then we sample the transformation weights $T_{1}$ and $T_{2}$ from uniform distribution $\text{Uniform}(-1,1)$, and generate $X^{1}=T_{1}\cdot[Z_{s}^{1},Z_{c}]$ and $X^{2}=T_{2}\cdot[Z_{s}^{2},Z_{c}]$. We randomly split $80\%$ data into the training set and the rest into the test set. To simulate unattainable MNI, we add Gaussian noise and random dropout during training to ensure the distribution has full support. We use a 3-layer multi-layer perceptron (MLP) with a hidden dimension of 512 as encoders for all methods. For DMVAE, we employ MLPs with the same architecture as decoders. We report the average linear probing accuracy on the test set over 3 random seeds.

We run all the methods on the synthetic data with combinations of different hyperparameter values. For DisentangledSSL, we use $\beta\in\{0.0,0.001,0.01,0.1,0.5,1.0,5.0,10.0,50.0,100.0,300.0,\\
500.0,1000.0\}$ and $\lambda\in\{0.0,0.001,0.01,0.1,1.0,10.0,100.0\}$.

For JointOpt, hyperparameter $a$ controls the joint mutual information terms $I(\hat{Z}_{s}^{1},\hat{Z}_{c}^{2};X^{1})$ and $I(\hat{Z}_{s}^{2},\hat{Z}_{c}^{1};X^{2})$, and $\lambda$ adjusts the mutual information term between representations, i.e.

$$ $\displaystyle\hat{Z}_{c}^{1*},\hat{Z}_{c}^{2*},\hat{Z}_{s}^{1*},\hat{Z}_{s}^{2*}$ $\displaystyle=\operatorname*{arg\,max}_{\hat{Z}_{c}^{1},\hat{Z}_{c}^{2},\hat{Z }_{s}^{1},\hat{Z}_{s}^{2}}I(\hat{Z}_{c}^{1};X^{2})+I(\hat{Z}_{c}^{2};X^{1})+a \cdot(I(\hat{Z}_{s}^{1},\hat{Z}_{c}^{2};X^{1})+I(\hat{Z}_{s}^{2},\hat{Z}_{c}^{ 1};X^{2}))$ $\displaystyle-\lambda\cdot(I(Z_{c}^{1};Z_{s}^{1})+I(Z_{c}^{2};Z_{s}^{2}))$ $$

We use $a\in\{0.01,0.1,1.0,10.0,100.0,1000.0\}$ and $\lambda\in\{0.0,0.001,0.01,0.1,1.0,10.0,100.0\}$.

For FOCAL, we tune the hyperparameters $a$ and $\lambda$, defined similarly to JointOpt, where $a$ controls the terms $I(\hat{Z}_{s}^{1};X^{1})$ and $I(\hat{Z}_{s}^{2};X^{2})$ and $\lambda$ adjusts the orthogonal loss between shared and specific representations. We use the same set of $a$ and $\lambda$ as that in JointOpt.

For DMVAE, we tune $\lambda$ which denotes the weight of the KL divergence term in contrast to the reconstruction loss. We use $\lambda\in\{10^{-7},10^{-6},10^{-5},10^{-4},10^{-3},10^{-2}\}$.

Additional results. We provide a comparison of the performance of the shared representation with baseline methods in Figure 7. Denote $\hat{Z}_{c}$ as the concatenation of the learned shared representations of $X^{1}$ and $X^{2}$, i.e. $\hat{Z}_{c}=[\hat{Z}_{c}^{1},\hat{Z}_{c}^{2}]$.
The ideal $\hat{Z}_{c}$ should maximize the shared information between $X^{1}$ and $X^{2}$, shown by high accuracy on $Y_{c}$, while minimizing unique information of $X^{1}$ and $X^{2}$, indicated by low accuracy on $Y_{s}^{1}$ and $Y_{s}^{2}$. Therefore, a top-left point is preferred in Figure 7.

As shown in Figure 7, DisentangledSSL consistently outperforms other methods across various hyperparameter settings, showcasing its ability to effectively capture shared information. The only exception occurs in Figure 7(a) when $\beta$ is set very high and the accuracy on $Y_{s}^{1}$ to drop to around 0.50 (equivalent to random guessing, indicating no information of $\hat{Z}_{c}$ on modality-specific features). In this scenario, DMVAE performs better. This happens because high values of $\beta$ cause the decoder-free contrastive objectives to collapse, with most representations converging to nearly the same point, a commonly-known issue in contrastive self-supervised learning (Oord et al., ).

Figure: (a) Test accuracy of $\hat{Z}_{c}$ on $Y_{c}$ versus $Y_{s}^{1}$.
Refer to caption: x3.png

For the modality-specific representations, as a supplement to Figure 4(b), we provide the complete results for both representations $\hat{Z}_{s}^{1}$ and $\hat{Z}_{s}^{2}$ on a full set of hyperparameters in Figure 8. DisentangledSSL outperforms all other methods across various hyperparameter settings.

Figure: (a) Performance of $\hat{Z}_{s}^{1}$ for DisentangledSSL across varying values of $\beta$.
Refer to caption: x5.png

Analysis for different levels of entanglement. We further examine the trade-off between expressivity and redundancy for the learned shared representations on synthetic data with varying levels of multimodal entanglement. In this scenario, some dimensions align across modalities with MNI attainable, while others remain entangled with MNI unattainable. Specifically, we split the 50-dimensional true shared latent $Z_{c}$ into two parts: the first 35 dimensions, denoted as $Z_{c}^{\text{mix}}$, and the last 15 dimensions, denoted as $Z_{c}^{\text{pure}}$. We then generate 100-dimensional observations $X^{1}$ and $X^{2}$, where the first 85 dimensions are generated under the same procedure as before, i.e. $X^{1}_{\text{mix}}=T_{1}\cdot[Z_{s}^{1},Z_{c}^{\text{mix}}]$ and $X^{2}_{\text{mix}}=T_{2}\cdot[Z_{s}^{2},Z_{c}^{\text{mix}}]$ followed by adding Gaussian noise and random dropout, while the last 15 dimensions are directly $Z_{c}^{\text{pure}}$.

We present the results on the synthetic data with a mixed level of entanglement in Figure 9.
Figure 9(a) shows the test accuracy of $\hat{Z}_{c}^{1}$ on $Y_{c}$, $Y_{s}^{1}$, and $Y_{s}^{2}$ on the left axis, alongside the MLP weight ratio on $X^{1}$ and $X^{2}$ on the right axis, for varying values of $\beta$.
To compute the MLP weight ratio, we first extract the diagonal of the inner product of MLP encoder’s first layer weight matrix, then calculate the ratio between the average of the last 15 “pure” dimensions and the first 85 “mixed” dimensions. This ratio indicates how much attention the encoder gives to the “pure” versus “mixed” dimensions, with higher values signifying greater focus on the “pure” dimensions. Figure 9(b) shows the corresponding test accuracy on $Y_{c}$ versus $Y_{s}^{2}$ in line plot.

Using such data with mixed entanglement levels, DisentangledSSL demonstrates a clear pattern where the learned information plateaus at certain $\beta$ values. As illustrated in Figure 9, the MLP weight ratio initially rises sharply from around 20 to nearly 80, then drops to 1 when $\beta$ becomes very large, indicating the collapse of the learned representations. With a large value of $\beta$ (while before the model collapses, i.e. $\beta\approx 10$), the encoders focus mainly on the “pure” dimensions. This is because a stronger information bottleneck constraint discourages the extraction of shared components from the “mixed” dimensions, which inevitably include modality-specific information due to unattainable MNI, and favors the “pure” shared components with no extra cost.

Figure: (a) Test accuracy of $\hat{Z}_{c}^{1}$ and the MLP weight ratio with varying $\beta$.
Refer to caption: x9.png

### H.2 MultiBench.

Experimental details. We follow the same dataset splitting, and utilize the same encoder architecture and pre-extracted features as Liang et al. ().
All models use representations (or concatenation of them, for FactorCL-proj, FOCAL, JointOpt and DisentangledSSL) with the same dimensionality.
We set the latent dimension to 300 across all datasets, except for MUSTARD, where it is set to 100 due to the smaller dataset size, and for MIMIC, where we use 360 to align with the output dimension of the GRU encoders. For FactorCL, we use their default hyperparameter settings. For other methods, hyperparameters are tuned based on validation set performance. For DisentangledSSL, we use $\beta=1.0$ and $\lambda=10^{-3}$ for all datasets, except for MOSI where $\beta=0.01$. For FOCAL and JointOpt, we use $a=1$ and $\lambda=10^{-3}$ across all datasets. We report the mean and standard deviation of the linear probing accuracy on prediction labels from the test set over 3 random seeds.

**Table 4: Prediction accuracy (%) of the representations learned by different methods on MultiBench datasets and standard deviations over 3 random seeds.**
| Dataset | MIMIC | MOSEI | MOSI | UR-FUNNY | MUSTARD |
| --- | --- | --- | --- | --- | --- |
| CLIP | 64.97 (0.60) | 76.87 (0.45) | 64.24 (0.88) | 62.73 (0.92) | 56.04 (4.19) |
| FactorCL-emb | 65.25 (0.45) | 71.80 (0.64) | 62.97 (0.81) | 63.29 (2.07) | 56.76 (4.66) |
| FactorCL-proj | 59.43 (1.70) | 74.61 (1.65) | 56.02 (1.26) | 61.25 (0.47) | 55.80 (2.18) |
| FOCAL (shared) | 62.69(1.46) | 75.93(0.23) | 62.10(0.44) | 62.38(0.58) | 54.83(3.02) |
| FOCAL (specific) | 62.13(1.49) | 75.41(0.54) | 61.61(1.27) | 63.17(0.96) | 58.21(2.21) |
| FOCAL (both) | 64.42 (0.34) | 76.77 (0.51) | 63.65 (1.09) | 62.98 (1.52) | 54.35 (0.00) |
| JointOpt (shared) | 63.01(0.59) | 76.69(0.28) | 65.02(1.96) | 62.51(1.02) | 54.83(4.82) |
| JointOpt (specific) | 65.81(0.49) | 74.40(0.94) | 53.89 (0.80) | 62.13(0.69) | 57.73(4.12) |
| JointOpt (both) | 66.11 (0.64) | 76.71 (0.14) | 64.24 (1.75) | 63.58 (1.45) | 56.52 (2.61) |
| DisentangledSSL (shared) | 63.16 (0.48) | 76.94 (0.22) | 65.16 (0.81) | 64.14 (1.53) | 54.11 (1.51) |
| DisentangledSSL (specific) | 65.73 (0.09) | 75.99 (0.60) | 51.70 (0.72) | 60.27 (1.28) | 61.60 (2.61) |
| DisentangledSSL (both) | 66.44 (0.31) | 77.45 (0.06) | 65.11 (0.80) | 64.24 (1.54) | 56.52 (2.18) |

Additional results. As a complement to Table 1, we present results of FOCAL and JointOpt for the learned shared and modality-specific representations, both combined (i.e. “both”) and separately in Table 4.
Although they exhibit a similar trend as DisentangledSSL, where shared representations are more important for MOSI and specific representations are crucial for MUSTARD, the distinction is less pronounced. Moreover, the overall performance is inferior to that of DisentangledSSL, indicating that DisentangledSSL achieves a better balance between coverage and disentanglement in the learned representations.

### H.3 High-Content Drug Screening

Experimental details. Following the setup in Wang et al. (), we utilize Mol2vec (Jaeger et al., ) to featurize the molecular structures into 300-dimensional feature vectors. For both molecular structures and phenotypes, we employ 3-layer MLP encoders with a hidden dimension of 2560. The dimensionality for shared and modality-specific representations is set to 32 across all methods and datasets. We tune $\beta$ based on validation set performance, with $\beta=5.0$ for RXRX19a and $\beta=1.0$ for LINCS, and set $\lambda=0.01$ for both datasets. For FOCAL and JointOpt, we set $a=1.0$ and $\lambda=0.01$. For DMVAE, we set the coefficient of the KL divergence term as $10^{-5}$. In addition, as in Lee & Pavlovic (), we introduce an InfoNCE loss for the shared representations to DMVAE in the high-content drug screening experiments to enhance its retrieving accuracy. We report the mean and standard deviation of all results on the test set over 3 random seeds.

Counterfactual generation. To further evaluate the learned modality-specific representations, we use different combinations of the shared and modality-specific representations to generate counterfactual samples, i.e. predicting phenotype of a drug on a different cell with its molecule shared latent but the phenotype specific latent of a different cell perturbed by other drugs. Since the factual observations of counterfactual generation are unavailable, we measure the performance distributional-wise, introducing the difference in Frechet distance (Diff-FD) as a metric, i.e. Diff-FD-c measuring the gain of using the correct batch of molecules, thus the information of the shared latent; Diff-FD-s measures the gain of using the correct batch of cells, thus the information of the specific latent.

To be specific, we train decoders with the factual combination of the shared latents from molecular structures and the modality-specific latents from phenotypes on the training set. We use “val recon” to denote the samples generated using the representations attained from validation set molecules and validation set phenotypes as input. Similarly, “test recon” refers to the samples generated using test set molecules paired with test set phenotypes, while “counterfactual” represents those generated using test set molecules paired with validation set phenotypes. Formally, Diff-FD-c is defined as FD(val recon; test recon) - FD(counterfactual; test recon), and Diff-FD-s as FD(val recon; test recon) - FD(counterfactual; val recon).
Note that a non-informative specific latent can lead to very high Diff-FD-c, and an overly informative modality-specific latent can lead to very high Diff-FD-s, thus we take both metrics into consideration together.

As shown in Table 5, DisentangledSSL outperforms JointOpt in both metrics and surpasses FOCAL in Diff-FD-s. While FOCAL shows a high Diff-FD-s, it learns overly informative modality-specific latents, which contains significant shared information, as highlighted in the simulation study results (i.e. the modality-specific representations of FOCAL have high accuracy on $Y_{c}$ in Figure 8), leading to a low value of Diff-FD-c. In contrast, DisentangledSSL demonstrate better effectiveness in capturing both shared and modality-specific information, while maximizing the separation between them, as indicated by its high values in both metrics.

**Table 5: Results on counterfactual generation.**
| Dataset | RXRX19a | LINCS |  |  |
| --- | --- | --- | --- | --- |
| Metric | Diff-FD-c | Diff-FD-s | Diff-FD-c | Diff-FD-s |
| FOCAL | 15.30(0.85) | 22.89(0.28) | 0.246(0.039) | 0.765(0.027) |
| JointOpt | 20.34(0.47) | 9.40(0.39) | 0.248(0.029) | 0.333(0.050) |
| DisentangledSSL | 20.92(0.71) | 11.36(0.36) | 0.277(0.031) | 0.401(0.085) |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Please check the weakness section.
soundness: 2
strengths: 1. The paper presents a rigorous, information-theoretic framework for disentangling data for different domains and modalities under conditions where MNI is unattainable.

2. Based on the theoretical analysis, the authors designed a two-step training algorithm, and provided an optimality guarantee for this proposed method theoretically.
summary: The paper introduces a self-supervised learning approach to disentangle shared and modality-specific information in multimodal data. The authors also explain that the optimal case in prior work is not doable based on Minimum Necessary Information (MNI), as the comprehensive analysis of the optimality. The experiments on vision-language and molecule-phenotype retrieval tasks demonstrate its effectiveness.
weaknesses: Weakness:
For the experiments on synthetic data, they only contain the data for unattainable MNI. The synthetic experiments for attainable MNI are also essential here to demonstrate the reliability of the theoretical bound. My suggestion is to add Gaussian noise produced by different variances as the degrees of MNI attainable or unattainable.


1. In Figure 4 (b), the superiority of DISENTANGLEDSSL is not significant compared to other baselines, such as JointOPT. The hyperparameter front of JointOPT is closer to the front of DISENTANGLEDSSL.

2. Table 1 shows the three versions of DISENTANGLEDSSL are unstable. For example, DISENTANGLEDSSL (shared) can achieve promising performance on MOSI while failing on MUSTARD. It would be better to explain why such a phenomenon exists, probably focusing on the main difference between these three versions and properties of multi-modal data. 

3. The performance gain shown in Table 2 is quite limited. Could the author explain if small improvements might be important in this domain? In addition, it would be better if the authors could provide the reason for this tiny improvement. 

4. The paper does not satisfy the page limitation.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: Although in most cases DisentangledSSL-both achieves highest performance, there are also several tasks where either  DisentangledSSL-shared or  DisentangledSSL-specific alone achieves top performance (and sometimes by quite a large margin over  DisentangledSSL-both, such as Mustard). Do you observe any trend / suggest any guidelines for when it is best to use each configuration?
soundness: 4
strengths: 1. The paper introduces a new perspective in how to perform effective multimodal representation learning. Specifically, the paper demonstrated that disentanglement of modality-specific and shared information into 2 separate representations can be effective in improving downstream task performance, which seems to be original and novel.

2. The authors provided extensive theoretical justifications and proofs for their proposed approach.

3. The experiments are quite comprehensive. They include both synthetic and real-world datasets, and the proposed method is evaluated against several multimodal SSL baselines on different tasks from drastically different domains. The proposed method achieved top performance in almost all tasks.

4. All experiment details are presented in the Appendix (high reproducibility).
summary: In this paper, the authors proposed a novel self-supervised representation learning method for multimodal representation learning. The proposed method disentangles the representation of multimodal information into shared information and modality-specific information, and uses a separate neural network to learn each representation. The authors supported their proposed method through extensive theoretical analysis and proofs. The proposed method was evaluated on both synthetic datasets and real-world multimodal datasets, where the proposed method achieved top performance over baselines in most tasks.
weaknesses: While the extensive use of information-theory notations allowed rigorous proof of the proposed approach, it isn't really easy for readers who isn't interested in the proofs and just want to learn about the concrete algorithm/methodology. Perhaps the authors should consider including a Pseudocode/Alg block either in the main text or in Appendix to clearly demonstrate the training process.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please respond to the weaknesses listed above.
soundness: 2
strengths: It makes sense to learn disentangled shared and modality-specific representations of multimodal data. 

The theoretical analysis to evaluate the quality of disentanglement, which can be applied even in cases where Minimum Necessary Information (MNI) is unattainable, is provided.

The efficacy of DISENTANGLEDSSL is demonstrated across a range of synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.
summary: The authors believe that "shared information between modalities is precisely what is relevant for downstream tasks" and "the modality gap ... results in misalignment between modalities, restricting the application of these methods in numerous real-world multimodal scenarios." Hence they motivate "the need for a disentangled representation space that captures both shared and modality-specific information". To address this need, they propose DISENTANGLEDSSL, an information-theoretic framework designed to learn disentangled shared and modality-specific representations of multimodal data. The authors have conducted a theoretical analysis to evaluate the quality of disentanglement, which can be applied even in cases where Minimum Necessary Information (MNI) is unattainable. The efficacy of DISENTANGLEDSSL is demonstrated across a range of synthetic and real-world multimodal datasets and tasks, including prediction tasks for vision-language data and molecule-phenotype retrieval tasks for biological data.
weaknesses: Some of the key assertions motivating this work are inaccurate, and the proposed graphical model is flawed, which together raise questions about the validity of the resulting framework.

For instance, the statement "shared information between modalities is precisely what is relevant for downstream tasks" is not universally accurate. What is relevant depends heavily on the specific downstream tasks. Furthermore, "shared information" is a vague term, especially when applied across modalities, making it problematic in this context.

The assertion that "the modality gap ... results in misalignment between modalities, restricting the application of these methods in numerous real-world multimodal scenarios" is also oversimplified. The modality gap is not the sole or primary cause of misalignment; other factors such as alignment methods, distributional shifts, domain shifts, sampling biases, and more can contribute significantly.

I suggest the authors to refine their motivation to more accurately reflect the complexities of multimodal relationships, while still maintaining the core idea of disentangling shared and modality-specific information.

The graphical model in Figure 2, intended to represent the generative process (as claimed in line 128), has two major issues. First, the authors have conflated the generative and inference processes. Only the pathway from Z to X represents the generative model, while the path from X to \hat{Z} corresponds to inference, which should not be included in the data-generating graphical model. This issue is straightforward to resolve by removing the inference process from the model, which would not affect the algorithm or results, or to create separate diagrams, one for the generative process and the other for inference to clarify.

The second issue is more severe: the assumption of a shared latent variable Z_c existing between two modalities may not hold. This assumption lacks foundation, as it is more likely that the relationship takes the form of two variables Z_c^1 and Z_c^2, with potential connections between them:  (1)  Z_c^1 -> Z_c^2, (2)  Z_c^1 <- Z_c^2, or (3) or Z_c^1 <-C-> Z_c^2, depending on the modalities involved. This discrepancy significantly undermines the justification for the proposed algorithm.  I encourage the authors to discuss the implications of using separate but potentially connected latent variables (Z_c^1 and Z_c^2) instead of a single shared Z_c, and how might this change affect the proposed algorithm and results. This could lead to a more nuanced and realistic model of multimodal relationships.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: Are there any alternative approaches for implementing the $I(Z^1;\hat{Z}_{c}^{1*})$ in $L_s^1$? If so, how does their performance compare to that of the orthogonal loss implementation?
soundness: 4
strengths: 1.The idea of applying conditional entropy bottleneck (CEB) in multi-modal self-supervised learning is novel.

2.Comprehensive theoretical analysis is conducted to prove the optimality of both modality-specific and modality-shared information.

3.DisentangledSSL demonstrates superior performance across tasks, including vision-language prediction and molecule-phenotype retrieval.
summary: This paper addresses the disentanglement of modality-shared and modality-specific information in multi-modal self-supervised learning and proposes DisentangledSSL. From an information-theoretic perspective, the modality-shared information is optimized using a conditional entropy bottleneck (CEB). Correspondingly, the authors formulate an optimization problem to isolate modality-specific information. Theoretical analysis of the optimality of each disentangled representation, particularly when Minimum Necessary Information is unattainable, along with experimental results, demonstrates the superiority of DisentangledSSL.
weaknesses: 1.How does the optimization objective for the modality-shared representation change from Eq. (1) to Eq. (3)? Why is $I(X^1;X^2)$ omitted?

2.Related studies, such as CoCoNet [1] and SimMMDG [2], which also address the separation of modality-shared and modality-specific information, are recommended for inclusion and comparison. Note that SimMMDG [2] can be easily adapted to a self-supervised setting by substituting supervised contrastive learning with the original self-supervised counterpart.

3.Ablation studies on the impact of $\beta$ and $\lambda$ in the MultiBench datasets should be provided to demonstrate the importance of separating modality-specific and modality-shared information in real-world applications. 

4.Some typos.
1)$Z^2$ should be $X^2$ in Eq.(6) and in the equation below Eq.(6).
2)$I(X^1;Z^1)$ should be $I(X^1;X^2)$ in Figure 3.
 
[1] Li J, Qiang W, Zheng C, et al. Modeling multiple views via implicitly preserving global consistency and local complementarity. TKDE, 2022.
[2] Dong H, Nejjar I, Sun H, et al. SimMMDG: A simple and effective framework for multi-modal domain generalization. NIPS, 2023.

Meta-review: The submission received the ratings of four reviewers, which recommended 5, 6, 6 and 8, averaging 6.25. Given the plenty of competitive submissions in ICLR, this stands at a score above the borderline. The reviewers' concerns focus on the unclear motivation, assumption and some insufficient demonstration in performance. After the substantial rebuttal by authors, most of reviewers considered the concerns well addressed, and one reviewer still maintain some concerns about the performance gain ~2%. After carefully checking the reviewer comments and the authors' feedback, the AC considered that although in real-world case, there are some confounding factors that affect the performance, however, for scientific research, we should allow some tolerance in ideal settings for clear understanding and overcoming problem one by one. Therefore, I tend to recommend acceptance towards the current submission, and hope all advice are well incorporated into the final submission.

Decision: Accept (Poster)