An Information Criterion for Controlled Disentanglement of Multimodal Data

Authors: chenyu  wang
sharut  gupta
xinyi  zhang
sana  tonekaboni
stefanie  jegelka
tommi  jaakkola
caroline  uhler
t u munich
i e |i
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-24 05:45:19

Abstract: Multimodal representation learning seeks to relate and decompose information available in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and 
enable tasks like counterfactual generation. However, separating these components is challenging due to their deep entanglement in real-world data. We propose $\textbf{Disentangled}$ $\textbf{S}$elf-$\textbf{S}$upervised $\textbf{L}$earning (DisentangledSSL), a novel self-supervised approach that effectively learns disentangled representations, even when the so-called $\textit{Minimum Necessary Information}$ (MNI) point is not achievable. It outperforms baselines on multiple synthetic and real-world datasets, excelling in downstream tasks, including prediction tasks for vision-language data, and molecule-phenotype retrieval for biological data.

Main paper: 
 introduction: 
 Multimodal representation learning integrates information from different modalities to form holistic representations, with applications ranging from vision-language tasks [40, 21, 25] to biological data analysis [39, 41] . Models like CLIP [25] use self-supervised learning to capture shared information, assuming that mutual information across modalities is key for downstream tasks [34, 31] . However, the modality gap-stemming from inherent differences in representation and content-causes misalignment and limits their effectiveness in real-world scenarios [18, 26, 10] . This highlights the need for a disentangled representation space that captures both shared and modality-specific information effectively. Such a space, accounting for both coverage and disentanglement of the information, is crucial for interpretability and tasks where distinct modalities provide essential insights [42, 17, 20] .Disentangled representation learning in multimodal data began with works on Variational Autoencoders and Generative Adversarial Networks [14, 4, 5] aiming to isolate data variations. Recent self-supervised methods have advanced this by learning shared and modality-specific information either sequentially or jointly [20, 15, 28] . Zhang et al. [42] emphasized the importance of learning disentangled representations for multimodal data in biological contexts. However, rigorous guarantees for disentanglement are lacking, especially when the Minimum Necessary Information (MNI) [7] point is unattainable. In many real-world applications, shared and modality-specific information are deeply entangled, i.e. the shared and modality-specific components are intertwined and result in similar observations, leading to unattainable MNI. Figure 1 illustrates an example in the context of high-content drug screens. We provide additional examples of unattainable MNI in Appendix B.In this work, we propose DISENTANGLEDSSL, a self-supervised approach for multimodal data that effectively separates shared and modality-specific information. Building on information theory, we devise a step-by-step optimization strategy to learn these representations and maximize the variational Figure 1 : Post-perturbation phenotype (X 1 ) (i.e., cellular images or gene expression after the application of a drug to cells) and molecular structure (X 2 ) of an underlying drug perturbation system where cancer cells are targeted and killed while healthy cells remain unaffected. The Venn diagram illustrates shared and specific information between modalities X 1 and X 2 : shared content is shown in red, modality-specific content in green, and entangled content due to unattainable MNI in orange. For example, for the drug mechanism, the molecular structure conveys full information, while the phenotype provides partial information (i.e. mechanisms causing cell death). Similarly, for the states of healthy cells, the phenotype specifies their cell states, whereas the molecular structure only indicates that the cells are unaffected without detailing their specific states. lower bound on our objectives. Unlike existing works, we tackle challenging cases where MNI is unattainable and provide a formal analysis of representation optimality. Empirical results show that DISENTANGLEDSSL outperforms baselines on various datasets, including multimodal prediction in MultiBench [16] and molecule-phenotype retrieval in high-content drug screening datasets [32, 3] . 
method: 
 In this section, we detail our proposed method, DISENTANGLEDSSL, for learning disentangled representations in multimodal data. We begin by outlining the graphical model that formalizes the problem in Section 2.1 and defining the key properties of "desirable" representations in Section 2.2. Subsequently, we describe the DISENTANGLEDSSL framework in Section 2.3. We provide theoretical proofs of the optimality in Appendix E, and specific training objectives in Appendix H.DISENTANGLEDSSL learns disentangled representations in latent space, separating modality-specific information from shared factors across paired observations (X 1 , X 2 ). This generative process is modeled in Figure 2 . Each observation is generated from two distinct latent representations: the modality-specific representations (Z 1 s and Z 2 s ) that contain information exclusive to their respective modalities, and a shared representation (Z c ) that contains information common to both modalities. We refer to these as the true latents. ). The modality-specific information for each modality is encoded by variables Ẑ1 s and Ẑ2 s . Note that for the true latents, Z 1 s and Z c are conditionally dependent on X 1 due to the V-structure in the graphical model. To preserve such dependencies in the inferred latents, Ẑ1 s and Ẑ2 s are conditioned on both the respective observations and the inferred shared representations, withẐ1 s ∼ p(•|X 1 , Ẑ1 c ) and Ẑ2 s ∼ p(•|X 2 , Ẑ2 c ).We establish information-theoretic criteria to ensure the shared and modality-specific representations are informative and disentangled, capturing key features while minimizing redundancy.The shared representations Ẑ1 c and Ẑ2 c should provide a compact yet expressive form of shared information. This trade-off between compression and expressivity is studied by the principles of the information bottleneck (IB) in both supervised and self-supervised settings [33, 12, 30, 29, 35] . IB objective has been utilized to learn the optimal representations Z 1 of X 1 with respect to X 2 under the Markov chain Z 1 ← X 1 ↔ X 2 . It balances the trade-off between preserving relevant information about X 2 , i.e. I(Z 1 ; X 2 ), and compressing the representation, i.e. I(Z 1 ; X 1 ) (see more details in Appendix C). The optimal representation should be both sufficient, i.e. I(Z 1 ; X 2 ) = I(X 1 ; X 2 ) [1, 29] , and minimal [1] . Based on these criteria, Z 1 is said to capture Minimum Necessary Information (MNI) [7] between X 1 and X 2 if I(X 1 ; X 2 ) = I(Z 1 ; X 2 ) = I(Z 1 ; X 1 ), indicating an ideal scenario of full disentanglement between X 1 and X 2 with no extraneous information, i.e. I(Z 1 ; X 1 |X 2 ) = 0 2 .. In general, it may be unattainable for an arbitrary distribution p(X 1 , X 2 ) (see Appendix F). Despite its significance, the optimality of latent representations when MNI is unattainable is often overlooked in prior work.We propose a definition of the optimality of the shared representations that applies to both scenarioswhen MNI is attainable or not, as defined in Equation 1:EQUATIONFormally, minimizing conditional mutual information, I(Z 1 ; X 1 |X 2 ), ensures the shared representation captures only the truly common information between X 1 and X 2 , excluding modality-specific details unique to X 1 . Compared with I(Z 1 ; X 1 ) in IB, it provides a more precise measure of compression and a more robust objective.Figure 3 : IB CurveThe constraint I(X 1 ; X 2 ) -I(Z 1 ; X 2 ) ≤ δ c ensures that Ẑ1 * c retains a substantial portion of the shared information between X 1 and X 2 , controlling the difference within the limit δ c and preventing significant information loss. We utilize the IB curve F (δ) 3 [12, 8] , representing the maximum I(Z 1 ; X 2 ) for a given compression level I(X 1 ; Z 1 ) ≤ δ, to illustrate the optimality in Figure 3 . MNI is depicted as point A, and Ẑ1 * c corresponding to δ c is shown as point C. When MNI is attainable, setting δ c = 0 achieves MNI. In contrast, Achille and Soatto [1] formulated the optimization asZ 1 = arg min Z 1 :Z 1 -X 1 -X 2 I(X 1 ; Z 1 ), s.t. I(Z 1 ; X 2 ) = I(X 1 ; X 2 ),leading to MNI when attainable. This holds in supervised settings, assuming the data label X 2 is a deterministic function of X 1 , as used by previous methods [12, 7, 24] . However, in general multimodal self-supervised scenarios where MNI is not attainable, this results in point B in Figure 3 , which includes information of X 1 that has little relevance to X 2 to satisfy the equality constraint, causing a gap between the objective and the ideal representation.The modality-specific representations Z 1 s and Z 2 s should capture information unique to each modality, being highly informative yet minimally redundant with the shared representations. The formal definition of these optimal representations is provided in Equation 2.Ẑ1 * s = arg maxZ 1 I(Z 1 ; X 1 |X 2 ), s.t. I(Z 1 ; Ẑ1 * c ) ≤ δs Ẑ2 * s = arg max Z 2 I(Z 2 ; X 2 |X 1 ), s.t. I(Z 2 ; Ẑ2 * c ) ≤ δs (2) 2 I(Z 1 ; X 1 |X 2 ) = I(Z 1 ; X 1 , X 2 )-I(Z 1 ; X 2 ) = I(Z 1 ; X 1 )-I(Z 1 ; X 2 ) = 0,where the first equality is due to the property of conditional mutual information, and the second is due to the Markov structure Z 1 -X 1 -X 2 . 3 The IB curve is concave [8] , monotonically non-decreasing, and upper bounded by line I(Z 1 ; X 2 ) = I(Z 1 ; X 1 ) and line I(Z 1 ; X 2 ) = I(X 1 ; X 2 ) (see details and proofs in Appendix D).To extract modality-specific information, Ẑ1 * s maximizes the conditional mutual information I(Z 1 ; X 1 |X 2 ), capturing details unique to X 1 without redundancy from X 2 . This same term is minimized for optimal shared representations in Equation 1. Integrating both representations together, the objectives ensure comprehensive coverage of all relevant information from each modality. Meanwhile, the constraint I(Z 1 ; Ẑ1 * c ) ≤ δ s limits redundancy between the modality-specific representation and the shared representation Ẑ1 * c , ensuring disentanglement by separating unique aspects of X 1 from shared components with X 2 . The parameter δ s controls the trade-off between information coverage and disentanglement.To achieve the optimal representations discussed in Section 2.2, we introduce a two-step training procedure. The first step focuses on optimizing for the shared latent representation to captures the minimum necessary information as close as possible. Then, the second step utilizes the learned shared representations in step 1 to facilitate the learning of modality-specific representations. This sequential approach is formalized in the optimization objectives given in Equations 3 and 4.Step 1: Learn the shared latent representations by encouraging the shared representation encoded from one modality to be highly informative about the other modality, while minimizing redundancy.EQUATIONStep 2: Learn the modality-specific latent representations based on the learned shared representations.EQUATIONThe hyperparameters β and δ balance relevance and redundancy in the shared and modality-specific representations respectively. We use the same values for both modalities since they operate on similar information scales. Our sequential training approach, instead of a joint one, stems from the selfsufficient nature of each optimization procedure, where a sub-optimal representation does not enhance the learning of the other. We provide theoretical proofs of the optimality of DISENTANGLEDSSL in Appendix E, and specific training objectives for each information term in Appendix H. 
experimental results: 
 We conduct experiments on a simulation study (results in Appendix I.1) and two real-world multimodal settings. We compare DISENTANGLEDSSL against a disentangled variational autoencoder baseline, DMVAE [14] , as well as various multimodal contrastive learning methods, including CLIP [25] , which aligns modalities to learn shared representations, and FOCAL [20] and Fac-torCL [17] , which aim to capture both shared and modality-specific information. Additionally, we test a joint optimization variant, JointOpt [24] , to demonstrate the benefits of our step-by-step approach.We utilize the real-world multimodal benchmark from MultiBench [16] , which includes curated datasets across various modalities, such as text, images, and tabular data, along with a downstream label that we expect shared and specific information to have varying importance for. We evaluate the linear probing accuracy of representations learned by each pretrained model, as shown in Table 1 .FactorCL-emb refers to the embeddings of FactorCL before projection heads, while FactorCLproj uses the concatenation of all projection head outputs. All models use representations (or concatenation of them, if applicable) with the same dimensionality. We present results for the combined shared and specific representations for FOCAL and JointOpt in 3.2 High-content Drug Screening.Dataset description. As characterized in Figure 1 , we use two high-content drug screening datasets which provide phenotypic profiles after drug perturbation: RXRX19a [32] containing cell imaging profiles, and LINCS [3] . We conduct train-validation-test splitting according to molecules. Models are pretrained to learn representations of molecular structures and corresponding phenotypes. We provide additional details on experimental settings in Appendix I.3.Molecule-phenotype retrieval using shared representations. We evaluate the shared representations on the molecule-phenotype retrieval task, which identifies molecules likely to induce a specific phenotype. The shared information, which connects the molecular structure and phenotype, plays a key role in this task. We tune β according to validation set performance and show results of top N accuracy (N=1,5,10) and mean reciprocal rank (MRR) on the test set in Table 2 . DISENTANGLEDSSL significantly outperforms baselines on both datasets, demonstrating its effectiveness in capturing relevant shared features while excluding irrelevant details. Notably, compared to the variant without the information bottleneck constraint, i.e. DISENTANGLEDSSL (β = 0), the full DISENTANGLEDSSL model preserves critical shared features, achieving superior performance in this retrieval task, where shared information is essential. Disentanglement measurement. To assess the effectiveness of learning modality-specific representations, we introduce the Reconstruction Gain (RG) metric, which quantifies the disentanglement between shared and modality-specific representations. We use 2-layer MLP decoders to reconstruct the original data from the shared and modality-specific representations, both individually and jointly, and compute the reconstruction R 2 for each case. A higher gain in R 2 when using combined representations indicates lower redundancy and better disentanglement. As shown in Table 3 , DISEN-TANGLEDSSL achieves the highest RG scores across both modalities and datasets, demonstrating superior disentanglement. In contrast, other methods show redundancy due to insufficient constraints for disentanglement during pretraining. We also test the representations from the pretrained model on the counterfactual generation task, with detailed results provided in Appendix I.3.Disentangled representations in VAEs and GANs. Disentangled representation learning originated from works on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), focusing on isolating underlying data variations linked to or independent of labels (e.g., digit identity vs. writing style in MNIST) [9, 22, 2, 4] . This concept extended to multimodal data like text and image [13] and content and pose in video [5] , typically using self-and cross-reconstruction with adversarial loss to learn shared and specific components. However, these methods often address simple cases with attainable MNI, lacking a comprehensive analysis for complex real-world multimodal scenarios where MNI is not attainable.Information bottleneck and its variants. The information bottleneck (IB) principle has been used to analyze and optimize deep neural networks and the learned representations from the information theory perspective in both supervised and self-supervised settings [33, 8, 30, 12, 27, 35, 38, 29] , with extensions like the conditional entropy bottleneck (CEB) [7, 13] being developed to better refine the information plane. IB has been applied to learn molecular representations from drug screening data [19] . However, these approaches primarily focus on extracting shared features while minimizing specific features, limiting their practical use. Other methods [24, 28] learn disentangled representations by optimizing mutual information and using adversarial loss but are confined to single-modality scenarios and do not address complex multimodal settings with unattainable MNI.Multimodal disentanglement in self-supervised learning. Recent studies have explored disentangled representation learning in multimodal self-supervised contexts. Liang et al. [17] factorizes and optimizes mutual information bounds to capture the union of shared and unique features, while not penalizing for redundancy between latents. Zhang et al. [42] learns disentangled representations of cell state for multimodal data in biological contexts. Liu et al. [20] and Li et al. [15] use mutual information optimization with orthogonal/MMD regularizations for disentangling multimodal time-series signals and T-cell receptor segments, respectively. However, these methods lack a general framework for understanding information content, particularly when data modalities are deeply entangled in real-world applications.In this section, we present two additional examples where the Minimum Necessary Information (MNI) is unattainable. These examples offer a clear understanding of how unattainable MNI manifests in data, providing practical insights and making the concept more accessible.Z 1 s Zc Z 2 s X 1 X 2True Latents Firstly, we show an example in a simple mathematical case. As illustrated in Figure 4 , we have true latent variables are independently drawn from a Bernoulli distribution, i.e. Z 1 s , Z c , Z 2 s ∼ Bernoulli(0.5), and observations X 1 and X 2 are generated according toX 1 = OR(Z c , Z 1 s ) and X 2 = OR(Z c , Z 2 s ).In this scenario, if we observe X 1 = 1, we are not able to distinguish whether this result is due to Z c = 1, or Z 1 s = 1, or both. Thus, extracting purely shared features from the observations is infeasible.Further, we provide an example in 3D geometry, as illustrated in Figure 5 . For information such as the height of the cone and the number of spheres, one modality conveys the complete information while the other conveys partial information. This ambiguity makes it challenging to classify the information as purely shared or modality-specific, placing it in an uncertain category.The Information Bottleneck (IB) principle provides a powerful theoretical foundation for our method. IB objective seeks to find a representation Z 1 of a random variable X 1 that optimally trades off the preservation of relevant information with compression [33] . Relevance is defined as the mutual information, I(Z 1 ; X 2 ), between the representation Z 1 and another target variable X 2 . Compression Figure 5 : Image modality (X 1 ) and text modality (X 2 ) of an underlying reality. The Venn diagram illustrates shared and specific information between modalities X 1 and X 2 : shared content is shown in red, modality-specific content in green, and entangled content due to unattainable MNI in orange. For example, for the cone's height, the image conveys full information, while the text (i.e. "tall red cone") provides partial information. Similarly, for the number of spheres, the text specifies it completely, whereas the image indicates there is at least one sphere. is enforced by constraining the mutual information between the representation and the original data, I(X 1 ; Z 1 ), to lie below a specified threshold. This is can be formalize in terms of the following constrained optimization problem:arg max Z 1 I(Z 1 ; X 2 ), s.t. I(Z 1 ; X 1 ) ≤ δ (5)where Z 1 is from a set of random variables that obey the Markov chain Z 1 ↔ X 1 ↔ X 2 . In practice, Equation 5 can be optimized by minimizing the IB Lagrangian in Equation 6.EQUATIONHere, the Lagrangian multiplier β controls the emphasis placed on compression versus expressiveness.Fischer [7] extends IB to the conditional entropy bottleneck (CEB) objective, which utilizes the conditional mutual information term I(Z 1 ; X 1 |X 2 ) in place of the mutual information term I(Z 1 ; X 1 ) in Equation 6:L = I(Z 1 ; X 1 |X 2 ) -βI(Z 1 ; Z 2 )While being equivalent to the IB Lagrangian in terms of the information criterion, CEB rectifies the information plane to precisely measure compression using the conditional mutual information, and offers a clear assessment of how close the compression is to being optimal.Note that although CEB aligns with our step 1 objective in Equation 3, its optimality has not been fully examined in Fischer [7] , particularly under scenarios where MNI is unattainable. Furthermore, Fischer [7] primarily addresses a supervised learning context, where X 2 serves as the label for X 1 . In contrast, our study tackles the more complex multimodal setting, where X 1 and X 2 are two distinct data modalities. We extend the analysis to both attainable and unattainable MNI cases, demonstrating the efficacy of our approach in capturing shared and modality-specific information under these challenging scenarios. This broadens the applicability of CEB beyond traditional labelbased supervised settings to multimodal data with complex modality entanglements.The information plane is a helpful visualization of the information bottleneck principle, which utilizes I(X 1 ; Z 1 ) and I(X 2 ; Z 1 ) as its coordinates that represents the trade-off between compression and prediction [33] . The frontier of all possible Z 1 s is called the IB curve or F (r) [12] as follows:F (r) := max Z 1 :Z 1 -X 1 -X 2 I(Z 1 ; X 2 ) s.t. I(X 2 ; Z 1 ) ≤ rThe IB curve has the following properties:• The IB curve is concave (Lemma 5 in [8] ) and monotonically non-decreasing.• The IB curve is upper bounded by line I(X 2 ; Z 1 ) = I(X 1 ; Z 1 ) and line I(X 2 ; Z 1 ) = I(X 1 ; X 2 ), according to the Markov relationship Z 1 -X 1 -X 2 . • When the MNI point is attainable for random variables X 1 and X 2 , the IB curve has the following formula [27, 12] :I(Z 1 ; X 2 ) = I(Z 1 ; X 1 ), if I(Z 1 ; X 1 ) ≤ I(X 1 ; X 2 ) I(X 1 ; X 2 ), if I(X 1 ; X 2 ) < I(Z 1 ; X 1 ) ≤ H(X 1 )and the Gateaux derivative of I(X 2 ; Z 1 ) with respect to p(z 1 |x 1 ) (as used in [24] ) doesn't exist at the MNI point (see details in Appendix G).This section explores how the step 1 objective, L 1 c , optimizes the shared representation between modalities by balancing expressivity and redundancy. We discuss its effectiveness in both scenarioswhen MNI is attainable or not.The step 1 objective L 1 c in Equation 3 seeks representation Z 1 that maximizes the information shared between the modalities, i.e. I(Z 1 ; X 2 ), while minimizing the information unique to each modality, i.e. I(Z 1 ; X 1 |X 2 ), to capture only the essential shared content. This aligns with the conditional entropy bottleneck (CEB) objective [7] , an extension of the IB Lagrangian L = I(Z 1 ; X 2 ) -βI(Z 1 ; X 1 ) (see details in Appendix C). While it serves as a robust objective for learning shared information, its optimality remains underexplored in Fischer [7] , particularly when MNI is unattainable. Additionally, Fischer [7] focuses on the supervised scenario where X 2 is the label of X 1 , whereas we address the multimodal case with X 1 and X 2 being two data modalities, demonstrating its effectiveness in both attainable and unattainable MNI scenarios.When MNI is attainable, Proposition 1 states that the step 1 optimization achieves MNI for any positive β. Proposition 1. If MNI is attainable for random variable X 1 and X 2 , maximizingL 1 c = I(Z 1 ; X 2 ) - βI(Z 1 ; X 1 |X 2 ) achieves MNI for any β > 0, i.e. I( Ẑ1 * c ; X 1 ) = I( Ẑ1 * c ; X 2 ) = I(X 1 ; X 2 ), where Ẑ1 * c := arg max Z 1 -X 1 -X 2 L 1 c .Proof. Based on data processing inequality and the Markov relationshipẐ1 c ← X 1 ↔ X 2 , I( Ẑ1 c ; X 2 ) ≤ I(X 1 ; X 2 ).Based on the non-negativity of conditional mutual information, I( Ẑ1 c ; X 1 |X 2 ) ≥ 0. Thus, for β > 0,L 1 c = I( Ẑ1 c ; X 2 ) -β • I( Ẑ1 c ; X 1 |X 2 ) ≤ I(X 1 ; X 2 ), ∀ Ẑ1 cwhere the equality holds when I(Ẑ1 c ; X 2 ) = I(X 1 ; X 2 ) and I( Ẑ1 c ; X 1 |X 2 ) = 0 Meanwhile, I( Ẑ1 c ; X 1 |X 2 ) = I( Ẑ1 c ; X 1 , X 2 ) -I( Ẑ1 c ; X 2 ) = I( Ẑ1 c ; X 1 ) -I( Ẑ1 c ; X 2 ) where the second equality is due to the conditional independence Ẑ1 c ⊥ ⊥ X 2 |X 1 in the Markov relationship. Therefore, L 1 c achieves maximality I(X 1 ; X 2 ) iff I( Ẑ1 c ; X 2 ) = I(X 1 ; X 2 ) = I( Ẑ1 c ; X 1 ), i.e. Ẑ1 c achieves MNI.When MNI is unattainable, for any representation Z 2 of X 1 , the inequalities I(Z 2 ; X 1 ) ≥ I(Z 2 ; X 2 ) and I(Z 2 ; X 1 ) ≥ I(X 1 ; X 2 ) can not achieve equality simultaneously. This indicates the existence of an inherent trade-off between capturing the entire shared information and avoiding the inclusion of modality-specific details. More precisely, under the condition of strict concavity of the I(Z 1 ; X 2 ) -I(Z 1 ; X 1 ) information curve 4 , such trade-off is presented in Proposition 2.Proposition 2. For random variables X 1 and X 2 , when the IB curve I(Z 1 ; X 2 ) = F (I(Z 1 ; X 1 )) is strictly concave, 1) there exists a bijective mapping from β in L 1 c to the value of information constraint δ c in the definition of optimal shared latent Ẑ1 * c in Equation 1; 2)∂I(Z 1 * β ;X 1 ) ∂β < 0, ∂I(Z 1 * β ;X 2 ) ∂β< 0, where Z 1 * β is the optimal solution corresponding to a certain β.Proof. Since the IB curve I(Z 1 ; X 2 ) = f IB (I(Z 1 ; X 1 )) is monotonically non-decreasing and strictly concave, it is monotonically increasing. When Z 1 = X 1 , f IB achieves the maximum point (H(X 1 ), I(X 1 ; X 2 )). Thus, I(X 1 ; X 2 ) ≥ I(X 1 ; X 2 ) -I( Ẑ1 * c ; X 2 ) ≥ 0 and is monotonically decreasing. Then there is a bijective mapping between δ = I(X 1 ; X 2 ) -I( Ẑ1 * c ; X 2 ) ∈ [0, I(X 1 ; X 2 )] and points in the IB curve.Since I(Z 1 ; X 1 |X 2 ) = I(Z 1 ; X 1 ) -I(Z 1 ; X 2 ), we have I(Z 1 ; X 1 |X 2 ) = f -1 IB (I(Z 1 ; X 2 )) - I(Z 1 ; X 2 ) := f -1 CIB (I(Z 1 ; X 2 )), where f CIB (r) = (f -1 IB (r) -r) -1 .Since f IB (r) ≤ r, the equality holds when r = 0, and f IB is strictly concave, we havedf IB dr ≤ df IB dr | r=0 < 1. Thus df -1 IB dr > 1. Then df CIB dr = ( df -1 IB dr -1) -1 > 0. Furthermore, since df 2 IB dr 2 < 0 (strict concavity of f IB ), df 2 CIB dr 2 = - 1 ( df -1 IB dr -1) 2•(-df 2 IB dr 2 ( df IB dr ) 2 ) < 0.Therefore, f CIB is also monotonically increasing and strictly concave. Then there is a bijective mapping between δ = I(X 1 ; X 2 ) -I( Ẑ1 * c ; X 2 ) ∈ [0, I(X 1 ; X 2 )] and points in the CIB curve f CIB .Since f CIB is increasing, the inequality constraint can be replaced by the equality constraint, i.e.Ẑ1 * c = arg minI(Z 1 ;X 2 )≥I(X 1 ;X 2 )-δ I(Z 1 ; X 1 |X 2 ) = arg min I(Z 1 ;X 2 )=I(X 1 ;X 2 )-δ I(Z 1 ; X 1 |X 2 )The corresponding Lagrangian function isEQUATION= f -1 CIB (I(Z 1 ; X 2 )) -β • I(Z 1 ; X 2 ) + β • (I(X 1 ; X 2 ) -δ)Based on the Lagrangian multiplier theorem, the optimal Ẑ1 * c is achieved whendL dI(Z 1 ; X 2 ) = df -1 CIB (I(Z 1 ; X 2 )) dI(Z 1 ; X 2 ) -β = 0 dL d β = -I(Z 1 ; X 2 ) + I(X 1 ; X 2 ) -δ = 0Thus,EQUATIONi.e. β is the slope of f -1 CIB when I(Z 1 ; X 2 ) = I(X 1 ; X 2 )-δ. Therefore, there is a bijective mapping between β = β-1 in L 1 c and points in f CIB , and thus δ. For any β 1 , β 2 > 0 andβ 1 > β 2 , we have β1 = β -1 1 < β -1 2 = β2 . According to formula 8, β1 = df -1 CIB (I(Z 1 ; X 2 )) dI(Z 1 ; X 2 ) | I(Z 1 ;X 2 )=I(Z * β 1 ;X 2 ) , β2 = df -1 CIB (I(Z 1 ; X 2 )) dI(Z 1 ; X 2 ) | I(Z 1 ;X 2 )=I(Z * β 2 ;X 2 ) (9)Since β1 < β2 and f CIB is a strictly concave function, f -1 CIB is strictly convex and we have I(Z * β1 ; X 2 ) < I(Z * β2 ; X 2 ). Furthermore, since f IB is monotonically increasing, we haveEQUATIONThe second property implies that as β increases, the learned representation becomes less informative and redundant, demonstrating the tradeoff between expressivity and redundancy when MNI is unattainable. Furthermore, the bijection mapping between β and δ c allows our step 1 optimization to navigate the information frontier across various β values, facilitating soft control and the segmentation of shared information into different degrees of granularity.In this section, we demonstrate the step 2 objective, L 1 s , ensures optimal coverage and disentanglement by showing its equivalence (or nearly equivalence) to the Lagrangian of Equation 2.The step 2 objective L 1 s in Equation 4 learns modality-specific representation Z 1 based on the optimal shared representations Ẑ1 * c and Ẑ2 * c learned from step 1. It aims to maximize the information coverage of the data observation X 1 through the combination of Z 1 and Ẑ2 * c , i.e. I(Z 1 , Ẑ2 * c ; X 1 ). Simultaneously, it promotes disentanglement by limiting overlap with the shared representation Ẑ1 * c of the same modality, indicated by I(Z 1 ; Ẑ1 * c ). Formally, L 1 s is the Lagrangian formulation of the constraint optimization for optimal modality-specific representations, as defined in Equation 2, however having the term I(Z 1 ; X 1 |X 2 ) substituted with I(Z 1 , Ẑ2 * c ; X 1 ). When MNI is attainable, as justified in Proposition 3, this substitution results in an equivalent objective. Consequently, L 1 s effectively guides the generation of optimal modality-specific representations. Proposition 3. If MNI is attainable for random variables X 1 and X 2 , arg maxZ 1 -X 1 -X 2 I(Z 1 ; X 1 |X 2 ) = arg max Z 1 -X 1 -X 2 I(Z 1 , Ẑ2 * c ; X 1 )where Ẑ2 * c is the representation based on X 2 that satisfies MNI, i.e.I( Ẑ2 * c ; X 1 ) = I( Ẑ2 * c ; X 2 ) = I(X 1 ; X 2 ). Proof. Based on the graphical model, Ẑ2 c ⊥ ⊥ X 1 |X 2 , Z 1 , we have I( Ẑ2 c ; X 1 |Z 1 , X 2 ) = 0. Thus, I(Z 1 , X 2 ; X 1 ) = I(Z 1 , X 2 ; X 1 ) + I( Ẑ2 c ; X 1 |Z 1 , X 2 ) (11) = I(Z 1 , X 2 , Ẑ2 c ; X 1 ) = I(Z 1 , Ẑ2 c ; X 1 ) + I(X 1 ; X 2 |Z 1 , Ẑ2 c ) Meanwhile, I(X 1 ; X 2 | Ẑ2 c ) -I(X 1 ; X 2 |Z 1 , Ẑ2 c ) = I(X 2 ; Z 1 | Ẑ2 c ) -I(X 2 ; Z 1 |X 1 , Ẑ2 c ) = I(X 2 ; Z 1 | Ẑ2 c )where the second equality is due to the conditional independenceEQUATIONwhere the first equality is due to the Markov relationshipX 1 ↔ X 2 → Ẑ2 * c . Then we have -I(X 1 ; X 2 |Z 1 , Ẑ2 * c ) = I(X 2 ; Z 1 | Ẑ2 * c ). Due to the non-negativity of conditional mutual information, I(X 1 ; X 2 |Z 1 , Ẑ2 * c ) = I(X 2 ; Z 1 | Ẑ2 * c ) = 0 Based on formula 11 and 13, I(Z 1 , X 2 ; X 1 ) = I(Z 1 , Ẑ2 * c ; X 1 ) Since I(Z 1 ; X 1 |X 2 ) = I(Z 1 , X 2 ; X 1 ) -I(X 1 ; X 2 ) and I(X 1 ; X 2 ) is a constant value irrelevant to Z 1 , maximizing I(Z 1 ; X 1 |X 2 ) is equivalent to maximizing I(Z 1 , X 2 ; X 1 ) = I(Z 1 , Ẑ2 * c ; X 1 ).When MNI is unattainable, as established in Proposition 4, such substitution yields an objective that is nearly equivalent, subject to the value of δ c that corresponds to the β used in step 1 optimization. Therefore, optimizing L 1 s leads to nearly optimal modality-specific representations.Proposition 4. For random variables X 1 and X 2 ,0 ≤ I(Z 1 , X 2 ; X 1 ) -I(Z 1 , Ẑ2 * c ; X 1 ) ≤ δ cwhere Ẑ2 * c is the optimal representation based on X 2 with respect to δ c as defined in Equation 1, i.e. Ẑ2 * c = arg min Z 2 I(Z 2 ; X 2 |X 1 ), s.t. I(X 1 ; X 2 ) -I(Z 2 ; X 1 ) ≤ δ c .Proof. According to formula 11 and 12,In this section, we explore the conditions for both MNI attainable and unattainable cases. Here, we use X and Y to represent the two modalities, rather than X 1 and X 2 in other sections. Although prior works have not provided a comprehensive necessary and sufficient condition for the attainability of MNI, and such analysis is beyond the scope of this paper, we provide several sufficient conditions for both MNI attainable and unattainable scenarios.We first present the sufficient conditions under which MNI is attainable. As outlined in Proposition 5, Proposition 6, and Proposition 7, MNI is attainable when the relationships between X and Y are either entirely deterministic or completely independent across each sub-domain. While the deterministic mapping might hold true when Y is the data label, it generally does not hold when X and Y are high-dimensional data modalities. In such cases, a deterministic relationship between the two modalities would imply that one can be fully inferred from the other, leaving no room for modality-specific information in one of the modalities.Proposition 5. For random variables X, Y , and representation Z derived from X (i.e., the Markov chain Z ← X ↔ Y holds), a sufficient condition for MNI to be attainable is that X → Y mapping is deterministic [7] .Proof. Denote Y = f (X) as the deterministic X → Y mapping. If the encoder is powerful enough, it can learn to reproduce the deterministic functionf , i.e. Z = f (X) = Y . Thus I(X; Z) = I(X; Y ) = H(Y ) = I(Y ; Z).While Fischer [7] identifies this as a necessary condition for attainable MNI, there are scenarios where X → Y is not deterministic, yet MNI is still attainable. We provide additional sufficient conditions for these more general cases in Proposition 6 and Proposition 7.Proposition 6. For random variables X, Y , and representation Z derived from X (i.e., the Markov chain Z ← X ↔ Y holds), a sufficient condition for MNI to be attainable is that Y → X mapping is deterministic. The condition is∀ X s × Y s , X ⊥ ⊥ Y |{(X, Y ) ∈ X s × Y s }, which indicates that ∀ X s × Y s , ∀ x, y ∈ X s × Y s , p(y|x) p(Y s |X s ) = p(y) p(Y s )For any y ∈ Y, choose Y s = {y}, and X s = {x : p(x|y) > 0}, we have p(y|x) = p(Ys|Xs) p(Ys) p(y), ∀ x ∈ X s . Thus, ∀ x 1 , x 2 sampled from p(X|Y = y), we have p(y|x1) = p(y|x2). Therefore, p(Y = y|X) is fully determined given the value of y and is irrelevant to the value of X. Then we show that I(Z; X) = I(Z; Y ).It is easy to see that deterministic mapping between X and Y is a special case of this condition.Regarding the sufficient conditions for MNI being unattainable, as outlined in Lemma 6 of Gilad-Bachrach et al. [8] , when variables X, Y have full support, i.e. p(x, y) > 0, ∀ x, y, MNI is unattainable.Definition 1. Let V and W be Banach spaces, Ω an open set in V and F a function that maps Ω into W . Then the Gateaux derivative of F at x ∈ Ω in the direction h ∈ V is defined asdF (x; h) = lim ϵ→0 F (x + ϵh) -F (x) ϵ = d dϵ F (x + ϵh) ϵ=0provided that this limit exists for all h ∈ VIn this section, we use X and Y to represent the two modalities, rather than X 1 and X 2 in other sections. In our setting, the domain is the Banach space corresponding to the product space representing the set of pairs of joint probability distributions i.e. Z × Y. The functionalF = I(Z; Y ) : Ω → R,where Ω is the product space representing the set of pairs of joint probability distributions for which this mutual information is defined. Following the Markov structure in our graphical model, we have (as used in [24] ) doesn't exist at the MNI point.I(Z; Y ) =Proof. Denote the MNI point between X and Y as p where I(X; Y ) = I(Z; Y ) = I(Z; X). When the MNI point exists, the IB curve can be represented as [27, 12] :I(Z; Y ) = I(Z; X), if I(Z; X) ≤ H(Y ) H(Y ), if H(Y ) < I(Z; X) ≤ H(X)or equivalentlyp(z|y) = p(z|x), if I(Z; X) ≤ H(Y ) 1 y=g(z) , if H(Y ) < I(Z; X) ≤ H(X)Based on this characterization, we know that there exist a direction h such that on perturbing p(z|x) along this directions, we either haveI(Z; Y ) = I(Z; X) < H(Y ) or I(Z; Y ) = I(X; Y ) = H(Y ) < I(Z; X). Without loss of generality, assume that perturbing by ϵh, where ϵ < 0 corresponds to the former case and ϵh, where ϵ > 0 to the latter. Consider the right directional derivative of I(Y:Z) at p in the direction h, we haved + F (p; h) = lim ϵ→0 + F (p + ϵh) -F (p) ϵ = 0Since we have p(z|y) = p(z|x) when p is perturbed in the direction by ϵh, where ϵ < 0, we can consider I(Z; Y ) as an identity function of I(Z; X). In such a case, F (p + ϵh 1 ) = H(Y ) + ϵh. As a consequence, perturbing p(z|x) to get p ′ is equivalent to the resulting perturbation in p(z|y) also being p ′ . As a result, we have EQUATION+ (p(z|y) + ϵh)p(y) h p(z|y) + ϵh - 1 p(z) ϵ=0 = h ̸ = 0Clearly, the left and right limits exist but aren't equal, hence proving the non-differentiability.Four terms are involved in DISENTANGLEDSSL, including maximizing the mutual information term I(Z 1 ; X 2 ) and minimizing the conditional mutual information term I(Z 1 ; X 1 |X 2 ) for the shared representations in step 1, as well as maximizing the joint mutual information term I(Z 1 , Ẑ2 * c ; X 1 ) and minimizing the mutual information term I(Z 1 ; Ẑ1 * c ) for the modality-specific representations in step 2. We introduce detailed formulations of the tractable training objectives for each of the four terms in this section.For the inferred shared representations, we model the distributions Ẑ1 c ∼ p(•|X 1 ) and Ẑ2 c ∼ p(•|X 2 ) with neural network encoders. Following the common practice in Radford et al. [25] , we use the InfoNCE objective [23] as an estimation of the mutual information term I(Z 1 ; X 2 ) in L 1 c .L c InfoNCE = E z 1 ,z 2+ ,{z 2- i } N i=1 -log exp(z 1 ⊤ z 2+ /τ ) exp(z 1 ⊤ z 2+ /τ ) + N i=1 exp(z 1 ⊤ z 2- i /τ) where τ is the temperature hyperparameter, z 2+ is the representation of the positive sample corresponding to the joint distribution p(X 1 , X 2 ), and {z 2- i } N i=1 are representations of N negative samples from the marginal distribution p(X 2 ).We implement the conditional mutual information term I(Z 1 ; X 1 |X 2 ) in L 1 c using an upper bound developed in Federici et al. [6] , i.e. I(Z1 ; X 1 |X 2 ) ≤ D KL (p(Z 1 |X 1 )||p(Z 2 |X 2 )).While the conditional distributions of representations are modeled as the Gaussian distribution in Federici et al. [6] , we instead use the von Mises-Fisher (vMF) distribution for p(Z 1 |X 1 ) and p(Z 2 |X 2 ) to better align with the InfoNCE objective where the representations lie on the sphere space. Specifically,Ẑ1 c ∼ vMF(µ(X 1 ), κ), Ẑ2 c ∼ vMF(µ(X 2 ), κ)where κ is a hyperparameter controlling for the uncertainty of the representations. Leveraging the formulation of the KL divergence between two vMF distributions, the training objective of L 1 c is to maximize:L 1 c = -L c InfoNCE + β • E x 1 ,x 2 µ(x 1 ) ⊤ µ(x 2 )Note that this objective establish connections with the alignment versus uniformity framework discussed in Wang and Isola [37] , where the conditional information bottleneck constraint corresponds to a higher weight on the alignment term, in contrast to the uniformity term.The inferred modality-specific representations are encoded as functionsẐ1 s ∼ p(•|X 1 , Ẑ1 c ) and Ẑ2 s ∼ p(•|X 2 , Ẑ2 c) with deterministic encoders, that takes both data observations and the shared representations learned in step 1 as input to account for the dependence structure illustrated in Figure 2 . The term I(Z 1 , Ẑ2 * c ; X 1 ) in L 1 s is optimized with the InfoNCE loss, where random augmentations of the data X 1 form the two views. Denote the concatenation of Z 1 and its corresponding Ẑ2 * c as Z1 , the InfoNCE objective has the following formula:L s InfoNCE = E z1 ,z 1+ ,{z 1- i } N i=1 -log exp(z 1⊤ z1+ /τ ) exp(z 1⊤ z1+ /τ ) + N i=1 exp(z 1⊤ z1- i /τ )where z1+ is the representation of the positive sample corresponding to the augmented view of X 1 , and {z 1- i } N i=1 are representations of N negative samples from the marginal distribution p(X 1 ). For the mutual information term between representations, I(Z 1 ; Ẑ1 * c ) in L 1 s , we implement it as an orthogonal loss to encourage the marginal independence between the shared and modality-specific representations, where the marginal distribution is approximated across a training batch.L orthogonal = ||[Z 1 i ] B i=1 ⊤ • [ Ẑ1 * ci ] B i=1 || F where B is the batch size, [Z 1 i ] B i=1 and [ Ẑ1 * ci ] B i=1are the concatenations of all the representations in a mini-batch, and || • || F is the Frobenius norm of the pairwise cosine similarities between each latent dimensions. The training objective of L 1 s in step 2 is to maximize:L 1 s = -L s InfoNCE -λ • L orthogonalEach experiment was conducted on 1 NVIDIA RTX A5000 GPU, each with 24GB of accelerator RAM. All experiments were implemented using the PyTorch deep learning framework.Synthetic data generation. We generate synthetic data based on the graphical model in Figure 2 . We sample d-dimensional true latents Z 1 s , Z 2 s , and Zc independently from N (µ d , Σ 2 d ). Using fixed transformations T1 and T2, we createX 1 = T1 • [Z 1 s , Zc] and X 2 = T2 • [Z 2 s , Zc].To simulate unattainable MNI, we add Gaussian noise to ensure the distribution has full support. Binary labelsY 1 s , Y 2s , and Yc are constructed from the true latents and used to evaluate the information content of learned representations via linear probing accuracy. Denote Ẑc as the combination of the learned shared representations of X 1 and X 2 , i.e. Ẑc = [ Ẑ1 c , Ẑ2 c ]. Ideally, Ẑc should achieve high accuracy on Yc and low on Y 1 s and Y 2 s , while the modality-specific representations Ẑ1 s and Ẑ2 s should show the opposite pattern. Additional details on experimental settings are elaborated below.Experimental details. We generate synthetic data X 1 and X 2 based on the graphical model in Figure 2 , with the dimensionality of 100 and dataset size being 90,000. To be specific, we sample 50-dimensional true latents Z 1 s , Z 2 s , and Zc independently from N (050, 0.5 × I50). Then we sample the transformation weights T1 and T2 from uniform distribution Uniform(-1, 1), and generateX 1 = T1 • [Z 1 s , Zc] and X 2 = T2 • [Z 2 s , Zc].We randomly split 80% data into the training set and the rest into the test set. To simulate unattainable MNI, we add Gaussian noise and random dropout during training to ensure the distribution has full support. We use a 3-layer multi-layer perceptron (MLP) with a hidden dimension of 512 as encoders for all methods. For DMVAE, we employ MLPs with the same architecture as decoders. We report the average linear probing accuracy on the test set over 3 random seeds.We run all the methods on the synthetic data with combinations of different hyperparameter values. For DISENTANGLEDSSL, we use β ∈ {0.0, 0.001, 0.01, 0.1, 0.5, 1.0, 5.0, 10.0, 50.0, 100.0, 300.0, 500.0, 1000.0} and λ ∈ {0.0, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0}.For JointOpt, hyperparameter a controls the joint mutual information terms I( Ẑ1 s , Ẑ2 c ; X 1 ) andEQUATIONc ; X 2 ), and λ adjusts the mutual information term between representations, i.e.Ẑ1 * c , Ẑ2 * c , Ẑ1 * s , Ẑ2 * s = arg max Ẑ1 c , Ẑ2 c , Ẑ1 s , Ẑ2 s I( Ẑ1 c ; X 2 ) + I( Ẑ2 c ; X 1 ) + a • (I( Ẑ1 s , Ẑ2 c ; X 1 ) + I( Ẑ2 s , Ẑ1 c ; X 2 )) -λ • (I(Z 1 c ; Z 1 s ) + I(Z 2 c ; Z 2 s ))We use a ∈ {0.01, 0.1, 1.0, 10.0, 100.0, 1000.0} and λ ∈ {0.0, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0}.For FOCAL, we tune the hyperparameters a and λ, defined similarly to JointOpt, where a controls the terms I( Ẑ1 s ; X 1 ) and I( Ẑ2 s ; X 2 ) and λ adjusts the orthogonal loss between shared and specific representations. We use the same set of a and λ as that in JointOpt.For DMVAE, we tune λ which denotes the weight of the KL divergence term in contrast to the reconstruction loss. We use λ ∈ {10 -7 , 10 -6 , 10 -5 , 10 -4 , 10 -3 , 10 -2 }.Results overview. We assess the performance of the learned shared and modality-specific representations for different values of β and λ, as shown in Figure 6 . For comparison, we also evaluate JointOpt, DMVAE, and FOCAL across different hyperparameter settings. Specifically, for JointOpt, we vary both a and λ, where a controls the joint mutual information terms I( Ẑ1 s , Ẑ2 c ; X 1 ) and I( Ẑ2s , Ẑ1 c ; X 2 ),and λ adjusts the mutual information term between representations, similar to DISENTANGLEDSSL. Given the shared representations Ẑ1 c and Ẑ2 c learned in step 1 for a fixed β, we then learn the corresponding modality-specific representations Ẑ1 s and Ẑ2 s with varying λ. Figure 6b shows the performance of DISENTANGLEDSSL in contrast to other baseline methods, where dots are connected according to a descending order of their corresponding λ values 5 . The ideal modality-specific representation Ẑ1s should maximize unique information from X 1 , shown by high accuracy on Y 1 s , while minimizing shared information with X 2 , indicated by low accuracy on Y c . Therefore, a bottomright point is preferred in Figure 6b . As illustrated in Figure 6b , DISENTANGLEDSSL outperforms all other methods across various hyperparameter settings, especially JointOpt, demonstrating the effectiveness of the stepwise optimization procedure. Results on Ẑ2 s are provided in Figure 8 . Additional results. We provide a comparison of the performance of the shared representation with baseline methods in Figure 7 . Denote Ẑc as the concatenation of the learned shared representations of X 1 and X 2 , i.e. Ẑc = [ Ẑ1 c , Ẑ2 c ]. The ideal Ẑc should maximize the shared information between X 1 and X 2 , shown by high accuracy on Y c , while minimizing unique information of X 1 and X 2 , indicated by low accuracy on Y 1 s and Y 2 s . Therefore, a top-left point is preferred in Figure 7 . As shown in Figure 7 , DISENTANGLEDSSL consistently outperforms other methods across various hyperparameter settings, showcasing its ability to effectively capture shared information. The only exception occurs in Figure 7a when β is set very high and the accuracy on Y 1 s to drop to around 0.50 (equivalent to random guessing, indicating no information of Ẑc on modality-specific features). In this scenario, DMVAE performs better. This happens because high values of β cause the decoder-free contrastive objectives to collapse, with most representations converging to nearly the same point, a commonly-known issue in contrastive self-supervised learning [23] . For the modality-specific representations, as a supplement to Figure 6b , we provide the complete results for both representations Ẑ1 s and Ẑ2 s on a full set of hyperparameters in Figure 8 . DISENTAN-GLEDSSL outperforms all other methods across various hyperparameter settings. Analysis for different levels of entanglement. We further examine the trade-off between expressivity and redundancy for the learned shared representations on synthetic data with varying levels of multimodal entanglement. In this scenario, some dimensions align across modalities with MNI attainable, while others remain entangled with MNI unattainable. Specifically, we split the 50dimensional true shared latent Z c into two parts: the first 35 dimensions, denoted as Z mix c , and the last 15 dimensions, denoted as Z pure c . We then generate 100-dimensional observations X 1 and X 2 , where the first 85 dimensions are generated under the same procedure as before, i.e.X 1 mix = T 1 • [Z 1 s , Z mix c ] and X 2 mix = T 2 • [Z 2 s , Z mix c ]followed by adding Gaussian noise and random dropout, while the last 15 dimensions are directly Z pure c . We present the results on the synthetic data with a mixed level of entanglement in Figure 9 . Figure 9a shows the test accuracy of Ẑ1 c on Y c , Y 1 s , and Y 2 s on the left axis, alongside the MLP weight ratio on X 1 and X 2 on the right axis, for varying values of β. To compute the MLP weight ratio, we first extract the diagonal of the inner product of MLP encoder's first layer weight matrix, then calculate the ratio between the average of the last 15 "pure" dimensions and the first 85 "mixed" dimensions. This ratio indicates how much attention the encoder gives to the "pure" versus "mixed" dimensions, with higher values signifying greater focus on the "pure" dimensions. Figure 9b shows the corresponding test accuracy on Y c versus Y 2 s in line plot. Using such data with mixed entanglement levels, DISENTANGLEDSSL demonstrates a clear pattern where the learned information plateaus at certain β values. As illustrated in Figure 9 , the MLP weight ratio initially rises sharply from around 20 to nearly 80, then drops to 1 when β becomes very large, indicating the collapse of the learned representations. With a large value of β (while before the model collapses, i.e. β ≈ 10), the encoders focus mainly on the "pure" dimensions. This is because a stronger information bottleneck constraint discourages the extraction of shared components from the "mixed" dimensions, which inevitably include modality-specific information due to unattainable MNI, and favors the "pure" shared components with no extra cost. Experimental details. We follow the same dataset splitting, and utilize the same encoder architecture and pre-extracted features as [17] . All models use representations (or concatenation of them, for FactorCL-proj, FOCAL, JointOpt and DISENTANGLEDSSL) with the same dimensionality. We set the latent dimension to 300 across all datasets, except for MUSTARD, where it is set to 100 due to the smaller dataset size, and for MIMIC, where we use 360 to align with the output dimension of the GRU encoders. For FactorCL, we use their default hyperparameter settings. For other methods, hyperparameters are tuned based on validation set performance. For DISENTANGLEDSSL, we use β = 1.0 and λ = 10 -3 for all datasets, except for MOSI where β = 0.01. For FOCAL and JointOpt, we use a = 1 and λ = 10 -3 across all datasets. We report the mean and standard deviation of the linear probing accuracy on prediction labels from the test set over 3 random seeds.Additional results. As a complement to Table 1 , we present results of FOCAL and JointOpt for the learned shared and modality-specific representations, both combined (i.e. "both") and separately in Table 4 . Although they exhibit a similar trend as DISENTANGLEDSSL, where shared representations are more important for MOSI and specific representations are crucial for MUSTARD, the distinction is less pronounced. Moreover, the overall performance is inferior to that of DISENTANGLEDSSL, indicating that DISENTANGLEDSSL achieves a better balance between coverage and disentanglement in the learned representations.Experimental details. Following the setup in Wang et al. [36] , we utilize Mol2vec [11] to featurize the molecular structures into 300-dimensional feature vectors. For both molecular structures and phenotypes, we employ 3-layer MLP encoders with a hidden dimension of 2560. The dimensionality for shared and modality-specific representations is set to 32 across all methods and datasets. We tune β based on validation set performance, with β = 5.0 for RXRX19a and β = 1.0 for LINCS, and set λ = 0.01 for both datasets. For FOCAL and JointOpt, we set a = 1.0 and λ = 0.01. For DMVAE, we set the coefficient of the KL divergence term as 10 -5 . In addition, as in Lee and Pavlovic [14] , we introduce an InfoNCE loss for the shared representations to DMVAE in the high-content drug screening experiments to enhance its retrieving accuracy. We report the mean and standard deviation of all results on the test set over 3 random seeds.Counterfactual generation. To further evaluate the learned modality-specific representations, we use different combinations of the shared and modality-specific representations to generate counterfactual samples, i.e. predicting phenotype of a drug on a different cell with its molecule shared latent but the phenotype specific latent of a different cell perturbed by other drugs. Since the factual observations of counterfactual generation are unavailable, we measure the performance distributionalwise, introducing the difference in Frechet distance (Diff-FD) as a metric, i.e. Diff-FD-c measuring the gain of using the correct batch of molecules, thus the information of the shared latent; Diff-FD-s measures the gain of using the correct batch of cells, thus the information of the specific latent.To be specific, we train decoders with the factual combination of the shared latents from molecular structures and the modality-specific latents from phenotypes on the training set. We use "val recon" to denote the samples generated using the representations attained from validation set molecules and validation set phenotypes as input. Similarly, "test recon" refers to the samples generated using test set molecules paired with test set phenotypes, while "counterfactual" represents those generated using test set molecules paired with validation set phenotypes. Formally, Diff-FD-c is defined as FD(val recon; test recon) -FD(counterfactual; test recon), and Diff-FD-s as FD(val recon; test recon) -FD(counterfactual; val recon). Note that a non-informative specific latent can lead to very high Diff-FD-c, and an overly informative modality-specific latent can lead to very high Diff-FD-s, thus we take both metrics into consideration together. As shown in Table 5 , DISENTANGLEDSSL outperforms JointOpt in both metrics and surpasses FOCAL in Diff-FD-s. While FOCAL shows a high Diff-FD-s, it learns overly informative modalityspecific latents, which contains significant shared information, as highlighted in the simulation study results (i.e. the modality-specific representations of FOCAL have high accuracy on Y c in Figure 8 ), leading to a low value of Diff-FD-c. In contrast, DISENTANGLEDSSL demonstrate better effectiveness in capturing both shared and modality-specific information, while maximizing the separation between them, as indicated by its high values in both metrics.When the IB curve is not strictly concave (i.e. partially linear), the properties hold except for the linear sections. In these cases, modifications like squared IB[12] can be used to achieve bijection mapping.For FOCAL, we tune the hyperparameters a and λ, defined similarly to JointOpt, where a controls the terms I( Ẑ1 s ; X 1 ) and I( Ẑ2 s ; X 2 ) and λ adjusts the orthogonal loss between shared and specific representations. For DMVAE, we tune λ which denotes the weight of the KL divergence term. We show the best-performing results of FOCAL and DMVAE across hyperparameters in Figure6b, with full results available in Figure8. 
 
 Appendix:  


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Review
main_review: main_review: ## Summary
This paper introduces DISENTANGLEDSSL, a self-supervised learning method designed to achieve controlled disentanglement of multimodal data. The method incorporates information-theoretic criteria and a two-step optimization strategy to learn disentangled representations even when the Minimum Necessary Information (MNI) point is unattainable. Empirical results on multiple datasets demonstrate that DISENTANGLEDSSL outperforms existing baselines in terms of disentanglement quality and downstream task performance.

## Strengths
1. The proposed method provides a unique approach to disentangling shared and modality-specific information in multimodal data.
2. The paper offers a theoretical foundation by leveraging information theory, particularly the information bottleneck framework.
3. Extensive experiments on several datasets shows its effectiveness.

## Weakness
1. It would be better to add feature distribution visualizations, such as TSNE.
title: title: Technically solid paper with conclusive results
main_review: main_review: ### **Summary**

This paper addresses the challenge of disentanglement between modalities for self-supervised learning regimes by introducing DisentangledSSL that learns disentangled representations across modalities. DSSL outperforms on V&L prediction and phenotyping tasks in biology. The paper builds on the concept of minimum necessary information and leverages this information-theoretic metric to ensure disentanglement between modality-specific and shared representations. 

### Strengths

- The motivation of the paper is clear and  concise
- The method of the paper is well-crafted, even if sometimes disjoint with the parts that were moved to the appendix.
- The results are conclusive and support the claims made in the paper
- Assuming that runtime of the two-step training process is similar to the benchmarks, this would be a feasible alternative to CLIP and other contrastive SSL methods across domains.

### Areas for Development/Questions

- Reading the paper, it is unclear what the “true latent” (L42) is. Is this supposed to be a conceptual explanation?
- The method section is relatively clear, although in parts disjoint. It would be helpful to indicate which parts are just establishing information and which are a novelty of the paper to further improve the clarity of the contributions. Additionally, it would strengthen the paper to link the appendices in this section for further explanations.
- Making the figure captions self-contained and linked to the main body would improve clarity.
- It’s very obvious that some sections from a full paper submission were moved down to the abstract. However, this needs to be done with care - introducing important abbreviations like IB (L303) should be in the main body. This reduces the readability of this paper.
- How does the two-step learning process compare to the benchmark in terms of wall time?
title: title: Good investgation of dientanglement of multimodal data
main_review: main_review: # Summary
This paper proposed an interesting view to handle multimodal representation learning that disentangles shared and modality-specific information using an information-theoretic framework. It introduces a step-by-step optimization strategy to learn these representations effectively, even when the Minimum Necessary Information (MNI) point is not attainable. 

# Strength
1. The proposed self-supervised method operates effectively even when the Minimum Necessary Information (MNI) point is not attainable, addressing a common limitation in multimodal learning. It is interesting to show the comparison of the IB curves of non-MNI and MNI.
2. The approach is grounded in information theory, utilizing the Information Bottleneck (IB) principle and its variants. By defining clear information-theoretic criteria, it ensures that both shared and specific representations are informative and disentangled. The paper provides rigorous analysis and formal proofs (in the appendices). 
3. Step-by-step optimization makes sense to me. Adopting a sequential optimization strategy to disentangle shared and modality-specific information is more appropriate for such a complex task.

# Weakness
1. Limited discussion on scalability. Although the paper demonstrates empirical success on datasets like MultiBench and high-content drug screening, there is limited discussion on how well the model scales with increasing data complexity or dimensionality.
2. The experimental results seem to not be promising compared with other baselines.