Learning Optimal Multimodal Information Bottleneck Representations
Authors: Qilong Wu
Yiyang Shao
Jun Wang
Xiaobo Sun
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Learning; Information Bottleneck
Primary area: general_machine_learning->representation_learning

Abstract: Leveraging high-quality joint representations from multimodal data can greatly enhance model performance in various machine-learning based applications. Recent multimodal learning methods, based on the multimodal information bottleneck (MIB) principle, aim to generate optimal MIB with maximal task-relevant information and minimal superfluous information via regularization. However, these methods often set regularization weights in an *ad hoc* manner and overlook imbalanced task-relevant information across modalities, limiting their ability to achieve optimal MIB. To address this gap, we propose a novel multimodal learning framework, Optimal Multimodal Information Bottleneck (OMIB), whose optimization objective guarantees the achievability of optimal MIB by setting the regularization weight within a theoretically derived bound. OMIB further addresses imbalanced task-relevant information by dynamically adjusting regularization weights per modality, ensuring the inclusion of all task-relevant information. Moreover, we establish a solid information-theoretical foundation for OMIB's optimization and implement it under the variational approximation framework for computational efficiency. Finally, we empirically validate the OMIB’s theoretical properties on synthetic data and demonstrate its superiority over the state-of-the-art benchmark methods in various downstream tasks.

Learning Optimal Multimodal Information Bottleneck Representations
Qilong Wu
Yiyang Shao?
Jun Wang *3
Xiaobo Sun
Abstract
V/
1/2
Dowastream Tasks
Optimal
Enution Recegation
Leveraging high-quality joint representations from multimodal data can greatly enhance model performance in various machine-learning based applications. Recent multimodal learning meth- ods, based on the multimodal information bot- tleneck (MIB) principle, aim to generate opti- mal MIB with maximal task-relevant information and minimal superfluous information via regu- larization. However, these methods often set ad hoc regularization weights and overlook imbal- anced task-relevant information across modali- ties, limiting their ability achieve optimal MIB. To address this gap, we propose novel multi- modal learning framework, Optimal Multimodal Information Bottleneck (OMIB), whose optimiza- tion objective guarantees the achievability of op- timal MIB by setting the regularization weight within theoretically derived bound. OMIB further addresses imbalanced task-relevant infor- mation by dynamically adjusting regularization weights per modality, promoting the inclusion of all task-relevant information. Moreover, we es- tablish a solid information-thhorrticall foundation for OMI''s s optimization and implement it un- der the variational approximation framework for computational efficiency. Finally, empirically validate the OMIB's 's theoretical properties syn- thetic data and demonstrate its superiority over the of-e------e-rrtt benchmark methods in various downstream tasks.
Happy
bo
Sentinent Analys
b
a1
b2
Soure
a.
Co
Anemalosa
Desection
a22
Figure 1: a) Venn diagrams for two data modalities (U1 and 02). The gridded area represents consistent information, while the non-gridded area denotes modality mation. Task-relevant information is highlighted in green, whereas superfluous information is shown in blue. b) An optimal MIB should exclusively contain task-relevant, non- superfluous information (i.e., ag, a1 and a2) to be utilized in downstream tasks for enhanced performance.
infor-
but each inspects only single, distinct part (e.g., tusk, leg). Consequently, they deliver conflicting descriptions, as their judgments are based solely on the part they touch.
In the context of machine learning, this parable underscores the significance of multimodal learning, which integrates and leverages multimodal data (akin the elephant's 's body parts) to grasp a holistic understanding, thereby enhancing inference and prediction accuracy. In multimodal learning, unimodal features are extracted from each modalities and fused with various fusion strategies, such as bensor-based (Zadeh et al., 2017: Liu al., 2018), attention- (Guo et al., 2020; Xiao et al., 2020; Zhang al., 2023), and graph- based (Arun et al., 2022; Huang al., 2021), to generate multimodal embeddings. However, major limitation of these methods is their potential to include superfluous and redundant information from each modality, increasing em- bedding complexity and the risk of overfitting (Mai et al., 2023; Wan et al., 2021).
1. Introduction
In the parable "Binnd men and an elephant", a group of blind men attempts to perceive the elephant's shape through touch,
'Equal contribution 'School of Statistics and Mathematics, Zhongnan University of Economics and Law "School of Finance, Zhongnan University of Economics and Law FiWudao "School of Medicine, Department of Human Geneties, Emory University. Correspondence to: Xiaobo Sun <xsun28 @ emory.edu>.
From an information theory perspective, comprehensive multimodal learning method should account for five fac- tors: consistency (Tian et al., 2021), specificity (Liu et al., 2024), complementarity (Wan et al., 2021), sufficiency (Federici et al., 2020), and conciseness (a.k.a. nonredun- dacy) (Wang et al., 2019). As illustrated in Figure la, on the input side, consistency describes information shared
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Optimal Multimodal Information Bottleneck Representations
Seicuadt Caualan Neead
where E,, F, I, and O represent the modality en- coder, multimodal fusion function, mutual information func- tion, and optimization objective, respectively. Vi, Vi,22, 2; and y denote the raw data of the i-th modality, its extracted rep- resentation, the MIB, and task labels. Particularly, 19 serves as the regularization parameter for constraining superfluous information between the MIB and the i-th modality.
Sul TRB
Exs
Dwnc
Ast Modality (ro)
ONE Black
VAS
Dic
Hewr
1-th (ore
Hour
Despite their promising performance, these methods face three major limitations. First, the achievability of optimal MIB is not guaranteed. Since the regularization parame- ters (c.g., As in Equation (1)) control the trade-off between sufficiency and conciseness, their values are critical for optimizing MIB (Tian et al., 2021). If the value is too small, superfluous information may be retained, leading to suboptimal MIB. Conversely, if too large, task-relevant information may be excluded due to an overemphasis on conciseness, compromising MIB's sufficiency. However, existing MIB methods determine these parameter values in an ad hoc manner, limiting their ability to achieve an optimal MIB. Second, an ideal MIB method should dynami- cally adjust regularization weights based on the remaining task-relevant information in each modality. Specifically, a modality should receive a lower regularization weight if a significant portion of its task-relevant information is left out of the MIB, and vice versa. However, existing MIB methods typically assign fixed, ad hoc regularization weights to each modality during training. When task-relevant information is imbalanced across modalities, some modalities may contain minor but crucial task relevae information (e.g.. 02 in U2 in Figure 1). If such a modality is assigned an excessively large regularization weight, its task-relevant information may be inadvertently excluded from the MIB (Fan et al., 2023). Fi- nally, these methods lack theoretical comprehensiveness, as they either fail to incorporate all five aforementioned fac- tors into the optimization objective or do not acknowledge their distinct roles in guiding optimization. For instance, the study in (Tian et al., 2021) overlooks complementarity, while CMIB-Nets (Wan et al., 2021) does not account for consistent, superfluous information. Additionally, in the theoretical analyses of methods such as (Fang et al., 2024; Wan et al., 2021), the two types of task relevant informa- tion consistent (e.g.. ag in Figure 1) and modality-speciiff (e.g., 01, a2) are not distinguished, despite their differing impacts on the optimization objective.
Dro
Arae, A TAB
Suckarde Gatoian Nicss
OOL7 Bork lear Sea Ailler Ilur slae
Figure 2: OMIB Framework. Here, 'C' represents the con- catenation operation. For the definitions of other notations, refer to the Section 4 and Table 1.
across input modalities (gridded area), while specificity refers to the information unique to individual modalities (non-gridded area). Both consistent and modality-specific information may contain task-relevant (green area) or su- perfluous (gray area) components. Complementarity per- tains to modality-specific, task-relevant information (a, and az), enabling multimodal embeddings to surpass unimodal ones in downstream tasks. On the output side, an opti- mal multimodal embedding (as shown in Figure 1b and Definition 5.4 below) must be sufficient, capturing max- imal task-relevant information- both consistent (ag) and complementary (41, (2)
ag)oscross across modalities. Meanwhile, it should be concise, minimizing both cross-modality (bg) and modality-specifft superfluous information (b,, (2) to reduce complexity. This optimal multimodal embedding can then be applied to various downstream tasks, such multimodal sentiment analysis (Mai et al., 2023) and pathological tis- sue detection using histology and gene expression data (Xu et al., 2024b)), for enhanced performance.
To this end, multimodal learning methods based on the Mul- timodal Information Bottleneck (MIB) principle have been developed, which generally follow a common paradigm: -ppelit representations are extracted and fused into MIB via deep networks. The MIB is then optimized to balance two objectives: maximizing mutual information between the embeddings and task-relevant labels for suf- ficiency; and minimizing mutual information between the embeddings and the raw input to purge superfluous informa- tion and ensure conciseness (Wang et al., 2019; Wan et al., 2021; Zhang al., 2023; Fang al., 2024). This process is formalized as:
To address these issues, we propose novel MIB-based mul- timodal learning framework, termed Optimal Multimodal Information Bottleneck (OMIB) to learn task-relevant optimal MIB representations from multi-modal data for enhanced downstream task performance. OMIB features theoretically grounded optimization objectives, explicitly linked the dynamics of all five information-thherrtiial factors during optimization, ensuring holistic and rigor- ous optimization framework. As illustrated in Figure 2,
x; Ev()),, F(I1, I2,..,,
(1)
O 2, v, := -max [3y; 2) - Σ (B((2)))
2

---

Learning Optimal Multimodal Information Bottleneck Representations
OMIB comprises two components, including task relevance branches (TRBs) that extract sufficient representations from individual modalities, and an optimal multimodal fusion block (OMF), where condallty-speciffe representations are fused by a cross-attention network (CAN) into MIB and optimized using a computationally efficient variational ap- proximation (Alemi et al., 2017). Adhering to the MIB principle, the OMF block maximizes sufficiency while min- imizing redundancy in the MIB. Particularly, by setting the redundancy regularization parameter in OMIB's 's objective function within a theoretically derived bound, OMIB guar- antees the achievability of optimal MIB upon convergence of the OMF block training. Furthermore, our approach dy- namically refines regularization weights per modality as per the distribution of their remaining task-relevant information. In summary, our contributions include:
Tensor-based, modeling high-order interactions (Chen et al., 2020; Zadeh al., 2017), (4) Subspace- based, projecting modalities into shared latent spaces (Yao et al., 2017; Zhou et al., 2021), and (5) Graph-based, representing modalities as graph nodes and edges (Parisot et al., 2018; Cao et al,, 2021). In addition, recent studies also discuss the issue of modality imbalance, where strong modalities tend to domi- nate the learning process, while weak modalities are often suppressed (Peng al., 2022; Zhang al., 2024). Though effective, these methods typically fail to account for super- fluous information and thus are prone to overfitting and sensitive to noisy modalities, limiting their practical robust- ness (Fang et al., 2024). MIB addresses these challenges by preserving task-relevant information while minimizing redundant content in the generated multimodal representa- tions.
We propose OMIB, novel framework for learning optimal MIB representations from multimodal data, with an explicit solution to address imbalanced task- relevant information across modalities.
2.2. Multimodal Information Bottleneck
The Information Bottleneck (IB) framework (Tishby et al., 2000) provides a principled approach for learning com- pressed, task-relevant representations. It was first applied to deep learning by (Tishby & Zaslavsky, 2015) and later extended through the Variational Information Bottleneck (VIB) (Alemi et al., 2017), which employs stochastic vari- ational inference for efficient approximations. Recently, has been adapted to more complex settings, such as multi-view (Wang et al., 2019; Federici et al., 2020) and multimodal learning (Tian et al., 2021). For example, L- MIB, E-MIB, and C-MIB (Mai et al., 2023) aim to learn effective multimodal representations by maximizing task- relevant mutual information, eliminating redundancy, and filtering noise, while exploring how MIB performs at dif- ferent fusion stages. Secondly, MMIB -Zaang (Zhang et al., 2022) improves multimodal representation learning by im- posing mutual information constraints between modality pairs, enhancing the model's 's ability to retain relevant infor- mation. Additionally, DISENTANGLEDSSL (Wang et al., 2024) relaxes the restrictions on achieving minimal suffi- cient information, thereby enabling the disentanglement of modality and modality-specific information and en- hancing interpretability. Lastly, DMIB (Fang et al., 2024) filters irrelevant information and noise, employing suffi- ciency loss to preserve task-relevant data, ensuring robust- ness in noisy and redundant environments.
We provide rigorous theoretical foundation that un- derpins OMIB's optimization procedure, establishing a clear connection between its objectives and the five tatnttttiiontttoetiicce factors: sufficiency, consis- tency, redundancy, complementarity, and specificity.
We mathematically derive the conditions for achieving optimal MIB, marking, to our knowledge, the first endeavor in proving its achievability under the MIB principle.
We validate OMIB's effectiveness on synthetic data and demonstrate its superiority over state-of-the-art MIB methods in downstream tasks such as sentiment analysis, emotion recognition, and anomalous tissue detection across diverse real-world datasets.
2. Related Work
2.1. Multimodal Fusion
Multimodal fusion methods can be categorized according to the fusion stage and techniques. Architecturally. fu- sion can happen at three stages: (1) Early fusion, which combines data at the feature level (Snoek et al., 2005), (2) Late fusion, integrating data at the decision level (Morvant et al., 2014), and (3) Middle fusion, where data is fused at intermediate layers allow early layers to specialize in learning unimodal patterns (Nagrani et al., 2021). From the technique perspective, fusion approaches include: (1) Operation- combining features through arithmetic operations (El-Sappagh et al., 2020; Lu et al., 2021), (2) Attention boeetioo-bbseed, using cross modal attention to learn inter- action weights (Schulz et al., 2021; Cai et al., 2023), (3)
However, these methods often rely on ad hoc regulariza- tion weights and overlook the imbalance of task-relevant information across modalities, limiting their ability to fully optimize the MIB framework.
3. Notations
Here, we list the mathematical notations (Table 1) used in this study.
3

---

Learning Optimal Multimodal Information Bottleneck Representations
Table I: Summary of notation.
Main Training. After the warm-up training, the model proceeds to main training, which includes optimal multi- modal fusion (OMF) block in addition to the TRBs. In the OMF, 24, Vi E {1,2}. is used to generate the mean μ E R and variance Σ E k of a Gaussian distribution using a variational autoencoder (V AVA)):
Notation
Description
y
Task-relevant label.
i-th modality.
Z
The sufficient encoding of ih for y.
E
MIB encoding.
N
The total number of observations.
M4. Σ VAE,( AVAE.)
H(*)
The entropy of variable
F(+)
information set inherent to variable (i.e., F(x) = H())) where V represents the parameters of VAE,, For effi- cient training and direct gradient backpropagation, the repa- rameterization trick (Kingma, 2013) is applied to generate Ç E R
1
The mutual information function.
4. Method
4 + 2, x ct, where e; ~ ()..
(6)
To clearly illustrate OMIB's framework, we start with the case of two data modalities (e.g., 1i and ts in Figure 2),
Çi and C2 are fused using a Cross-Attention Network (CAN) (Vaswani al., 2017), whose architecture is detailed in Appendix H:
which can be readily extended to multiple modalities by
adding additional modality branches (see Appendix D.I).
We also provide a rigorous theoretical foundation for our methodology Section 5.
{ - CAN( S2, OAAN),
where & is the post-fusion embedding, which is then passed to a task-relevant prediction head Dec to generate the final prediction ÿ:
Warm-up training. This phase consists of two task rele- vance branches (TRB) corresponding to and U/2 respec- tively. In the i-th TRB, Uy is first encoded into a sufficient representation 2 E Idd for task-relevant labels y:
Dec(E, Dec)
(8)
Meanwhile, replaces the stochastic noise e; in 0; 'ss TRB to fuse with 24, yielding ii for computing LTRB, and updating Enc; and Dec;
2 Ennc(1i 0 Enc:), s.t. T(way);
Ú = Decs ((:.(((z.))
where Enc, is an encoder, Bne. denotes its parameters. To ensure maximal sufficiency of 24 for y, we concatenate with a stochastic Gaussian noise, 6; E & = N(, T), before feeding it to a task-relevant prediction head Dec; (see Appendix H) to yield the predicted output C::
As established in Proposition 5.1, the loss function for up- dating the components OMF (i.e., VAE, CAN, and Dec) to achieve optimal MIB, &, is given by:
i Dec; e:])
(3)
1 N
LOMP
Σ
B E.g |- logg(("
N
Via this step, Enc; is optimized to extract maximal task- relevant information from Us as requires a higher signal- to-noise ratio in 2; to yield accurate prediction from its corrupted version. The loss function for updating Enc, and Dec; is:
(10)
+ 3 (KL p + (W((,,)) I) +r KL | 2 (W I)).
where ,9 is a hyper-parameter constraining redundancy be- tween Ç; and 2;, and is dynamically adjusted weight balancing the regularization of 02 relative to en- The im- plementation of -log q (ylE) is task-specific, as previously stated. K LL(p (2 [[V / (0. I)] represents the KL-divergence between and the standard normal distribution. As shown in Proposition 5.2, is explicitly computed during training as:
LTRA.
Ev,l- log p()) 24, eq]
N
(4)
log logp(" N
n=1
Note that the implementation of log Log,(ū,/2s, e,) is task- specific. For classification tasks, is implemented as C'((ill)). where CE is the cross-entropy function; for SVDD-based anomaly detection (Ruff et al., 2018). it is | -cll where is the unit sphere center of normal observa- tions (see Appendix H); for regression tasks, is -100. yll- The algorithmic workflow of the warm-up training is de- scribed in Appendix I..
KT(p(0Sk"" ""1"
1-tanh Σ
E., E.g KL 21 (((
N
Furthermore, Proposition 5.7 provides a theoretical upper bound for setting 3, ensuring that our methodology achieves optimal MIB. The algorithmic workflow of the main training procedure detailed in Appendix L..

---

Learning Optimal Multimodal Information Bottleneck Representations
Inference. During inference, the TRBs disabled, and the trained modality-specinfe encoder (Enc;) and OMF gen- erate optimal MIBs for test data to be used in downstream tasks.
Proposition 5.1 (Variational upper bound of OMIB's objective function). The loss function Lour in Equa- tion (10) provides a variational upper bound for optimizing the objective function in Equation (17) and can be explicitly calculated during training.
5. Theoretical Foundation
Proof. See Appendix B.
Due to space constraints, we focus on the theoretical anal- ysis of two data modalities in this section and defer the analysis of multiple data modalities (2 3) to Appendix D.2.
Moreover, when substantial portion of task-relevant infor- mation remains in v2 relative to v1, the value of should be small to encourage incorporating more information from U2 in subsequent training iterations. Simultaneously. r should bounded to prevent over-regularizing information from 22. Thus, can be mathematically expressed as:
5.1. Optimal Information Bottleneck for Multimodal Data with Imbalanced Task-Relevan Information
As proposed in (Alemi al., 2017; Federici et al., 2020; Mai et al., 2023; Wang et al., 2019), the Information Bottleneck (IB) principle aims to optimize two key objectives:
r x vile) [(: 02l5)
r (0,
(18)
(1) mazimize [(y; and (2) minimize [(u;)) (12)
where I(y; uy(E) represents the amount of task-relevar in- formation in 2 not encoded in ç, and u is an upper bound. In this study, is set to 2, it is implemented using lahn function as in Equation (11), which is justified by the following proposition.
where y represents task-relevant labels, v the input data, the IB encoding. The first objective maximizes 2's ex- pressiveness for y, while the second objective enforces z's conciseness. These objectives can be formulated as:
Proposition 5.2 (Explicit formula for r). Equation (11) satisfies Equation (18), providing an explicit formula for computing r during training.
max Tyy 2) s.t. T(u; 2) S Tes
(13)
where Le is the information constraint that limits the amount of retained input information. Introducing a Langrange multiplier 3 > O, the objective function is reformulated as:
Proof. See Appendix B.
In the next section, we establish a theoretical bound for B, ensuring that & attains optimality during the optimization of the objective function in Equation (17).
max 1 (yr 2) (1(1;)).
(14)
For two data modalities, propose modified objective function to account for imbalanced task-relevant informa- tion across modalities:
5.2. Achievability of Optimal Multimodal Information Bottleneck
min ((g) min - -T(E ( rT((; u2)), (15)
Assumption 5.3. As illustrated in Figure 1, given two modalities, e1 and 02, the task-relevant information set {a} consists of three components: ap, 41, and az Specifically, og is shared between both modalities, while and a2 are specific to U1 and 122 respectively. The task-relevant la- bels y are determined by {a}. Moreover, tn and d2 contain modality-specine superfluous information bj and b2, respec- tively, in addition to shared superfluous information by-
where r > 0 is dynamically adjusted parameter control- ling the relative regularization of 02 with respect to 21. In Equation (15), Y can he replaced with z;. To see this point, let in denote the information in o1 that is not encoded in ø1. Then, we have:
v(E: I(5;21,6)) ((5)2)) +
[E: új (a)) = 0: F(E) nP(e1) =0
Definition 5.4 (Optimal multimodal information bottle- neck). Under Assumption 5.3, the optimal MIB, Eopts for t) and 022 satisfies:
= 2((
(16)
F(Copt) (ag. 01, a2):
(19)
Similarly, v2) = I(5; 22). Thus, the objective function in Equation (15) can be rewritten as:
ensuring that Eops encompasses all task-relevant information (ag, aj and a2) while exempting from superfluous informa- tion (bg, b, and b2).
min ((E) min - -T(5:)) + B(T(E: (rrr)++I((g:: rI(E: 22)). (17)
5

---

Learning Optimal Multimodal Information Bottleneck Representations
Case (r))(P)) (P2)
(Fe(P1) Fre
case
Frd(
0.55
0.75
0.70
0.65
0.85
0.60
0.60
M
0.55
0.55
M
0.55
1e:t8
Le:ts
Xeg
16.3
Se1
10:00
Le
Lext
16d
16-
162
161
Se1
11
b1
12
02
b2
Dg
112
0g
6g
1
42
b2
bg
(5,00 > (100)
d1 (100) <dg:(500) (500)
di (3000 dg, (300)
Figure 3: The impact values on classification accuracy on synthetic data, in and e2 represent sample vectors of two modalities, respectively. Fiet()) denotes task-relevant information. "a" sub-vectors denote task-relevant information, while "b" superfiuous information. d1 and d21 denote the dimensions of modality-speciff a1 and ag. My is the computed upper bound.
Lemma 5.5 (Inclusiveness of task-relevant information). Under Assumption 5.3, the objective function in Equa- tion (17) guarantees:
In this study, we set Mu = 3 (((( (1v)) [(mr:0)))
tighter upper bound for
(I+r) [((r))))) given that E (0, 2). and Mi ((((e))+ S M, lower bound for accelerate training. Importantly, both Mr and M,, can be computed priori from the training data using the Mutual Information Neural Estimator (MINE, (Belghazi et al., 2018)) to estimate H-) and [(=-)) (see Appendix E).
F(() 2 {ao, a1, a2),
(20)
provided that B (0, M.). where M..
[[+ [eiwe]]
Proof. See Appendix C
6. Experiment
Note that H(v1) + H(v2) [(11 represents the to- tal information encompassed by the two data modalities. Intuitively, larger total information content requires incor- porating more information from each modality into the MIB. This is achieved by setting a lower Mu, ensuring that all task-relevant information is included in the MIB.
Due to space constraints, defer detailed task-specific experimental settings to Appendix and implementations of network architectures to Appendix H. Detailed descriptions of the benchmark methods and evaluation metrics are pro- vided in Appendix and Appendix K respectively. The best and second- best performing methods in each experiment are bolded and underlined, respectively.
Lemma 5.6 (Exclusiveness of superfluous information). Under Assumption 5.3, the objective function in Equa- tion (17) ensures:
Table 2: Classification accuracy of synthetic features vs. OMIB genB--geeeratdd MIB on simulated datasets.
F(E) C (ao, a1, )22
(21)
Datasets
Imbulanced (SIM-I)
balanced(SIM-II)
Consistent& relevant
0.707
0.686
Proof. See Appendix C
Modality-speciffcc& relevant
0.737
0.744
Unimodal
0.748 / 0.82
0.792 / 0.78
Authentic optimul MIB
0.909
0.908
Proposition 5.7 (Achievability of optimal MIB). Under Assumption 5.3, the optimal MIB Eops is achievable through optimization of Equation (17) with 3 E (0, M.].
Union of two modalities
0.858
0.866
OMIB OMIB-gencrand geBeratcdtt MIB
0.892
0.890
6.1. Datasets
Proof. Lemma 5.5 and Lemma 5.6 jointly demonstrate that
F(E) 2 {an, a a2) and F()) C {ag, a, 02), given 3 E (0, Mu]. Thus, F(E) = {ag. a1, a2). which corresponds to Sopt Definition 5.4. This completes the proof.
To facilitate the analysis of OMIB's performance and validate Proposition 5.7, we simulate three Gaussian- based two-modality dataset, SIM-(I-IIT) for classifica-
LI
6

---

Learning Optimal Multimodal Information Bottleneck Representations
Table 3: Comparison of multimodal fusion methods for emotion recognition on the CREMA-D.
non-MIB- blBB-based
MIB- MlB-based
Methods
OMIB
Concat
BiGated
MISA
deep IB
MMIB-Cu Cui
MMIB Zhang
E-MIB
L-MIB
C-MIB
Acc
53.2
58.4
57.7
54.1
57.3
56.7
61.4
58.1
57.0
63.6
tion (see Appendix F). Each dataset contains all four types of information (fconsistent, modality-seeciffc] x (task-relevant, superftuous).. Moreover, they are synthe- sized with varying distributions of task-relevant information across modalities.
distributions observed in practice. Each dataset is designed for a binary classification task with labels y E {0, 13. In each experimental case, 3 is gradually increased from 10-6 to 10, well exceeding the proposed upper bound M,. The generated MIBs are fed into the trained OMF prediction head to predict y during testing. As shown in Figure 3, the prediction accuracy consistently peaks across all cases when using MIBs generated with /3 near or below M,, but rapidly declines as 3 further increases. This observation aligns with our theoretical analysis, empirically confirming that optimal MIB is achievable when I Mu. Notably, since My is a tight upper bound, peak performance may still be observed for 3 values slightly above M..
The emotion recognition experiment is conducted on CREMA-D (Cao et al., 2014). an audio-visual dataset in which actors express six basic ciootions-happy. sad, anger, fear, disgust, and nehoral-through both facial expressions and speech. The MSA experiment utilizes CMU-MOSI (Zadeh et al., 2016), which encompasses visual, acoustic, and textual modalities, with sentiment intensity annotated scale from -3 to 3. The pathological tissue detection exper- iment involves eight datasets derived from healthy human breast tissues (0x-hNB-(A-H]] and human breast cancer tissues (10x-hBC-(A-HH) (Xu et al., 2024b), where each dataset comprises gene expression and histology modalities. OMIB is trained on the healthy datasets and applied to the cancer datasets for pathological tissue detection. Detailed descriptions of these datasets are provided in Appendix G and Table 7.
As detailed in Appendix F, let æ1 [ao; bo: a1; b1] and 22 = [ao: bo: d2; b2] denote feature vectors of two obser- vations in Modality I and II, respectively. Here, dn and by correspond to the task-relevant and superfluous sub-vectors shared by both modalities. 01, are modality-specific resk-relevant sub-vectors, while b1, by modality superfluous sub-vectors. By design, the authentic optimal MIB is [ag; 01; a2], which is used to predict and com- pared against the prediction using OMIB DMIB-generated MIB. Additionally, we evaluate prediction accuracy using other feature sub-vectors, including unimodal information (a11 or x2)), consistent task-relevant information ([oo)), modal- specific task-relevant information (fa; a2)), and complete information (Jao: bo; 01; b1: ag; b2)). This experiment is con- ducted using SIM-I and SIM-II, corresponding to the cases of imbalanced and balanced task-relevant information, re- spectively. Table demonstrates that DMIB-generated MIB achieves prediction accuracy most comparable to the au- thentic optimal MIB, surpassing all other feature sub-vector configurations that either omit task-relevant information include supertluous information. These results further validate the optimality of OMIB-generaa MIB.
Table 4: Comparison of multimodal fusion methods for sentiment analysis on the CMU-MOST dataset.
Method
Acc7 (t)
Acc2 (?)
F1(t)
MAE(L)
Corr(t)
Concat
41.5
81.1
82.0
0.797
0.745
BiGated
41.8
82.1
83.2
0.787
0.738
MISA
42.3
83.4
83.6
0.783
0.761
deep IB
45.3
83.2
83.3
0.747
0.785
MMIB-Cui
45.7
84.3
84.4
0.726
0.782
MMIB-B -ulng
46.3
85.0
85.0
0.713
0.788
DMIB
40.4
83.2
83.3
0.810
0.784
E-MIB
48.6
85.3
85.3
0.711
0.798
L-MIB
45.8
84.6
84.6
0.732
0.790
C-MIB
48.2
85.2
85,2
0.728
0.793
OMIB
48.6
86.9
87.1
0.709
0.802
6.3. Emotion Recognition
6.2. Empirical Analysis of OMIB Performance Using Synthetic Data
Here, we compare the accuracy of classifying actors' emo- tion types in the CREMA-D dataset using OMIB and ten benchmark methods, including three non-MIB-based fusion methods (concatenation, FiLM (Perez et al., 2018), and BiGated (Kiela et al., 2018)) and seven MIB-based otate-of- the-art (SOTA) methods (E-MIB, L-MIB, and C-MIB (Mai et al., 2023) The classification accuracy of each method is reported in Table 3. OMIB outperforms all other methods, achieving improvements of 8.9% and 3.6% over the best- performing non-MIB -on--MB-bbaa based (concatenation) and
To empirically validate the effectiveness of our proposed Ji's upper bound in achieving optimal MIB, we simulate three two-modality datasets (SIM-(I-II)) corresponding to three experimental cases (case i-iii) (see Appendix F). Regarding task-relevant information, Modality I dominates Modality II in SIM-I, Modality II dominates Modality I in SIM-II, and both modalities contribute equally in SII-IIII thereby cover- ing the three primary cross-mode modal relevantt information
7

---

Learning Optimal Multimodal Information Bottleneck Representations
Table 5: Comparison of multimodal fusion methods for anomalous tissue detection performance on the 10X-hBBC(A-D)
datasets
Target Dataset
non-MiB-hasee
MIB-based
Metric
OMIB
Concat
BiGated
MISA
deep IB
MMTB-Cui
MMIB-Zhang
DMIB
E-MIB
L-MIB
C-MIB
AUC
0.537
0.489
0.498
0.522
0.623
0.626
0.423
0.511
0.598
0.496
0.728
10x-hBC-A
F1
0,884
0.821
0.873
0.878
0.894
0.897
0.865
0.877
0.891
0,881
0.904
AUC
0.866
0.518
0.499
0.379
0.818
0.817
0.849
0.643
0.770
0.481
0.903
I0X-hBC-B
F1 0.654
0.352
0.213
0.102
0.559
0.583
0.607
0,330
0.483
0.213
0.663
AUC
0.638
0.563
0.586
0.433
0.765
0.662
0.743
0.598
0.659
0,511
0.743
10x-hBC-C
F1 0.750
0.727
0.754
0.693
0.822
0,783
0,827
0.759
0.786
0.723
0.820
AUC
0.555
0.540
0.495
0.484
0,501
0,604
0.642
0.530
0.652
0.503
0,640
10X-hBC-D
F1
0.509
0.494
0.450
0.443
0.465
0.524
0,540
0.483
0.564
0.465
0.566
AUC
0.649
0.528
0,520
0,455
0.677
0.677
0.664
0.571
0.602
0.498
0.754
Mean
F1
0.699
0.599
0.573
0.529
0.685
0.697
0.710
0.612
0.681
0.571
0.737
(E-MIB) fusion methods, respectively. These results under- score OMIB's superiority in enhancing emotion recognition performance.
pendix K). Table 5 demonstrates that OMIB consistently surpasses the best-performing benchmark method by an av- erage laap of 11.4% in AUC and 3.8% in F1-score across the target datasets, confirming its superiority in anomaly detection in multi-modal setting.
6.4. Multimodal Sentiment Analysis
To evaluate OMIB's s effectiveness in improving downstream tasks involving three modalities, we conduct MSA on the CMU-MOSI dataset, which includes visual, acoustic, and textual modalities. Specifically, OMIB and the same bench- mark methods from Section 6.3 are used to predict a real- valued sentiment intensity score for each utterance, ranging from -3 to 3. Evaluation metrics for this experiment are mentioned in Appendix K. Additionally, OMIB consistently outperforms all benchmark methods across all evaluation metrics, highlighting its ability to generate high-quality MIB in modal setting for enhanced regression tasks such as the MSA.
Table 6: Ablation studies on the CREMA-D dataset.
w/o Warm-up w/o cross-aitn
w/o OMF
w/o r
Full
Acc
60.3
61.5
59.5
62.2
63.6
6.6. Ablation Study
To gain deeper insight into the key components of OMIB, we conduct a series of ablation experiments on the CREMA-D dataset (Table 6). First, we examine the effect of removing the warm-up training (wwoo warm-up'), which leads a 5.5% decline in accuracy. Next, we replace the CAN with simple concatenation fusion ('w/o cross-attn' attn', resulting in 3.4% drop in accuracy. We also evaluate the effect of replacing the entire OMF block with simple concatenation fusion ('w/o OMF". which significantly degrades model performance by 6.9% in accuracy. Finally, we assign equal regularization weights to 21) and T(E: 22) by omitting (" ( r') and observe a performance decline of 2.3% in accuracy. In a nutshell, the degraded performance observed after removing OMIB's key components highlights their critical roles in ensuring model performance.
6.5. Anomalous Tissue Detection
In this experiment, we aim identify anomalous tissue regions from the eight human breast cancer datasets (10x- hBCC-(--D),, which include gene expression and histology modalities. Due to the scarcity of tissue region annotations, we adopt the SVDD strategy (Ruff et al., 2018) for anomaly detection. Specifically, the model is trained exclusively on the eight healthy datasets (10x-hNB-(A-HH) to learn compact hypersphere in the latent space, confining the multimodal representations of inliers. The trained model is then applied to the four breast cancer target datasets, gen- crating multimodal representations whose distances to the center of the hypersphere serve as anomaly scores, based on which anomalous regions are identified. The benchmark methods are the same as those in Section 6.3 and modified to accommodate the SVDD strategy. The implementation details of OMIB for this task, are provided in Appendix H. The detection results are evaluated using the AUC and F1 scores, calculated based on the anomalous scores (see Ap-
6.7. Complexity Analysis
We first provide a theoretical analysis of OMIB's 's complex- ity. OMIB's 's modality-peecific encoder (Enc), task-relevant prediction head (Dec and Dec), and VAEs are implemented as Multilayer Perceptron (MLP), convolutional network, or graph convolutional network, each with a complexity of (NN,, where N denotes the number of samples (He & Sun, 2015; LeCun et al., 2002; Wu et al., 2020). For the CAN network, our implementation (see Appendix H) has a time
8

---

Learning Optimal Multimodal Information Bottleneck Representations
Buddhiraju, K. M., Porwal, A., Lati, R. N., Zarco-Tejada, P. J., Peleg, Z.., al. Multimodal earth observation data fusion: Graph-based approach in shared latent space. In- formation Fusion, 78:20-39, 2022.
8
1.4
Noin Soinghhingg phag0
2.0
1.2
1.0
1.3
0.5
1.0
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual information neural estimation. In Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80, pp. 531-540, 2018.
114
Figure 4: Runtime per epoch during warm-up and main training phase on synthetic data.
complexity of O(N M²) (Vaswani et al., 2017), where M represents the number of modalities. Since M is typically small, .M can be treated as a constant. Thus, OMIB's ' over- all thcoretical complexity ((V). We also empirically evaluate OMIB's is scalability to input size using the SIM- III dataset. Explicitly, we sample six datasets with sizes: 1 x 105, 2 x 10S, 4 10 6 10 8 10 and 1 x 106, while keeping the experimental settings identical to those of case iii in Section 6.2. We conduct separate analyses for the warm-up and main training phases, both of which demonstrate scalability to input size, as shown in Figure 4.
Cai, G., Zhu, Y., Wu, Y., Jiang, X., Ye, J., and Yang. D. A multimodal transformer to fuse images and metadata for skin disease classification. The Visual Computer, 39(7): 2781-2793, 2023.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-390, 2014.
Cao, M., Yang. M., Qin, C., Zhu, X., Chen, Y., Wang. J., and Liu, T. Using deepgen identify the autism spectrum disorder from multi-site resting-state data. Biomedical Signal Processing and Control, 70:103015, 2021.
7. Conclusion
We have proposed the OMIB framework, designed to learn optimal MIB representations that effectively capture all task-relevant information. Through theoretical analysis, we demonstrate that adjusting the weights of the IB loss across different modalities facilitates the achievement of optimal MIB. Our experimental results show that OMIB outperforms existing MIB-based methods. Furthermore, our approach is robust, successfully achieving optimal MIB regardless of whether the SNRs between modalities are balanced or imbalanced.
Chen, R. J., Lu, M. Y., Wang. J., Williamson, D. F., Rodig, S. J., Lindeman, N. I., and Mahmood, F. Pathomic fu- sion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41(4)7757-77, 2020.
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
Impact Statement
Cui, S., Cao, J., Cong, X., Sheng, J., Li, Q., Liu, T., and Shi, J. Enhancing multimodal entity and relation extrac- tion with variational information bottleneck. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing. 32:1274-1285, 2024.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Du, Y., Hu, J., Hou, S., Ding, Y., and Sun, X. A methodolog- ical framework for measuring spatial labeling similarity. arXiv preprint arXiv:2505.14122 2025.
Acknowledgments
We would like to thank Wenlin Li. Yan Lu, Zhengke Duan, and Junqi Li for their help with the experiments.
ElpSapp S., Abuhmed, T., Islam, S. R., and Kwak, K. S. Multimodal multitask deep learning model for alzheimer's 's disease progression detection based on time series data. Neurocomputing, 412:197-215, 2020.
References
Alemi, A. A., Fischer, Dillon, V., and Murphy, K. Deep variational information bottleneck. In International Conference on Leurning Representations, 2017.
Fan, Y., Xu, W., Wang, H., Wang, J., and Guo, S. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029-20038, 2023.
Arun, P. V., Sadeh, R., Avneri, A., Tubul, Y., Camino, C.,

---

Learning Optimal Multimodal Information Bottleneck Representations
Fang, Y., Wu, S., Zhang, S., Huang, C., Zeng, T., Xing. X., Walsh, S., and Yang. G. Dynamic multimodal infor- mation bottleneck for multimodality classification. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 7681-7691, 2024.
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P. Bagher Zadeh, A., and Morency, L.-P. Efficient low- rank multimodal fusion with modality-specifie factors. In Gurevych, I. and Miyao, Y. (eds.), Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 2247- 2256, Melbourne, Australia, July 2018, Association for Computational Linguistics.
Federici, M., Dutta, A., Forré, P., Kushman, N., and Akata, Z. Learning robust representations via nult in- formation bottleneck. In International Conference Learning Representations, 2020.
Lu, M. Y., Chen, T. Y., Williamson, D. F., Zhao, M., Shady, M., Lipkova, J., and Mahmood, F. Ai-based pathology predicts origins for cancers of unknown primary. Nature, 594(78611:106-111) 2021.
Guo, W., Zhang, Y.. Cai, X., Meng. L., Yang, J., and Yuan, X. Ld-man: Layout-driven multimodal attention network for online news sentiment recognition. IEEE Transactions on Multimedia, 23:1785-172 1798, 2020.
Mai, S., Zeng, Y., and Hu, H. Multimodal information bottleneck: Learning minimal sufficient unimodal and multimodal representations. IEEE Transactions on Multi- media, 25:4121 41344, 2023.
Hazarika, D., Zimmermann, R., and Poria, S. Misa: Modality-invarian and-specific representations for mul- timodal sentiment analysis. In Proceedings of the 28th ACM international conference on multimedia, pp. 1122- 1131, 2020.
Morvant, E., Habrard, A., and Ayache, S. Majority vote of diverse classifiers for late fusion. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR Inter- national Workshop, S+ SSPR 2014, Joensua, Finland, August 20-22, 2014. Proceedings, pp. 153-162. Springer, 2014.
He, K. and Sun, Convolutional neural networks at con- strained time cost. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5353- 5360, 2015.
Nagrani, A., Yang. S., Arnab, A., Jansen, A., Schmid, C., and Sun, Attention bottlenecks for multimodal fusion. In Proceedings of the International Conference Neural Information Processing Systems, volume 34, pp. 14200- 14213, 2021.
Huang. J., Lin, Z., Yang. Z., and Liu, W. Temporal graph convolutional network for multimodal sentiment analysis. In Proceedings of the 2021 International Conference Multimodal Interaction, pp. 239-247, 247, 2021.
Kiela, D., Grave, E., Joulin, A., and Mikolov, T. Efficient large-scale multi-modal classification. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 32, 2018
Parisot, S., Ktena, S. I., Ferrante, E., Lee, M., Guerrero, R., Glocker, B., and Rueckert, D. Disease prediction using graph convolutional networks: application to autism spec- trum disorder and alzheimer's 's disease. Medical image analysis, 48: 17-130, 2018.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, I.., et al. Pytorch: An imperative style, high-performance deep learning library. In Proceedings of the International Conference on Neural Information Processing Systems, volume 32, 2019.
LeCun, Y., Bottou, L., Orr, G. B., and Müller, K.-R. Effi- cient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Li, W., Xu, Y., Zheng, X. Dual advancement of representation learning and clustering for sparse and noisy images. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1934-1942, 2024.
X., Han, S., Wang. J., and Sun,
Peng, X., Wei, Y., Deng, A., Wang, D., and Hu, D. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/VV Conference on Com- puter Vision and Pattern Recognition, pp. 8238-8247, 2022.
Liu, W., Cao, S., and Zhang, S. Multimodal consistency- specificity fusion based on information bottleneck for sentiment analysis. Journal of King Saud University Computer and Information Sciences, 3012): 2024. ISSN 1579-
Perez, E., Strub. F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with general con- ditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
10

---

Learning Optimal Multimodal Information Bottleneck Representations
Ruff, L., Vandermeulen, R., Goernitz, N., Deecke, L., Sid- diqui, S. A., Binder, A., Müller, E., and Kloft, M. Deep one-class classification. In Dy, J. and Krause, A. (eds.), In- ternational Conference on Machine Learning, volume 80 of Proceedings of Machine I Learning Research, pp. 4393- 4402. PMLR, 10-15 Jul 2018.
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., and Philip, S. Y. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning sys- tems, (oms,32(11:44 32(1):4-24) 2020.
Xiao, G., Tu, G., Zheng, L., Zhou, T., Li, X., Ahmed, S. H., and Jiang, D. Multimodality sentiment analysis in social internet of things based on hierarchical attentions and esat-ten with mbm network. IEEE Internet of Things Journal, (116):27788-2733 2020.
Schulz, S., Woerl, A.- Jungmann, F., Glasner, C., Stenzel, P., Strobl, S., Fermandez, A., Wagner, D.-C., Haferkamp, A., Mildenberger, P., et al. Multimodal deep learning for prognosis prediction in renal cancer. Frontiers in oncology, 11:788740, 2021.
Xu, K., Ding, Y., Hou, S., Zhan, W., Chen, N., Wang. J., and Sun, X. Domain adaptive and fine-grained anomaly detection for single-cell sequencing data and beyond. In Proceedings of the Thirty International Joint Con- ference on Artificial Intelligence, pp. 1125-6133 2024a.
Snoek, C. G., Worring, M., and Smeulders, A. W. Early versus late fusion in semantic video analysis. In Proceed- ings of the 13th amnual ACM international conference Multimedia, pp. 399-402, 2005.
Xu, K. Lu, Y., Hou, S., Liu, K., Du, Y., Huang, M., Feng, H., Wu, H., and Sun, X. Detecting anomalous anatomic regions in spatial transcriptomics with stands. Nature Communications, 15(1):2233, 2024b.
Tian, X., Zhang. Z., Lin, S., Qu, Y., Xie, Y., and Ma, L. Farewell to mutual information: Variational distillation for cross- cross-modal person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 1522-1531, 2021.
Xu, K., Wu, Q., Lu, Y., Zheng, Y., Li, W., Tang, X., Wang, J., and Sun, X. Mcatrd: Multimodal anomalous tissue region detection enhanced with spatial transcriptomics. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 39, pp. 12918 12926, 2025.
Tishby, N. and Zaslavsky, N. Deep learning and the infor- mation bottleneck principle. In 2015 ieee information theory workshop (inw), pp. 1-5. IEEE, 2015.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0000057, 2000.
Xue, Z., Gao, Z., Ren, S., and Zhao, H. The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation. In International Conference on Learning Representations, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L.,, and Polosukhin, I. Atten- tion is all you need. In Proceedings of the International Conference on Neural Information Processing Systems, volume 30, 2017.
Yao, J., Zhu, X., Zhu, F., and Huang. J. Deep correlational learning for survival prediction from multi- modality data. In International Conference on Medical Image Comput- ing and Computer-Assiste Intervention, pp. 406 414. Springer, 2017.
Wan, Z., Zhang, C., Zhu, P., and Hu, Q. Multi-view informatioo--ootleene representation learning. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10085-10092, 2021.
Zadeh, A., Zellers, R., Pincus, E., and Morency, L.-P. Mul- timodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82-88, 2016.
Wang, C., Gupta, S., Zhang. X., Tonekaboni, S., Jegelka, S., Jaakkola, T., and Uhler, C. An information criterion for controlled disentanglement of multimodal data. In UniReps: 2nd Edition of the Workshop on Unifying Rep- resentations in Neural Models, 2024.
Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. In Palmer, M.. Hwa, R., and Riedel. S. (eds.), Proceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1103-1114, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
Wang, Q., Boudreau, C., Luo, Q. Tan, P.-N., and Zhou, J. Deep multi-view information bottleneck. In Proceed- ings of the 2019 SIAM International Conference Data Mining, pp. 37-45. SIAM, 2019.
Zhang, S., Yin, C., and Yin, Z. Multimodal sentiment recognition with multi-task learning. IEEE Transactions on Emerging Topics in Computational Intelligence, 7(1): 200-209, 2023.
Wolf, F. A., Angerer, P., and Theis, F. J. Scanpy: large- scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018.

---

Learning Optimal Multimodal Information Bottleneck Representations
Zhang, T., Zhang, H., Xiang, S., and Wu, T. Information bottleneck based representation learning for multimodal sentiment analysis. In Proceedings of the 6th Interna- tional Conference on Control Engineering and Artificial Intelligence, pp. 7-11, 2022.
Zhang, X., Yoon, J., Bansal, M., and Yao, H. Multimodal representation learning by alternating unimodal adapta- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 27456- 27466, 2024.
Zhou, J., Zhang, X., Zhu, Z., Lan, X., Fu, L.,, Wang, H., and Wen, H. Cohesive multi multi-modality feature learning and fusion for covid-19 patient severity prediction. IEEE Transactions on Circuits and Systems for Video Technol- ogy, 2255) 2021.
12

---

Learning Optimal Multimodal Information Bottleneck Representations
Appendix
A. Proofs of Mutual Information Properties
Properties A.1. Properties of mutual information and entropy:
i) [(x v) 2 0. (1: O.
a) ()) [(=:0) + [1x; =ly)-
ii [211:22-- ikn+1) = 1(211-1 ;2n))
- -1(13-- inn [x+1)).
iv) F(())) = - [(21:0g( = - (x113
1) (f _ F(n)) - 02)1;02)= H(22),
H()1,02) F(v2)) = F(v1) H(n)
vi) ( #0 - (())) = (vv)) + (v2)
-1(1):0)
vi) 1f (()))()((())) = + 02) = H(v)) + H(v2)
U((1))
Proof. The proofs of properties ii, and iii can be found in (Cover, 1999). For property iv, we first observe that:
F(y) 0 p(u, 2) p(v)p(z)
(22)
This implies that and are statistically independent. Consequently, we have
T(u: = Σ plu, log p(y, 2) p(v)p(2)
plu, 2 log p(y)p(2) p(y)p(z)
Σ
(23)
2.2
Σ
p(u, 2)
8.z
0
Given that 2(y; T(x; 2) + T(x; ely), and noting that I(x:; y: 2) 2 0 and T(x;=10) 0, it follows that:
I(y: =) 0 - T(x; z) and 7 ((;; =/v) = 0
(24)
Therefore, we obtain that:
(25)
[(x; v/2) I(x: v) [x: ys 2) = I(x; v)
13

---

Learning Optimal Multimodal Information Bottleneck Representations
For property
(11) = JI pv, u2)g p(ui, v2)
p(uz (p(2))
=s F(12)C((1)))
JJ (o(ur,e2)on 01.
p(ezlur) p((r))
(26)
p(v1)p(U2)
p(o2)log(p(v2)) = H(vz).
In addition, for [(11, V2), we have:
H()1, F(m)
p(e,, va)log(p(er 02))
ff 21.22
p(m ((looo(((op)))))))m())))
(27)
p(u Moo))oo((((i)) = H(m)) F(m)).
For property vi, we have:
(())) H(22) 0 p(U1, # p(C))p(U2).
(28)
The mutual information [(11; U2) is defined as:
((v1502)= Jf plui, 02) log p(u1, v2)
p(vn )(v2) dui duz &
Jf p(u1, v2) log puu1, v2) du dog Is p(u,, v2) logp(vi) dui dug
If p((,, U2) log p(2)) dei duz
If p(v1, U2) log ((v) duj duz- ʃf p(ul, 02) log p(v)) du duz
(29)
+ [f p(1., u2) log p(ri. 12) du duz
/ p(r)) logg(v)) den / pluz) log p(2)())))))))))))))()))))) des
+ [f p(01,22) 22) log p(vi, v2) dui duz H(v)) + H(2)) (v),,))) v2)
Since (v1) n H (v2) # - 1 (c: 02) # O, we have ( v2) = H(v1) + ((22) - (211)22)
For property vii, we first clarified that:
F(22) n F(11) 0 - p(11,2)) v2)),
(30)
14

---

Learning Optimal Multimodal Information Bottleneck Representations
Therefore, have:
H (11, = JJ
p(1,, #2)log(((11 v2))
JJ =(tr))(ee)llogoleeeee)t)))
01
(31)
p(or)oo((p(ti))
- -(vo))oo((((e))
P1
1/2
(v1) + H(2))
B. Proofs of Proposition 5.1 and Proposition 5.2
For convenient reading, the equations used in the proof' copied from the main text:
LOMF Σ R. Reg log
r)).
(32)
which is copied from Equation (10).
N Σ
r= -anh In N ==1
E2, Eea
(33)
which is copied from Equation (11).
min ((E) = min -((::) + B(T(5))) + +2);
(34)
which is copied from Equation (17)
T(y; v1k)) I(y: v2l5)
(35)
which is copied from Equation (18).
Proposition B.1 (Proposition 5.1 restated). The loss function, LOME, in Equation (32) provides variational upper bound for optimizing the objective function in Equation (34), and can be explicitly calculated during training.
Proof. For [(E: y), we have:
I(E:v)= E))og Eolog ply.E)
p(v)p(E)
(36)
p(ylE)
dydEp(y. Eloo p(v)
Let g(vlE) be a variational approximation to p(yle) and we have:
KLIp(vle)lla(lee)) 2 log 2 dyplule) log log (
(37)
Based on the above inequality, have (Alemi et al., 2017):
T(E:t) / dydErlu. ((vlE)
p(y)
/ dyddErly log g(vlE) - dydEr(y, p(v) p(1)
(38)
f dydEp(y Eloo ((vl)) H
15

---

Learning Optimal Multimodal Information Bottleneck Representations
HY) be ignored it is fixed during training. Therefore:
T(E: / (ondEp(w.Elogg(lsee q(vl<)
(39)
/ dydddd(iddddedddapppt 22, 22, S1. S2, E2. ((yl5)
Furthermore, because is a function of Ça and S2 (i.e., 5 (AN (2)), we have [(:: 21) S I(S1. S2:21) 21) and [(: 22) (2) Using the Markov property, we have Ç1 22 and C2 1 21, which leads
(40)
2() S I(SI, (2:21) = 1(112))) + I(22:2 1(1122))
Similarly, [({: & T(22; 22). Therefore:
p(S(lz)) p(G))
z)) ((G
((Gi))) = dC:dz:p(Si zi)log
Vi E [1,2]
(41)
(etr((i) I) be a variational approximation to p(Gi), we have:
T(E: 24) T(G; (4 / log((s))
/ dgdde.p(G, 2:1oog p(Gi/z))
pog
(42)
/ dGddzp(Gi 200 (()) Vi {1,2}. N(O, I)
Put Equation (39) and Equation (42) together, we have:
- I(Ev)) + 8 ((E) + +r((Eiz2))
- / (((()))+++(((()a))
21, /
dEdC (eld///seoee.....G/G/aaaaaaaaaaa
(43)
+ 8( offdeepll de1p((z) / / dcsp(Gal2))oog
2(G] I)
+r dzap(zs) /
dCsp(Gal<>)log
p (((a)2))
NO..
Note that p(=1, 22, s)..))paa. p(21). and p(z2) can be approximated using the empirical data distribution (Alemi et al., 2017; Wang ct al., 2019), which leads to the objective function:
N
/
L
(44)
p(c (2) 1(0,1)
p((3125)
+ 8
dcip(sFl<PMog
+r d<ap(<3
NOO.
Given Ci = + Eg X ej in Equation (6), we have:
1 N N
C =
& (( + -,
Σ
Br | (ogg(v"(""") KL
(45)
LOMF
This completes the proof.
Proposition B.2 (Proposition 5.2 restated). Equation (33) satisfies Equation (35), thus providing an explicit formula for computing during training.
16

---

Learning Optimal Multimodal Information Bottleneck Representations
Proof. Firstly, =1 and 22 sufficient encodings of modalities 01 and v2 for y, respectively. Let ü, represent the superfluous
information in i; that is not encoded in 2;- Then, we have:
[w: v:(5) = I(y: 2e, 0.15)
21)1) +
1yy
(46)
=0 F(y)n(())=)
(1; (1,2}.
Let 21 = { and =22, {t, then we have:
11 vil)) (15 ((1,, - I(E: v) = [(=4g) 1(Ei)),
(47)
(y;; 02lE) = (2))) = (22, - (E: y) = [<sig) ((:2)).
Then I(y: 21 be expressed as:
T(y: 21 I(za:t) v) T(g:8)
(y) (v)22)) (v) + H(llE)
(vlE) - Hullz))
=- ME)dE / (llE))ogppull)e + plzz)ddzs / plulza)log p(ul za)dy
-- JJ p(E)p(ulE)ogg (p())z)) p(y]E)
p(y)2z4) ddE
Jf p(z))p(ll24)0gg pl k) p(yl=z)) Idzddy
p(yl (E)
II p())p(v(E)(gg p(ule)
dEdy [I] [(SS)hi//(Kooogpptaaaaaaee
p(y)21)
p(ul (2) dządy + [] pl p(y(E)
+ JJ p(z))(v1)))0og
(CKKL(Q(/(((()))))))))) p(y)log p(y2z)))
+ f (cW0K110(()222))0000)))) +
p(u)logplulE)d
(48)
/ [tta)K/Uppll))))bb4443)adaadd +
p(u)log ply(E) dy
p(ul =4)
f [(())))((((((||<<//))))))))
/ (lc))K((((()))))bl(((()).). + / p(E)p(ule)log p(y15) p(u)|2))
dydg
/ {(]))uμt(((())l(((>))))))
/ ((ca)KK.//(/))))xx)((()))) [(C)KIA//////)/())))))))
f [(()xx)k(((((()]]x>>>xxxx))){)
/ /ccc)K(ppllll))llb)(())))))))
E., [2,1KK((66])))))00011111
17

---

Learning Optimal Multimodal Information Bottleneck Representations
Similarly, [y: =2({) E,, [KL(p())2s (p((((( and f(anva 19) I(araa (
can be calculated as:
T(;; v2l5)
Eg KL(p(ul 2s) p(ul)))
I (y:v115)
Ex (L 21) p(y()))
(49)
Be Beg
Finally, we have:
1 N "1"6000001-3)))000"""""""""
1 tanh In N
Beg Ee
(50)
[(3 v2l5)
I(y; v1
1 tanh(In
T(y: 01 k))
T(y;
This completes the proof.
C. Proofs of Lemma 5.5 and Lemma 5.6
As proposed in Section 5.1, the objective function of MIB can be written
min (()) min ((: ((((()1) +(((222))
(17)
Based on Assumption 5.3 in Section 5.2, we have:
F(y) = {a] = {ao. 01, a2);
F(11) = {ao, 01 by bu), (v2) = (ao, 02 02,02,22 b2, bul:
{ao} 0 {ai
{ao} {a2} 0, (ai) n {a2] = 0,
{bi} n {ao} 0, (6,) n 0,
vi E 1,,1,2)
T1,; = {a} n ((v)) = {a0.a1}, 02); = [[a n ((22) = {aa. 02)-
Definition C.1. The relative mutual information between encoding z and task-relevant label y is defined as the ratio of their mutual information their total information:
T(2:8) T(2:1) 7(x;)) = F(:)UF(y) F(z,8) H(=) + Hyy) 1(2;g)
T(2:y)
Compared to mutual information, relative mutual information more accurately reflects the amount of task-relevant informa- tion (i.c., (E: in total information (i.c,, F(E) U F(y)), which aligns more with the objective of maximizing task-relevant
information in MIB. Consequently, replace VE: with I(E: v) in Equation (17) in the following analysis.
Lemma C.2 (Lemma 5.5 restated). Under Assumption 5.3, the objective function Equation (17) ensures:
F(E) 2 (ao, a1, a2),
(51)
when B My. where My (mm
Proof. Let {&1} = ai, a1, ((E))) ) represent the task-relevant information in a that is not included E. It is obvious:
(5) F(y), F(E) 0,
(5.) n =. (a2) = 0,
(52)
(5) n F(22)= = 0. n
F(22) = 0.
18

---

Learning Optimal Multimodal Information Bottleneck Representations
If (61) 00 let 5' = - (.). Using properties in Appendix A, have:
v(S:v1) v1) = m(): - v1)1)
(6)))(())=0(5)))
[(5:01) - (((iv1) - 1(v1:E11E)
= (1(11G) <0
v(5) - u(5); = T(E:v2) - T(g, E1;
(())))())))MM))
(12=:611)===
v(E:v2) v()
T(12:E1 (
0
E1:
I(5:y)
f(g':v) - [(E:0)
F(E, E1.8)
F(
[[5, E1:3))
I(E: y)
E1,,
F(S, v) F(G)
F((..) (())C)())5))
F(E, E1,y)
Ei);
F(E, E1.V) > 0
For ((E) - E(("), we have:
E(E) ((g') 7(5':1) - ((E:)) + ((T(((:))))T(()) +()()))))))))) - rT('::
I(y:Ei) (I(1):
F(C,E1,y)
When 3 < F( E(E) - E(E') > 0, so optimizing the loss function will drive ş toward g until (ε)) =
namely F({) 2 {a1}. We further suppose { = (feo, a1, a220/(((aa a1, a2) F())) n {az} represent the task-relevant I(nnaa) information 02 that is not included in ş. Similarly, if (62) + and B < (((( the optimization will update g
until } 0, namely F(E) 2 {az}.
Moreover, let { (fau, 01 a2 02)/((ag. 01, a2) F(E))) (ao) represent the task-relevant information in ag that not included E. If {Eo} # let E' {E. E Then we have:
T(S:0) - T(g':0)) T(Sivs) - T(g, Eo; v.)
(Eo)nF(E)=1)
= [(::v)) - [(E:v)) 1(11:60k
-I((v; < V, (1,2).
19

---

Learning Optimal Multimodal Information Bottleneck Representations
(os,
T(E:V)
T(E':0) I(E:v)
F(S. Eo, v)
V(E.V)
I(S, Evi
I(E:3)
FCC Eo. y)
F(C.9) F(E))
-F(E.0) as (E0)
(6))nF(()=0
Iyy:
F(E, Eo: v)
(ly >
F(E, y)
For ((E) - E(E'), we have:
- - (()(( I(E'iv) + () - [(5:
+(((v - v2)) Ely:Eo) ((1(v1:5o) (v2:Eo))
F(E, Eo, 9)
T(viE)) Therefore, when (156::2)(((0ns555)) the optimization will update until {Eo} 0. namely 2 {ao}
Put together, the optimization procedure ensures F(5) 2 {ao. a2) when:
((u:E1) 8 < UBB mén ((E,:E)) (T(: F(E, E2, F(E, Eo. ()(r Eo) - r (w2; Eo))
T(y: E2)
I(y:
(53)
is a lower bound of U Ug. When 3 < Mu. the optimization procedure guarantees F(E) 2 {ao, 01, a2). This completes the proof.
Finally, we prove in Lemma C.3 below that My ((2)
Lemma C.3. U in Equation (53) satisfies: UB Ma where Ma (Ho))
H+) and I(:)) can be estimated using MINE (Belghazi et al., 2018) (see Appendix E). Proof. As shown in Equation (53)
1(y: (((r:E)) r ((,, F(s. v)) + (+(22)
1y::
1ly:
UB min
(51) C F(S.9), v), we have F(g,E1,)) F(E,v) so that:
1(9:E1)
((. [(E1:E))F(E.)) v)
(51 S {ai), {ai} S {ei}, and {a1} C [u). i. f&i C {er} and {&} S {v}. Then, according to property v in Properties A.1, ((y:)) = H()) and [(v1;E1) = H(E)), which leads to:
H(E))
I(v::)))((E.)) ((L))
F(E.V) v)
T(vE2)) Similarly, (f(e (8.6..)))
is simplify to "F((.))
Moreover, F(g, Eo. v) = F(g, y) since (Eo) C F(C, y). Then, it follows
that:
I(y: Eo)
I(y: Eo)
F(g, En, ()(( (l(22))))) F(E. ((E:))(((viEE)) (12;
20

---

Learning Optimal Multimodal Information Bottleneck Representations
(E6) S {ao), (2o) (v1), S (u2), and {oo} S (v)... :. (Eo) (m1). (6o) S (v2), and {&o S (v). Thus, by property v in Properties A.1, T(y: Eo) H(Go). (o): = H(6)), and T(22; = H(G)), which collectively lead to:
Toy: (():)) + (v2:
((G))
F(C, (H)) rH)
F(..)
min
Vr 20
F(5,8) rF(
E(y:
T(y:E2)
minn
Thus, we have:
1(y:60)
UB, ((1:5) + + (12;6))
1
(1+r)F(E.V)
1
(1+r)
F(V1, 02) :F(Es)CC(
(H))) ((11)22))
Ma
This completes the proof.
Lemma C.4 (Lemma 5.6 restated), Under Assumption 5.3, the objective function in Equation (34) is optimized when:
F(() S {ag, a1, a1,
(54)
Proof. Let represent superfluous information that is specific to e1 and not incorporated into Ç. Then, we have:
1 $ {ao, ai, a2), C (()), >
(55)
r($1:)) (2) = fox 0.
Let E (E, 2.). The objective function becomes:
(()) -((:)) + (
+r(()
(56)
[[( 3(I &1y01) +
Then we have the following equations:
($))0())=0
I(E,21;01) 1((:e1) [(112111)
(57)
[(vi: 21)
(11)0(v2)=9
1(5,21102) 1(E:v2)
(v22 21
(58)
21

---

Learning Optimal Multimodal Information Bottleneck Representations
I(E:y)
T(g:u) - T(g, 81;0)
F(C.V) v)
F 5.
T(E:U) +
T(:: 81
F(E, v)
F(31,6,9)
E(
(59)
F(E, v)
F(
I(21.E:
I(21.5:9)
F(E, v)
F(21) + P(C. y)
VR (C.0)
20
Put together, have:
((E) - ((5)
= (f(E:v) - T((,, (1:)) + ((()),))))) - T(g:)))) + +r(((t) - T(E: w2)))
(60)
> 0, if {81} #
For superfluous information & specific to U2, we arrive at the same conclusion. Finally, let ån represent superfluous information that is shared by the two modalities and not encoded in Ş. Then, we have:
L0 & foo, 41, )2), { F(m)), (2) c F(22).
T(2;; v1) > 0, [(2:: 02) > 0, [(fo; = 0, (5) n [
(61)
Let Let (E, 20). The objective function becomes:
{(²) - ī(É:v) + ((((()))) + +r((Eu
(62)
- T(E5 (o: + (T( 20; + r((, 20; 02))
Then we have the following equations:
T(E:W)
T(ãu, {)
Î(E:)) Î(c, Bu:
F(E.V) v)
F(2o, E.
I(E:U) +
T(y: ZulE)
I( {:
F(E, v)
F(ão, C,y)
((ão.E:v)
T(20, Evy)
(63)
F(E, v)
F(2o. E.
(1Bo.E:V.
I(20. E:y)
F(E, v)
F(20) F(E,
0
(Eo)0(E)=0
((1,2o 1(5:01)
[(vi: Eolg)
(64)
[(v1: 20) > 0
22

---

Learning Optimal Multimodal Information Bottleneck Representations
Similarly, have I(E, 02)) v2) (u225 20) 0.
Put together, have:
E(E) P(E)
(T(E:)) I(E, 20: + - -
+((((E.)(:0)) T(g: u2)))
(65)
> 0, if {Eo} + 0.
In nutshell, the optimization procedure continues until does not encompass superfluous information, shared or modality specific, from 11, That is. F(E)= C {ao, a1, a2). This completes the proof.
D. Extension to Multiple Modalities
1
V2
b11
b22
a11
a22
aoa
boo
a13
a23
a33
b33
1/3
Figure Venn diagrams for three data modalities (v), U2, and v3). The gridded area represents consistent information, while the non-gridded area denotes modality-speeifff information. Tesk-relevant information highlighted in green, whereas superftuous information is shown in blue.
The theoretical analysis of multiple modalities (2 3) is exemplified using three modalities, U1, U2, and v3, which yet can be readily extended to more modalities. All mathematical notations remain consistent with those in Table Section 3.
U2. and v3, the task-relevant information set {a} consists of seven parts-40o- -00. Q11, a22. 033, Q12, 013, a23, as illustrated in Figure 5. Specifically, aoo is shared by all three modalities, while a. is shared between modality pairs (v,, vy), Vi,j E {1, 2, 3}, < j. Meanwhile, a; is specific to v, Vi E {1, 2, 3}. The task-relevant labels y are determined by {a}. On the other hand, superfluous information is represented by (b) (boo, bi bz2, bas, b12, bis, b2a). Here, bno shared by three modalities, while by shared between modality pairs (v,, 0z), j E 11, < j. Meanwhile, by specific e, Vi (1.
Assumption D.1. Given three modalities,
Based on the above assumption, the optimal MIB has the following definition:
Definition D.2 (Optimal multimodal information bottleneck for three modalities). The optimal MIB for three modalities defined as the MIB that encompasses all task-relevant information and free of superfluous information. Let Copt three
23

---

Learning Optimal Multimodal Information Bottleneck Representations
denote the optimal MIB, and it can be explicitly expressed as:
F thrret {000, d11, 022, 033, 412. 013, 023).
(66)
In the following sections, we first demonstrate the method for achieving the optimal MIB, followed by theoretical analysis establish its theoretical foundation.
D.1. Method
The warm-up and main training phases follow those for two modalities Section 4, except for an additional modality Ug. The loss function LTRB remains the same for each modality in Equation (4), while the loss function LOMP becomes:
LOMP= Σ Bes Eeg Ee,
8 (K(p((F)<)) (KK I
(67)
+ +n +
where { = CAN((I f2, {3, (CAN). Analogous Equation (11) proved by Proposition 5.2 in Section 5.1, r1 and 72 are dynamic during training and explicitly calculated as:
N Σ
tanh In
Eej Eeg
N nm1
(68)
N
22 tanh In
Σ nm1
Be, Eeg
N
Moreover, as proposed in Lemma D.4 in Appendix D.2, when 3 in Equation (67) is upper-bounded by M (1 (())) Eigrsss Messyy)) the optimization of LOMP ensures the achievability of Eapt-
D.2. Theoretical Foundation
Under Assumption D.1, we have:
F(y) {a} {aoo, @11 022, 033, 012, 413, d23).
F(v1) = { {000, 011, 012; 013, bı, b12, d1s, bo),
F(02) = {a0o, 022, 012; 023, b2, b1z, 82g, bo}-
F(va) = {aao 033, 413: 023, by, b1s, b23, buk.
{aiz] n {awe] 6, Vaij. Guv {a}, where az # Que
(biz) n [ann) 0, Abij (b), Oun E {a}
I( ={a} n F(v1) = {aoo, a11 a12, a13),
I(y: 02) = [aa n F(22) = {aoo, 022, 012, 423}- I(y; mq) = [a) n ((ss) = =(ao, 433, 013, 423
Analogous to analysis Section 5.1, the objective function for obtaining optimal MIB can be formulated as:
max ((E) = max I ((: 3(I +2) +22(([))))
(69)
which is equivalent to:
mjn ((S) min - ((::)) ((1 ()) re: za)).
(70)
The Proposition 5.1 in Section 5.1 can be trivially modified by adding the red 23) term and applied here to establish TOME F Equation (67) as variational upper bound for Equation (70).
24

---

Learning Optimal Multimodal Information Bottleneck Representations
1.2.1. ACHIEVABILITY OF OPTIMA INFORMATION BOTTLENECK FOR THREE MODALITIES
Lemma D.3 (Inclusiveness of task-relevant information for three modalities). Under Assumption D.I, the objective function in Equation (69) ensures:
F(C) 2 {000; 011, 022, 433, 012; 413; 023).
(71)
when B & Ma where M = (1+ H(w:)- (e000--iigeeae Eigeryaa (1ann)))"
Proof. We first analyze under which condition & can include all the task-relevant information specific to single modality. Let { = (fama, 011, d22, 433; 012, 413: d/3)/ //(0)/0/0 a11, 022, 433, 012, 013, a23} n I(E))) n {an} represent the task-relevant information an that is not included Ç. It is obvious:
{ c F(y), (() 0, [ay)= 0. Vay E {aa//ann). ((22) = 0. (((z)) 0, ((va) = (0)0(((aa)-0(())))(3)))) ((za) =
F(22)
(51) + 0, let 5'= = (C: } and we have:
((E) -ī(E: v) + ((I(E:))) +ri)(E:0)) +22((:: 03))
=0, =0,115))
=0. =0,16 (6
-ī(E:v) + ( (T((5)1)) +r((2))))
((12):11)) )++2((()s))) +
(1x:)115) ))
-f((E))) + -((E:v)+ +3 + (E,
((E) ((5') -((:v) + + +r2(()(r)))))
=((,, - -( ((C,Ei)))))
Using properties in Appendix A, have:
T(g,E1;))
T(E:W)
flv) Î(E:U) = F(E) (F(G)
F(E) F())
=F(v) as (6))F(())
((E,E13))
[(::))
F(E.
F(C.V)
(E)0(E)=0
F(E,V) v)
E(v:EL)
F(E.V)
(6: (E)=
- -(::0))-((EEi -((,E(1)))) = - 1(01:Ei1k)
(72)
= -v(
25

---

Learning Optimal Multimodal Information Bottleneck Representations
Thus:
A(E) - ((5') [(g, - [(g:)) (1())1) T(y;Ei) (1(11;11)
- (I (1)
(73)
P(E.))
When Hyss1)
((rr1)) ((5)~(()) 0. Therefore, optimizing the loss function will drive & toward 5 until (51 0, such
that F(5) 2 {an]. We further suppose { (fag0, 411, 022, 033, 012, 013, (23) (/(ooo 411, 022, 033, 012, 413, 423
I(E))) n { represent the task-relevant information in 022 that not included in &, and (E)
(fago: @11 422, 433, 012, 413, /23))((a000- a11. 022, 433, 012, 413, 423 n I(E))) n {ass} represent the task-relevant infor- T(nna))
mation in 033 that is not included in Ş. Similarly, if {E2} } o 0 and B < (n the optimization will update { I(gnEa)
the optimization will update & until until (E2) = 0, leading to F(E)= 2 {az2}: }: and (Es} * 0 and B3 < =2l(0ssaa)((((fmm))
} = 0, leading to F()) 2 {azs).
We then analyze under which condition & can include all the task-relevant information shared by two modalities. Let {612) = ((0g,, 011, 022, 033; 012; 013; 023) (foon, a11; 022, 033; 012, 013, 423 (T(E)) {a1z) represent the task-relevant information in 012 that not included Ç. is obvious:
(É12) c F(w), (E12) ((E)) 0, (612) (an) = 0, Vag E {aa)/(112) ( = 0, 1612 n F(z)) =
If (É12) # let (C, E12), then we have:
E(E) = [(E:8) + (()(e)))))))))))))))))) +22((E)
=0, -0:116010P(000
-ī(E:v) 77(2))
((v3:61215)
= + (
Therefore, ((E) - E(g') can be written as:
E(E) e(g') = -ī({:)) B([(E; +n
+ (ra.a:/0-euuuuaannoooor////////-
+ (G (12.
Using properties in Appendix A, we have:
I(g,E12:8)
I(E:
v(E,E12;0) - Î(E:u) = U(E) (F(y) U F(G12)
F(E)
=F()) ((1))CF())
I(E,E12:V)
I(E:V)
F(E.V)
F(E.
16:=)D(S)=0
1(vi EI2lE)
F(E.V)
E12:
> 0
F(E.V)
26

---

Learning Optimal Multimodal Information Bottleneck Representations
((1))(()=))
v(5:m)) T(g, (12;m) = ((11:51215)
(74)
-1111i(12)) <
Similarly, obtain that u(5: - ((2,112)2)) = -1(22:
Thus:
A(E) R(5)) = I(s. (1238) [(E:v) + 3 (
- ((5 - =1 (12122))
(12: B(I(U1i(12) + (12((12))22)
(75)
F(E,2)
When 3
1(r:f12)
(()) e(E') 0. Therefore, optimizing the loss function will
drive & towards ç' until {612}
0, such that F(E) 2 {a12)- Similarly, suppose that {61s)
(f000- 011, d22: 433, @12. 413; (23) (22))(((u) /((000 a11: U22, 033: 012, 413, 223) ( represents the task-relevant informa- tion in 013 that is not included in ç: and (E23) = (fa00, 011, 422; A33, 412 013, 023), /(f0000 Q11, 022, 433, 012, 413, 423) N (())) n } represents the task-relevant information in a23 that is not included in E. Following the above procedure, we ((x+[1))
conclude that if {613} + 0 and B <
the optimization will update until {É1s} leading to
FIS.)l((ee:ssss))r//(/ae T(y(E2))
F(E) 2 {ais}: if {É23} A 0 and B < ((t (VVx)ess)) the optimization will update & until {É2s} 0, leading to F()) 2 {a23).
Finally, we analyze under which condition & can include all the task-relevant information shared by the three modalities. Let { = (fama, 011, a22, 43s, 012, 413, 023 }/ (fa00, d11, 022, 433, 012, 013, a2s} n I(E))) n {avo} represent the task-relevant information doo that is not included Ç. It is obvious:
(6)) F(w), (5() =0,
( 00 Vai E {{a///aok)
(5o} n F(n)) 00, W/ (1,2,3)
If (50) + 0, l't = Eo), and ((E) {(E') can be written as:
((E) ((5')
[((::)) v) + ((T(E: + +r r2T vz))
(-i((E + 5(I (
U2) 12d Eoi us))
T(g, - [({:v) [(E: T(S. Eo:
+ +r((((E:02) ((,, v2) + r2
[(5, Eo: vo))
Using properties in Appendix A, we have:
I(E, Eo: v)
I(E: v) F(E) U F(y)
Eo - Î(E:8) = F(E) U (F(y) F(Co))
=F(v) (É))CF())
I(E:3)
F(E.V)
F(E. v)
1(v: EolE)
F(E, y)
((u:E0)
>
F(E.N)
27

---

Learning Optimal Multimodal Information Bottleneck Representations
150)0(5)=0
((:vv)) Eo; (ol1:oll)
(76)
--1(vii <
Similarly, we obtain that u2) - [[, Eo; (1(: and ((5:03) - I(S, Eg: = - -1(z Eo).
Thus:
M(E) - ((E)) = I(E, Eo I(E:n) + 3 ()) E1)
+ +ri(1(E:02) - 1(5 v2)) + r2 (2(((; us) - ((E,Cu:))))
(77)
Ely: -B(((11: n (12: Eo) + (v3::
F(E.y)
When 3 (( ((22)))) ( (vsske))) ((E) ((E') 0. Therefore, optimizing the loss function will drive toward 5 until {fo} = such that F(E) 2 }
I(nEo)
Put together, the optimization procedure ensures F(E) 2 [000: Q11, 0222 033, 412; 013, ag3} when:
<< <BB = min(U U U UB
U
(78)
where UB}
1(nE.)
U
UB
MynEa)
UB
T(y(E12)
UB3
= FEaaW/oonda)))r//tsstsl))
T(
U
I(ysEa)) Fitalles
and
U
f(y:6o)
We complete the proof by proving that M2 (1+n H(n)) Eisreisa ((vaw))) is lower bound of B3 in Lemma
D.4 below. That is, when 3 < M the optimization procedure guarantees F(E) 2 {a0,, 011, 422, 033, 412, 013; 423)
Lemma D.4. UB in Equation (78) satisfies: UB3 M3. where M (L- H(w.) Eigiiejaa (vvre)))
H(:) and I(z:) can be estimated using MINE (Belghazi al., 2018) (see Appendix E).
Proof. As shown in Equation (78), UB}
1(x55)) ((ri))
By property in Properties A.1, UB} can be simplified as:
1(yi Ei)
UB!- = Twi (I).
-(
(())
H(S))
F(E.y)
F(E.9)
Similarly, have UB3- r ((22k)))
T(vEE))
and UB3
1(
F(.()
ra ((ra))
("F((.0)
28

---

Learning Optimal Multimodal Information Bottleneck Representations
U can be simplified
[y::
UBS
((viei((( + (11((22i112)
-1(v:612) ((re))()))
H ((112)
(79)
F(E.2)(
((12)
tri
((11)
(1E.)
Similarly, have UB²
and U
(1+1))
UB3 can be simplified as:
Eo:
UB3- = ((( + +ra((12: Eu) +
red((s:
= :(
(Éo)
F(
H(E))
+ri
(Eo)
H
= :
(1+ F()
Therefore, for Vr1, 12 > O, we have:
+r)) min(
F(g.V) F(E.v))) r2F(E.)) (1 (1+ (1+r2) v) (ni + F(.))(())) v)
= =min(UB), UB, UBS UBJ, UB, UBS. UBJ)
1
= ~UB+ +1
(2)
(12r)+2) + + + (211822) - ((11i0)) ((22 v3) + U2;
(2 v3))
For the term [(11;02) + ((11)1)) [()2 I(v]: u2i U3), we have:
[(11:0)) + (()1 [(22: 03) I(11:02) 03) [(11:0)) + [()1;0)) [12; 03) I(v1: 02) [(01;0)) [(22: 03),
[(11;1)) + (() (2)) 1(11002) 03) (2)))2) + [()1:0)) [(22: 02) I(U1:0s) [(01 [(22) vy),
[(11;12) + va) (2) ((1152200) < (111)2) + vv) /(v2; 03) [(22) [(01v2)) ((11; va).
To calculate I(V1: 02) + (()1 + [(v2) v3) U(U1 vz), we up the individual inequalities, yielding::
T(ei v2) + vg) [222; 03) T(u M2: 03)
(((1) + I(u23 03) + 1(11) + I(p;; 03) T(e 02) + T(11:
2
Σ
I(v:);))
15K<j53
29

---

Learning Optimal Multimodal Information Bottleneck Representations
We then have:
UB (1 + +2) + H(v)) - H(3)) (211)22) ((11)03) (v2; vz) + [(1 U2; vz))
M
+1+ H(v)) - Sisicssa((uso)))
This completes the proof.
Lemma D.5 (Exclusiveness of superfluous information for three modalities). Under Assumption D.I, the objective
function in Equation (69) is optimized when:
F() C {0on, 411, 022, 033; 012, 013, (23)
(80)
Proof. We begin by analyzing the change in the loss function of our optimization after adding modality-specific superfluous information Ç. Let represent U1 -specific superfluous information that incorporated into &. Obviously:
{ $ {a00, Q11, 422, 433, 012, 413, 023), (&) c ((:))) > 0, (()r:)) (E) n (é) 0, (22) n 0, (va) (&) 0.
(81)
Let f (. The difference in loss function between & and is computed as:
(() - (()) B(() (2
- (-I(g:u) + (-It:))--B((())===/((()))+)+))
(82)
= -((E):)) - -((E,Eiv))) 8 m(Em)
- (rr ((::a)))
Here we have:
I(E:y)
I(E1,
7(E:8) - 7(E,)::))
F(E.y)
F(EI, E,y)
I(E:0) +
I(:: E.10)
I(E,,5:0)
F(E. v)
F(
I(Ê1,Eiv)
1()),Eiv)
(83)
F((E v)
F(E1, E,
I(E1:5:V)
I(E1:5:))
F(E, v)
F(E)) + F(E.V)
18.115
0
(()))(())
I((i(E:))) [(E:v])
I(viiEilE)
(84)
=((r1:))) >
30

---

Learning Optimal Multimodal Information Bottleneck Representations
(6)))F(v2==
I((1,E;0) - 1(5;22) = [122: 6115)
(85)
0
1 ((),); (5:
(86)
Thus, have E(E) - (()) 0, if (Éi) o For superfluous information éa specific U2 and Ea specific vs, arrive at the same conclusion. Next, analyze the change in the loss function of our optimization after adding superfluous information shared by two modalities to Ç. Specifically, let {12 represent the superfluous information that is shared between modalities U1 and 222 and not incorporated into Ç. We have:
E12 & {ava, 011, 022, 033, 012, 413, 023). (612] c F(1r), (612) c F(U2),
11)12;01) > 0, I [((12: u2) > 0, T((12;v) 0, (5) (f12 0, (1a) ($12) = 0.
(87)
Let { (E, E12). The difference in loss function between & and g computed as:
(() (()) - + 7(6 (2 (-i(E:))+ (+(
(88)
(?(E:v)- (oso))-T((s, - 7((122 m()
(r((((((22::vz)) - T(E: 02) + (f112 I(Eiv)))
Here we have:
T(E:V) T(E12.E:1)
î(g:v) 7(612.
F(C.V)
F(E12,E,1)
=0
I(E:w) +
F(E. v)
F(E12,E.V)
1(512
I(E12, E:y)
(89)
F(E.V)
F(É12,E,2) E,y)
I(É12.
I(E12:E:1)
F(E, v)
F(112)
F(E.V)
>
(6)))M())))
T(112,E:01) m({:m) = ((111
(90)
((vriEie)
1(112,5:02) - 02):0)) [
f12) 0
(91)
31

---

Learning Optimal Multimodal Information Bottleneck Representations
(
(( - ((: v3) = (va E1215)
(92)
0
Thus, have ((5) - (()) > 0, (f12} + 0. For superfluous information Ç13 shared between modalities ii and ug well as Egs shared between modalities 12 and v3. we arrive at the same conclusion. Finally, we analyze the change the loss function of our optimization after adding superfluous information shared by all three modalities g. Let represent the superfluous information shared by all three modalities and not incorporated into g. Then, have:
Eo $ {aau, 011, O2n, 433; 012, 413, 02s): (Eo) F(e)), (En) F(22), (5o) F(Us), T(Go; vn) > 0, /(o: v2) > 0, (Eo: Uz) > 0, ((G::)) 0, (5) {Eo} 0.
(93)
Let E = (E, The difference in loss function between and is computed as:
(()) ((5) = -((E:)) + +x2())
(-î(g:u)+ + +
(94)
(ī(S:v) - -T(E, + 3 (( ((E:v)
+ +ri((((o.::vz)) - I(E: 12 r2 I(E:v)))
Here we have:
I(E:y)
I660
I(E:v) - I(Eo, c:0)
F(E.v)
F(Eo. E.
-0 :1
I(E:y) +
I(y: EolE)
T(E..E:0)
F(E, 3)
F(Eo. E.
I(Éo.E:V)
I(Eo.
(95)
F(C.V)
F(Eo,E, E,
I(Eor Eit)
I(Eo: Ein)
FF(E v)
F(Eo) F(E, v)
(EOL(EE)
>0
(50))P())=0
I(fo.E:v1) I(E:en)
E(v1:EolE)
(96)
[(vi:Eo) >
I(60. 5: v2) I[[: (0)2
(97)
32

---

Learning Optimal Multimodal Information Bottleneck Representations
IE60 vi) [(5: v3) (vv3 Eo) > 0
(98)
Thus ((E) > 0, {&o} # 0. Put together, the optimization procedure continues until & does not encompass superfluous information, specific to or shared by ti, 222 and Ug. That is, F(5) C {avo, Q11, 422, 433, A12, 013; 023). This completes the proof.
Proposition D.6 (Achievability of optimal MIB for three modalities). Lemmna D.3, and Lemina D.5 jointly demonstrate that the optimal MIB Sopt- achievable through optimization of Equation (69) with 3 E (0, My).
Proof. From Lemma D.3 and Lemma D.5, we have F(E) 2 {ago: Q11, 422, 033, 412, 413, 02s) if 3 E M.). and F(E) S
{aoo: 011, 022, 033: 412; 013: (23), respectively. Thus, F()) {Ago, a11 022; 433 012. 413: a23), which corresponds to Sopt-three in Definition D.2.
To expedite the training process, we can also set M2 502%_ H(v:)- Eisiessa (1rre))) as an upper bound and Mf :=
1 5 (()))) as a lower bound for 3, resulting in B E |MM M,].
E. Estimation of Mutual Information and Information Entropy
We apply the Mutual Information Neural Estimation (MINE) method (Belghazi et al., 2018 8) to estimate the information entropy of each data modality and the mutual information between data modalities. Given two modalities X and Z, MINE employs a neural network, implemented as two-layer Multi-Layer Perceptron (MLP) network with ReLU activation function (Belghazi et al., 2018), to learn a of functions (Te)aee) Each function Te X x Z -> maps sample pairs to real values, enabling mutual information estimation as:
I(x; 2) = sup Exxz [Tp] - log Px (PxxpPle"" OEe
(99)
Here, Exxz [To) represents the expected value of To calculated using sample pairs from the joint distribution Pxz, and Exx Po [e'") represents the expected value o' calculated using sample pairs from the product of marginal distribution Px & Pz. The joint distribution Pxz is approximated using matched sample pairs (X, Z). while Px Pz is approximated using perturbed pairs (X, Z'), where Z' is obtained by shuffling Z. The information entropy H H(X) is computed as the mutual information of X with itself:
H(X)= I(X;X)
(100)
Specifically, in this case, Z and Z' are replaced by X and X', where X' is obtained by shuffling X.
F. Synthetic Data
Following (Xue et al., 2023), simulate pairs of Gaussian observations and task labels, 21 Rd,, 32 &Rdy where I and 22 represent observations from two modalities with dimensionalities dj and d2, respectively, and y E {0, 1) represents the corresponding binary label. The feature vectors of 21 and 22 are defined as:
Ti = [oo ai; ao; a1), x2 = [bo; b2; ao; 02),
(101)
where
ag E dR ~ NO, Ia,) denotes consistent, task-relevant information shared between the modalities;
aj E ddi (0, Ian)) and 02 12 ~ AO Lan)) represent modality task-relevant information;
bo E Rd (0, Tgg) is consistent, superfluous information;
by E Rdz ~ (0, Tapa) and by E Rd22 NNO0, Idza) are modality-specific, superfluous information.
33

---

Learning Optimal Multimodal Information Bottleneck Representations
Here, (0, Ia) denotes multivariate Gaussian distribution with mean 0 and identity covariance matrix / of dimensionality d. The dimensions satisfy:
dp do dg + d11 + d12, d2 = do + dy + d21 d22
(102)
The label y E {0,1] } is generated using Dirac function A, which depends solely the task-relevant information dg. 41. and (2:
y = ( [ao: a1; [1;a2]) > 0)
(103)
where 8 E Rdo tdat ~ (O, La 1:d--in+ii +d21) is randomly sampled vector serving separating hyperplane, and (+,))
denotes the inner product operation.
By adjusting do. d11, and d21, we can control the distribution of task-relevant information across the two modalities, enabling the simulation of imbalanced task-relevant information. Specifically, as illustrated in Figure 3, we simulate three SIM datasets (SIM- (IM-(I-III)) be used in three experimental cases, respectively (see Section 6.2). Firstly, for all cases, we set do = dg 200. For SIM-I used in case i, we set d₁ (500) d21 (100) that a₁ has significantly greater impact on determining y, compared to dg- This configuration implies that Modality dominates Modality II in terms of task-relevant information. For SIM-II used in case ii, we switch the setting of dn and d12, making Modality II dominant over Modality I. Finally, for SIM-III used in case iii, we set d11 = d12 = 300 to ensure both modalities contribute equally to task-relevant information.
G. Detailed Dataset Description
SIM. See Appendix F.
CREMA-D. CREMA-D is an audio-visual dataset designed to study multimodal emotional expression and perception (Cao et al., 2014). It captures actors portraying six basic emotional states happy, sad, anger, fear, disgust, and neutral houtual-through facial expressions and speech.
CMU-MOSI. CMU-MOSI (Zadeh et al., 2016) consists of 93 videos, from which 2, 199 utterance are generated, each containing an image, audio, and language component. Each utterance is labeled with sentiment intensity ranging from -3 to 3.
10x-hNB- (A-H- 10x-hBC-(A-D). The 10x-hNB-(A-H]] datasets comprises eight datasets derived from healthy human breast tissues, while the 10x-hBC-[A-D) datasets contain four datasets from human breast cancer tissues (Xu et al., 2024b). As shown in Figure 6, each dataset corresponds to tissue section and include gene expression and histology modalities. For each tissue section, gene expression profiles (i.e., gene read counts) are measured at fixed spatial spots across the section. During data preprocessing, genes detected in fewer than 10 spots are excluded, and raw gene expression counts are normalized by library size, log-transformed, and reduced to the 3,000 highly variable genes (HVGs) using the SCANPY package (Wolf et al., 2018; Li et al., 2024; Xu et al., 2024a; Du et al., 2025). The corresponding histology image is segmented into 32 32 region patches centered around each spatial spot, from which pathological patches are identified for anomaly detection. OMIB and baseline models are trained on the 10x-hBC- (A-H) datasets to learn multimodal representations of normal tissue regions within a compact hypersphere in the latent space. The trained models are then applied to the (Ax-hBC-(A-D) datasets during inference.
H. Detailed Network Architecture Implementation
-odality-specific encoder.
We implement the encoder as follows:
The SIM datasets: A two-layer MLP with the GELU activation function, outputting 256-dimensional embeddings.
The CREMA-D dataset: Both video and audio encoders use ResNet- producing 512-dimensional outputs.
The CMU-MOSI dataset: ConvlD employed for both the audio and visual modalities, while BERT is utilized for the textual modality, with all three encoders producing 512-dimensional embeddings.
The (Ox-ANB-(A-H) & 10x-hBC-(A-D) datasets: ResNet- 18 and a two-layer graph convolutional network are used for the histology and gene expression modalities, respectively, with both encoders producing 256-dimensional embeddings.
34

---

Learning Optimal Multimodal Information Bottleneck Representations
Table 7: Overview of the experimental datasets.
Dataset
Type
Numher samples (Anomaly proportion)
SIM-(I-III)
Training
9000
SIM-(I-II)
Test
1000
CREMA-D
Training
6,698
CREMA-D
Test
744
MOSI
Training
1281
MOSI
Test
685
10x-hNB-A
Training
2364
10x-hNB-B
Training
2504
10x-hNB-C
Training
2224
10x-hNB-D
Training
3037
10x-hNB-E
Training
2086
10x-hNB-F
Training
2801
10x-hNB-G
Training
2694
10x-hNB-H
Training
2473
10x-hBC-A
Test
346 (12.43%)
10x-hBC-B
Test
295 (78.64%)
10x-hBC-C
Test
176 (27.84 (475)
10x-hBC-D
Test
306 (54.58%)
Task-relevant prediction head. We implement task-relevant prediction head as follows:
The SIM and CREMA-D datasets: The prediction head is implemented as a single linear layer (input X 512 X 100) followed by a softmax layer for classification, producing a k-dimensional output, where is the number of classification types. The TRB loss LTRB is cross-entropy;
The CMU-MOSI dataset: The prediction head is implemented as a single linear layer MLP (input X 50 X 1), outputting a single real value. LTRB is mean squared error;
The Ox-hNB-(AAHH) and 10x-hBC-(A-D) datasets: The prediction head is implemented under the SVDD framework (Ruff et al., 2018; Xu et al.. 2025) as a two-layer MLP (input X 256 X 256) with LeakyReLU activation functions, producing 256-dimensional latent multimodal representations. LeRR is defined as:
N
LTRB =
110
el12 + A R(O),
N
i=1
(104)
N
N
where ý denotes the output of the prediction head, e the center of the hypersphere, ((O) the function that regularizes model parameters θ for reducing model complexity and preventing model collapse, is the regularization weight.
Variational Autoencoder.
The VAE is implemented as two-layer MLP with two heads, outputting the μ and E, respectively.
Cross-Attention Network. For datasets with two modalities, the cross-attention is implemented as:
g = Attn (IG) [|22: Wo, WK. Wv)
(105)
where Attn represents the standard attention block, Wo. WK. and Wy denote learnable projection matrices for queries, keys, and values respectively. The operator represents concatenation along the feature dimension.
35

---

Learning Optimal Multimodal Information Bottleneck Representations
For datasets with three modalities, the cross-attention is extended as:
5 = Attn ([G [<2][]]; Wo. WK. Wv)
(106)
Finally, Linear layer is applied to map & back to the same dimensions as Ç1 and {2.
I. Experimental Settings
All experiments are implemented using PyTorch (Paszke al., 2019), with the following settings:
SIM. We use the Adam optimizer with a learning rate of le-4 and train the model for 100 epochs. The dataset consists of 10,000 samples, split into training and test sets with 9:1 ratio.
The model is trained using the SGD optimizer with a batch size of 64, momentum of 0.9, and weight decay of le-4. The learning rate is initialized at le-3 and decays by a factor of 0.1 every 70 epochs, reaching final value le-4. The dataset is divided into a training set containing 6,698 samples and test set of 744 samples.
CREMA-D.
CMU-MOSI. We employ the Adam optimizer with a learning rate of le-5. All other hyperparameters and settings follow (Mai al., 2023). 2,199 utterances are extracted from the dataset, which are split into training set (1,281 samples) and test set (685 samples).
(UX-hNB-(A-HH&& 10x-hBC-(A-D). We use the Adam optimizer with a learning rate of le-4 and a weight decay of 0.1. The training batch size set to 128. The final multimodal representation has a dimensionality 256.
Abnormal tiesue
detection
Histology medality
Gene expression modsity
Detacted pathology region
Gare
esgression
Figure 6: Genomic multi-modal applications. Genomic data can be divided into two modalities: the histology modality and the gene expression modality. The histology modality comprises tissue image, while the gene expression modality consists of gene expression vectors, where each spot corresponds to vector composed of multiple gene expression values. These two modalities are spatially aligned through shared spatial information. By integrating and analyzing both modalities, abnormal regions within the tissue can be effectively detected.
J. Benchmark Methods
Here, we briefly describe the eight benchmark methods used in this study. For non-MIB- methods:
Concat refers to simple concatenation of multi-modal features, which yet is the most widely used fusion approach.
BiGated (Kiela et al., 2018) flexibly integrates information from different modalities through a gating mechanism.
MISA (Hazarika et al., 2020) decomposes data into modality-invariin and modality-specific representations, using alignment and divergence constraints for better multimodal representation.
36

---

Learning Optimal Multimodal Information Bottleneck Representations
For MIB-based methods:
deep IB (Wang et al., 2019) extends VIB multi-view setting, maximizing mutual information between labels and the joint representation while minimizing mutual information between each view's latent representation and the original data;
MMIB-Cui (Cui et al., 2024) addresses the issues of modality noise and modality gap in multimodal named entity recognition (MNER) and multimodal relation extraction (MRE) by integrating the information bottleneck principle, thereby enhancing the semantic consistency between textual and visual information;
MMIB-Zhang (Zhang et al., 2022) effectively controls the learning multimodal representations by imposing mutual information constraints between different modality pairs, removing task-irrelevant information within single modalities while retaining relevant information, significantly improving performance in multimodal sentiment analysis;
DMIB (Fang et al., 2024) effectively filters out irrelevant information and noise, while introducing a sufficiency loss to retain task-relevant information, demonstrating significant robustness the presence of redundant data and noisy channels.
E-MIB, L-MIB, and C-MIB (Mai al.. 2023) aim learn effective multimodal and unimodal representations by maximizing task-relevant mutual information, eliminating modality redundancy, and filtering noise, while exploring the effects of applying MIB different fusion stages.
K. Evaluation Metrics
Emotion Recognition, we use accuracy (Acc) as the evaluation metric. For Multimodal Sentiment Analysis, we use the mean absolute eиror (MAE) and Pearson's correlation coefficient (Corr) evaluate the predicted scores against the true scores. Additionally, as sentiment intensity scores can be divided into positive and negative categories, Fl-score and polarity accuracy (Acc-2) also utilized to evaluate prediction results binary classification task. Additionally, the interval of [-3,3] contains seven integer scores to which each predicted score is neared to. This allows the using of categorical accuracy (Acc-7) to evaluate the prediction results. Finally, for the Anomalous Tissue Detection task, we evaluate performance using AUC score and F1-score. The AUC score is calculated by varying the anomaly threshold over all tissue regions' anomalous scores. To compute the Fi-score, a threshold is first identified such that the number of regions exceeding it matches the true number of anomalous regions, after which the F1 score computed for regions whose scores are above this threshold.
37

---

Learning Optimal Multimodal Information Bottleneck Representations
L. Algorithmic workflow of OMIB
Algorithm Warm-up training
Input: Modality Uk. &t E {1, Maximum epochs Emaz, Batch size N.
Notation: Ency: Unimodal encoder for modality uks Decg: Task-relevant prediction head for modality uks 2g: Latent representation of modality vgi eg: Stochastic Gaussian noises;
Output: Ence and Deck-
I: Initialize Enc, and Decu, vk E {1,
2: while epoch < Emaz do
3: Sample a batch {ut 1 (1.....,. from each modality & E (1,2):
4: for each E ,I, N do
5: for each modality k E [1,2] do & (
6:
7:
C NNO I);
8:
Decx ef]:
9:
end for
10:
end for
11:
Compute LTRB, as in Equation (4) for each modality & E (1,2};
12: Update Ence and Decs using gradient descent;
end while
return Ence and Deck
38

---

Learning Optimal Multimodal Information Bottleneck Representations
Algorithm 2 Main training
Input: Modality Uk. Unimodal encoderEncc,, Task-relevant prediction head Deca. Ak E (1,2). Maximum epochs Emnns
Batch size N.
Notation: VAEg: Variational encoder for modality thi Sx= Latent representation of modality Uy after reparameterization; CAN: Cross-attention network; Dec: OMF task-relevant prediction head; MINE: Mutual Information Neural Estima-
tion (MINE); ex: Standard Gaussian samples.
Output: Enck, vk E {1, 2), OMF (V AEg. Ak E [1,2]. CAN, and Dec).
[1,2} do 2: H (wa) MINE (U), U&);
for each modality
3: end for
4: T(11; MINE("1, 02):
5: Sample 33 from the range [M. Myl, where Mi == a(H(()))+ (()))) Ma = ((Mo
6: while epoch < Emaz do
7: Sample a batch {uj E (1,2..... N}} from each modality k E (1,2):
8: for each i E f1,2,..., _]] do
9:
for each modality k E [1,2] do of = Ence(1e (vf)): p 2 (zf);
10:
12:
C p4 + Eg x eg
13:
end for
14:
E' CAN G));
15:
for each modality i E {1,2} do
16:
0 = ((ex((=t et)):
17:
end for
18:
Dec(g');
Adjust defined in Equation (11);
20:
end for
21:
Compute LOMP in Equation (10), and LTRA, in Equation (4) for each modality i E (1,2):
22:
==LoM> + LTTB, + LTRBI;
23:
Update parameters of Enck, V AEg, CAN, Deca, and Dec using gradient descent;
24: end while
25: return Enck, 4k E (1,2}, OMF (V AV,F, Vk (1,2), CAN, and Dec)
39


Reviews:
[{"review_id": "dJEkz4HgRB", "reviewer": "Reviewer_Z1rd", "summary": "This paper proposes OMIB, a novel framework for learning optimal Mutual Information Bottleneck (MIB) representations in multimodal learning. The authors address the challenge of imbalanced task-relevant information across modalities, which is a key issue in multimodal fusion. OMIB employs a dynamically weighted regularization strategy to optimize mutual information while mitigating redundancy and preserving modality complementarity. The approach leverages VAEs, CAN, and a two-phase training strategy to ensure efficient and adaptive multimodal representation learning. Theoretical derivations establish the conditions for achieving optimal MIB, and the framework is validated on both synthetic and real-world datasets across multiple tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Questions For Authors**\n\n**Q1.** This is a good point. To mitigate potential instability arising from extreme KL-divergence ratio ($KL_r$), we adopt several strategies. First, the raw $KL_r$ is not directly used; instead, the weight $r$ is computed as $1-tahn(\\cdot)$ and bounded, thus preventing extreme values. Since the $tahn$ function is smooth and saturates, $r$ will not change abruptly in case of significant information imbalance (e.g., very large $KL_r$), thus promoting the training stability. Second, we will add a small constant $\\epsilon$ to the denominator in the computation of $KL_r$ to avoid division by $0$ and enhance numeric stability. Third, the batch-averaged KL ratio (i.e. $\\frac{1}{N}\\sum_{n=1}^N r_n$) is used to compute $r$, thus smoothing out sample-level fluctuations of information ratio. Finally, Proposition 1 guarantees the convergence to the optimal MIB under bounded $r$ and $\\beta$, regardless of the information imbalance. Empirical results from our synthetic experiments (SIM-I and -II), where one modality is designed to contain significantly more or less information than the other, further demonstrate this point.\n\n**Q2.** Thanks for this insightful comment. While CAN does not include an explicit contrastive loss, our training objective implicitly enforces diversity. As proved in Lemma 1, CAN's training objective function (Equation 17) ensures that both consistent (shared) and diverse (modality-specific) task-relevant contents are captured in the MIB upon convergence. In particular, the redundancy penalty terms ($I(\\xi;z_i)$) in the objective function prioritize diverse, informative content as modality‐specific features incur less redundancy compared to shared features. Empirically, our synthetic experiments (Table 2) show that our method significantly surpasses MIB that exclusively contains shared\\& task-relevant content, and achieves performance comparable to the authentic optimal MIB. These results confirm the effectiveness of our OMIB framework in accounting for information diversity. We acknowledge that incorporating a contrastive learning term could further promote diversity. However, adding such a term would complicate the theoretical analysis of achieving the optimal MIB and might disrupt the delicate balance between minimizing redundancy and maximizing task relevance established by our current objective. We consider this an intriguing direction for future work, particularly in exploring whether contrastive loss and redundancy penalty are functionally equivalent in enforcing diversity.\n\n**Q3.** Thank you for highlighting this practical concern. The primary goal of this work is to establish a rigorous theoretical foundation for achieving optimal MIB under the classical MIB paradigm. To allow tractable analysis, we adopt simplified assumptions, including the availability of all modalities, which admittedly do not fully capture real-world complexities. However, these assumptions align with the spirit of Ali Rahimi’s NIPS 2017 Test of Time Award remark—*“Simple experiments, simple theorems are the building blocks that help us understand more complicated systems”*.\n\nYour suggestion of incorporating additional mechanisms for handling missing modalities is valuable. One promising approach would be to use modality-complete data to train an auxiliary VAE that maps one modality's observations to the variational parameters of the other. In cases of modality-incomplete data, the available modality could then approximate the variational representations of the missing one via the reparameterization trick prior to CAN-mediated fusion. However, integrating this component would alter the framework's architecture and training objectives and compromise its theoretical guarantees, deviating from our study's original focus. Rather, these extensions constitute our *de novo* future work with a more practical orientation.\n\n**Q4.** We conducted an additional experiment using the MM-IMDb dataset—a challenging text-visual dataset with severe class imbalance. Specifically, it consists of 25,959 sample pairs across 23 movie genres, with the largest class containing 13,967 samples and the smallest 338, representing a 41-fold imbalance. A stratified split was applied to form training (60%), validation (10%), and testing (30%) sets. We evaluated our method and nine benchmark methods using the macro F1-score (see Table below). The results demonstrate that our method outperforms the benchmarks in this more challenging setting.\n\n|Methods|Concat|BiGated|MISA|deep IB|MMIB-Cui|MMIB-Zhang|E-MIB|L-MIB|C-MIB|OMIB|\n|-|-|-|-|-|-|-|-|-|-|-|\n|F1-score|0.218|0.309|0.334|0.374|0.353|0.373|0.303|0.377|0.357|0.409|", "claims_and_evidence": "The paper makes several key claims:\n- OMIB achieves optimal MIB by dynamically balancing modality contributions using an adaptive weighting factor r. Supported by Proposition 2, which explicitly derives r and validates it with synthetic data experiments (SIM-{I-III}). \n- OMIB effectively reduces redundancy and preserves complementary information across modalities. Validated through ablation studies, which show significant performance degradation when OMIB’s fusion mechanism (OMF) or CAN is removed. \n- OMIB outperforms SOTA MIB-based and non-MIB-based fusion methods across multiple benchmarks. Empirical results on CREMA-D, CMU-MOSI, and tissue detection datasets . \n- OMIB maintains computational efficiency and scalability. Complexity analysis confirms O(N) time complexity and scalability tests.", "methods_and_evaluation_criteria": "Yes, however, the datasets chosen (CREMA-D, CMU-MOSI, tissue detection) are strong, but not the hardest multimodal learning benchmarks.", "theoretical_claims": "- The authors provide a rigorous information-theoretic foundation and mathematically derive an upper bound for the mutual information regularization parameter, β. \n- Theoretical results ensure sufficiency, consistency, redundancy, complementarity, and specificity in learned multimodal representations.\n- Proposition 3 validates the achievability of optimal MIB, but a more detailed analysis of robustness conditions (e.g., adversarial robustness) is missing.", "experimental_designs_or_analyses": "Strengths: Synthetic Experiments (SIM-{I-III}), comparison against SOTA methods on real-world datasets, ablation studies and complexity analysis. \n\nWeaknesses: No explicit handling of missing or corrupted modalities, no robustness analysis for biased datasets, generalization to unseen datasets or domains is not tested.", "relation_to_broader_scientific_literature": "The related work section is thorough and well-referenced.", "essential_references_not_discussed": "The related work section is thorough.", "other_strengths_and_weaknesses": "Strengths: \n+ Avoids direct MI computation by reformulating the learning process using VAEs and CAN, making optimization more tractable.\n+ Adaptive modality weighting via r-regularization ensures balanced modality contributions, preventing over-reliance on a dominant modality.\n+ Two-phase training strategy (warm-up + main training) reduces gradient conflicts and improves stability in learning task-relevant features.\n+  Benchmark performance is strong, consistently surpassing state-of-the-art methods in multiple tasks.\n+  Scalability is validated, confirming OMIB’s efficiency for large-scale multimodal datasets.\n\nWeaknesses: \n- Important challenges remain, including robustness to missing modalities, generalization to unseen data. See below.", "other_comments_or_suggestions": "None.", "questions_for_authors": "- The weighting mechanism depends on the KL-divergence ratio, which may still introduce instability in cases where one modality has significantly less information than the other. Can the authors comment on this?\n- The Cross-Attention Network (CAN) enhances modality fusion by ensuring that complementary information is shared across modalities. While CAN improves fusion, it is unclear if it explicitly enforces diversity. A contrastive learning term could further ensure that different modalities contribute non-redundant information.\n- In cases of completely missing modalities, CAN alone may not be sufficient without additional mechanisms such as modality imputation or self-supervised learning. The experiments assume all modalities are always available. Real-world multimodal settings often face missing, corrupted, or misaligned modalities (e.g., a failed camera or noisy audio in speech datasets).\n- The chosen datasets (CREMA-D, CMU-MOSI, tissue detection) are strong benchmarks, but they are not the hardest challenges in multimodal learning. E.g. Evaluation on medical imaging datasets with severe class imbalance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RW5Q6xxXZ1", "reviewer": "Reviewer_rM9B", "summary": "The paper proposes the OMIB framework to learn optimal multimodal information bottleneck (MIB) representations. It introduces a theoretically grounded objective that sets the regularization weight ($\\beta$) within a derived bound and dynamically adjusts weights per modality (using parameter $r$) to balance imbalanced task-relevant information. OMIB combines modality-specific encoders with an optimal multimodal fusion (OMF) block that uses cross-attention, and it is implemented via a variational approximation. Experiments on synthetic data and other tasks (emotion recognition, sentiment analysis, and anomalous tissue detection) are conducted.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weaknesses**\n\n**W1.** Thank you for reminding us of this important point. **Robustness to estimation errors**: MINE is a theoretically validated estimator with strong consistency, which is achieved by optimizing the Donsker-Varadhan (DV) representation, a lower bound of the true MI (Belghazi et al., 2018). The optimal function that tightens the DV bound can be approximated using neural networks owing to universal approximation theorems, ensuring MINE's convergence to the true MI as sample size grows. Our experiments use datasets with relatively large sample sizes (e.g., Pathological Tissue (10,129)), the estimation error thus is expected to be minor. To further mitigate batch-induced bias, we apply exponential moving averaging on gradients during MINE’s optimization. Additionally, in its original study, MINE was shown to accurately estimate mutual information for generating superior single-modal IB for MNIST classification, empirically demonstrating its reliability. Finally, the theoretical bounds $M_u$ and $M_l$ are designed conservatively to tolerate estimation errors. For instance, $M_u$ is set to $\\frac{1}{3 (H(v_1)+H(v_2)-I(v_1;v_2))}$, which intentionally tightens the upper bound to ensure $\\beta$ remains within a safe range even if $H(v)$or $I(v_i;v_j)$ are slightly misestimated, thereby safeguarding convergence to optimal MIB. **Scalability:** MINE scales linearly with data dimensionality and sample size, as shown in its original work. Moreover, its computational cost is amortized since it is only used once to estimate $M_l$ and $M_u$ prior to training.\n\n**W2.** We apologize for the confusion. Actually, OMIB adopts the late-fusion strategy (the OMF block) similar to L-MIB (Mai et al. 2022), where unimodal representations are first condensed to reduce noises via variational IB encoders before fusion using a light CAN. This contrasts with the heavier early fusion occurring in the full information space. To empirically verify OMF's scalability to large-scale datasets, we generated six large synthetic datasets as in SIM-I–III and measured OMF's training and inference time per epoch (see the table below). The results indicate that OMF scales linearly with large-scale datasets.\n||CAN||\n|-|-|-|\n|Samples|Main Training Time per epoch (s)|Inference Time (s)|\n|1e+5|0.2|0.2|\n|2e+5|0.6|0.7|\n|4e+5|1.1|1.0|\n|6e+5|1.6|1.6|\n|8e+5|1.8|1.8|\n|1e+6|2.2|2.1|\n\n**W3.** Since most MIB studies consider up to three modalities, we followed this routine. To address your concern about modality scalability, we conducted an additional experiment on synthetic datasets with up to five modalities. Each modality consists of a shared task-relevant, a unique task-relevant, and a unique task-irrelevant component. To isolate the impact of the number of modalities, the dataset size was fixed at $10^5$. The observed time costs (see Table below) scale approximately quadratically with $M$, consistent with the expected $\\mathcal{O}(M^2)$ complexity.\n|Modalities|Main Training Time (s)|Inference Time (s)|\n|-|-|-|\n|2|41|0.30|\n|3|82|0.64|\n|4|159|1.15|\n|5|267|1.88|\n\n**W4.** Thank you for this excellent comment, which coincides with the most significant challenge of deep MIB learning proposed by Shwartz-Ziv and LeCun (2023). Our primary goal is to establish the theoretical achievability of optimal MIB under the classical MIB paradigm, where downstream task labels are available. As noted by Tian et al. (2020), the optimal MIB inherently depends on the specific task, since what is relevant for one task may be irrelevant for another. So optimal MIB may not be well defined without task labels in the first place. In our formulation, both shared and Modality-Specific Task-Relevant (MSTR) contents are considered, which further complicates the extension to label-free settings. Regarding self-supervised learning (SSL), while recent methods (e.g., DISTANGLEDSSL) have attempted to learn optimal MIB representations without labels, they typically rely on the strong MultiView assumption (Sridharan et al. 2008) that neglects MSTR. When this assumption is violated, MSTR can be excluded. Although this issue can be mitigated by adding regularization to increase the MI between the representations and inputs, the achievability of task-specific optimal MIB is agnostic. Thus, the guarantee of achieving optimal MIB with SSL in the presence of MSTR could be unattainable. For semi-supervised learning, a potential approach is to train OMIB on the available labeled data and then propagate labels to unlabeled data, with regularization such as a prior label distribution from labeled data to enhance generalization. However, the achievability of optimal MIB is not guaranteed either.\n\n**Essential References Not Discussed** \n\nWe also thank you for pointing us to the DISTANGLEDSSL paper, which offers valuable insights into the SSL-based MIB. We will include and discuss it in the related work section of the revised version.", "claims_and_evidence": "The authors support their claims with both theoretical proofs (for weight bounds and dynamic adjustment) and extensive experiments. Synthetic data validates the optimality of $β$ (Figure 3), and downstream experiments show consistent performance gains (e.g., +11.4% AUC improvement in anomalous tissue detection). However, some gains are modest, for instance, sentiment analysis on the CMU-MOSI dataset.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate. The framework is designed to address key MIB challenges (sufficiency vs. conciseness and modality imbalance), and the use of standard metrics (classification accuracy, AUC, F1) on diverse datasets (CREMA-D, CMU-MOSI, 10x-hBC) is well justified.", "theoretical_claims": "I checked the correctness of the proofs presented for Propositions 1–2. The derivations appear logically sound based on standard information theory properties and variational approximations.", "experimental_designs_or_analyses": "The experimental designs are generally sound. The paper evaluates OMIB on synthetic datasets to verify theoretical claims and on multiple real-world supervised tasks for empirical validation. The use of ablation studies (Table 6) to assess the impact of the warm-up phase, cross-attention, and the dynamic regularization factor is a strong point.", "relation_to_broader_scientific_literature": "The paper builds on existing work in multimodal fusion and the information bottleneck framework It addresses known limitations of ad hoc regularization by providing a theoretically derived bound and dynamic adjustment. This situates OMIB as a meaningful extension in the area.", "essential_references_not_discussed": "The paper doesn't cite [1], which tackles a similar challenge but in a broader self-supervised learning context. That work defines optimal shared and modality-specific information, enabling the learning of disentangled multimodal representation spaces. It develops a theoretical framework to assess the quality of disentanglement, even in scenarios where the Minimum Necessary Information (MNI) cannot be achieved—a situation common in real-world applications.\n\n[1] https://openreview.net/pdf?id=3n4RY25UWP", "other_strengths_and_weaknesses": "Strengths:\n\n- Introduces a rigorous theoretical foundation for dynamically balancing modality-specific information using a derived $\\beta$ bound and parameter $r$.\n\n- Validates the approach across synthetic and supervised tasks, demonstrating consistent performance improvements.\n\n- Conducts ablation studies that clearly show the importance of key components (warm-up, cross-attention, OMF block).\n\nWeaknesses:\n\n- The theoretical bounds (e.g., $M_u$) reply on estimating $H_{v_i}$ and $I(v_i;v_j)$, which is non-trivial in practice. The paper briefly mentions using MINE but does not discuss robustness to estimation errors or scalability to high-dimensional data.\n\n- While the cross-attention network’s $O(N⋅M^2)$ complexity is noted, its impact on training/inference time and scalability to large-scale datasets is unexplored. Comparisons with lighter fusion mechanisms (e.g., late fusion) would strengthen practicality claims.\n\n- Modality Scalability: The extension to $>3$ modalities is mentioned but not empirically validated. \n\n- The work focuses on supervised learning. Extensions of the OMIB framework to semi-supervised or self-supervised scenarios are especially very important given the current landscape in AI.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NQYyNaV3gf", "reviewer": "Reviewer_BfWV", "summary": "The author introduces a theoretically guaranteed multimodal information bottleneck approach. This method dynamically adjusts the regularization weights of each modality by considering the varying degrees of task-relevant information across different modalities. Theoretically, the optimization objective proposed by the author is of a remarkably straightforward form, and the practical loss function serves as an upper bound to this theoretical objective, thereby ensuring the feasibility.\n\n\n#### update after rebuttal:  I  don't  change   my assessment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Experimental Designs Or Analyses:**\n\n**Q1.** Thank you for this insightful observation regarding synergistic interactions between modalities. In response, we conducted additional experiments using synthetic data with two modalities ($x_1,x_2$), where $x_1=[a_0;b_0]$, $x_2=[a_1;b_1]$, and $y=\\Delta(a_0^Ta_1>0)$. All  $a_i$ and $b_i$ are sampled from the standard Gaussian, yielding 10,000 sample pairs. Theoretically, in this setting, the TRBs would be unable to distinguish between $a_i$ and $b_i$, leading to the encoder to randomly incorporate information. However, $L_{OMF}$ of the OMF block can capture the inter-modal synergic interactions and inform the encoders to extract task-relevant information. To validate this, we tested our method in two configurations: $L_{OMF}$ is either involved (*w-$L_{OMF}$*) or not (*wo-$L_{OMF}$*) in optimizing the modality encoders. For comparison, we also evaluated a single-modality baseline (*Single*) and the union of the two modalities (*Union*). The table below shows that *w-$L_{OMF}$* achieves the highest accuracy, followed by *Union* and *wo-$L_{OMF}$*, while *Single* performs near random guessing. These results align with our expectations and demonstrate OMF block's effectiveness in capturing inter-modal synergetic interactions.\n||ACC|\n|-|-|\n|*Single*|0.487/0.489|\n|*Union*|0.626|\n|*wo-$L_{OMF}$*|0.725|\n|*w-$L_{OMF}$*|0.812|\n\n**Q2.** As you suggested, we increased the training epochs from 100 to 1,600 and observed that our method slightly improved, with macro-accuracy increasing from 63.6% to 65.4%. We agree that introducing KS or VGGSound datasets can help to more accurately evaluate our method. However, due to the limited rebuttal period and the very large data sizes involved, we plan to leave these results in the revised manuscript later. \n\n**Questions For Authors:**\n\n**Q1.** Thank you for pointing us to these recent methods. Since OGM-GE is originally designed for the classification task, we have included it as a benchmark for CREMA-D only (note that CMU-MOSI is a regression task, and anomalous tissue detection is an SVDD-based unsupervised task). Specifically, OGM-GE achieves a 61.0% accuracy on CREMA-D. Regarding MLA, we regret that its codes have been withdrawn, preventing us from adding it as a benchmark. Nonetheless, we have now discussed both methods in the Related Work and Experiment sections of this revision.\n\n**Q2.** Concatenation can be viewed as a generalized addition for combining the representation $z$ and noise $e$. To see this point, for concatenation, we have $[z^T,e^T] [\\matrix{ W_1 \\cr W_2}] =z^TW_1+e^TW_2$, which reduces to additive form $(z+e)^TW$ when $W_1= W_2$. Notably, conditional GAN (Mirza et al. 2014) also integrates noise vectors via concatenation for controlled generation. Moreover, using concatenation allows us to replace $e$ with the fused MIB $\\xi$ during main training without modifying TRB's network architecture, thus offering more flexible cross-modal interactions compared to addition. Our ablation study on the CREMA-D dataset further confirms that the concatenation of $e$ (or $\\xi$) with $z$ slightly outperforms a simple additive combination (macro-accuracy: 63.6\\% vs. 62.4\\%).\n\n**Other Comments Or Suggestions:**\n\nProperties 1 has been removed from the main text.", "claims_and_evidence": "Most of the methods and propositions in the article are supported by theoretical foundations. However, Equation (3) raises some questions, which will be elucidated in the ‘Questions For Authors’", "methods_and_evaluation_criteria": "The information bottleneck framework proposed by the author addresses the overfitting predicament inherent in multimodal learning, whereas the adaptive regulation of regularization weights effectively addresses the challenges posed by imbalanced learning scenarios.", "theoretical_claims": "I carefully checked the proofs related to section 5.1 (Appendix B) and skimmed through the proofs associated with section 5.2 (Appendix C). The proof section is quite rigorous, and there is no obvious issues.", "experimental_designs_or_analyses": "The simulated two-modality dataset is quite intriguing. However, it does not account for scenarios that require simultaneous decision-making across both modalities, such as sarcasm. For instance, consider $a, b \\sim N(0, I_D)$, with the label $y = \\Delta(a^T b > 0)$. In this case, $I(a; y) = I(b; y) = 0$, while $I(y; a, b) > 0$.\n\nFor the audio-visual dataset, it would be beneficial to include datasets other than CREMA-D, such as KS or VGGSound. Regarding CREMA-D, simply increasing the number of training epochs can enable a straightforward fusion method like concatenation to achieve a score close to 70 with Resnet18. Therefore, to rule out the possibility that the method merely converges quickly, it is advisable to either increase the number of learning epochs or introduce additional datasets.", "relation_to_broader_scientific_literature": "The ingenious construction of the loss in the article and the proofs related to information theory in this section may provide valuable insights for future research in multimodal learning. A well-designed information bottleneck could also potentially benefit downstream tasks.", "essential_references_not_discussed": "The article's citations are quite comprehensive.", "other_strengths_and_weaknesses": "The article is logically structured, with clearly defined and reader-friendly symbols. The theoretical section is particularly detailed and rigorous.", "other_comments_or_suggestions": "Since the derivations are all included in the appendix and only the conclusions are presented in the main text, Properties 1 can be removed from the main text.", "questions_for_authors": "In the experiments, the non-MIB-based methods only include some basic approaches. I am curious about how they compare with newer methods:\nPeng, Xiaokang, et al. \"Balanced multimodal learning via on-the-fly gradient modulation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\nZhang, Xiaohui, et al. \"Multimodal representation learning by alternating unimodal adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nI harbor a degree of skepticism towards the reasoning in Equation (3), which posits that concatenating $e_i$ enhances the model's learning by improving the signal-to-noise ratio. Typically, the introduction of signal-to-noise ratio considerations involves additive noise, such as ( z_i^{noise} = z_i + e_i ), rather than direct concatenation. Moreover, the ablation studies do not include relevant content to substantiate this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}]

Author rebuttal:

Meta-review: This work presents OMIB, a theoretically grounded framework for learning optimal Mutual Information Bottleneck representations in multimodal learning by addressing imbalances in task-relevant information across modalities. OMIB dynamically adjusts regularization weights per modality and uses a two-phase training strategy with VAEs and cross-attention-based fusion to optimize information retention and reduce redundancy. Experimental results on synthetic and real-world tasks demonstrate its effectiveness in learning efficient and adaptive multimodal representations.

The paper has several strength, the synthetic experiments and the comparison against SOTA methods on real-world datasets, ablation studies and complexity analysis are well done. The work introduces a rigorous theoretical foundation for dynamically balancing modality-specific information using a derived  bound and parameter. In future work , the authors could explicitly handle missing or corrupted modalities, perform a robustness analysis for biased datasets, and empirically validate the extension to more than 3 modalities. 

Overall, this is an interesting work that is of interest to the conference.
Decision: Accept (poster)